What version of Filemaker are you trying to install. I have a 6.0 version of FileMaker Server that I can try to install on a test server, but if your trying to install a newer version or filemaker pro, and your on a Linux server it won't work. Filemaker only made one product that would install on a Linux/Unix system.
Let me know if you'd like me to give it a try.