Hi, I got the same problem when I try to install a Filemaker Server in our new server, first I think the problem was in the arquitecture of the server because we are using a Intel Core Quad Server, after many test, I discover the problem is an incompatibility issue with Java, please tray the next steps:
1.- Open Java Panel in the control panels
2.- Go to the Tap Java in the Java Control Panel
3.- Select the Option "View" to see the Updates and the instalation
4.- If you are running Java JRE 6.11 just mark as Disable.
5.- If don have another version available, go to the next page: http://java.sun.com/products/archive/j2se/6u10/index.html
6.-Download Java JRE version 6.7 and install.
7.- Restar your server
8.- Open Filemaker Deployment Assistant
I hope it helps you.
Greetings