According to the following KB article, FMS12 isn't compatible with 10.9:

http://help.filemaker.com/app/answers/detail/a_id/11930

 

However, it is possible to get it to work with a few adjustments. The database server works just fine, but WPE does not.

Here is how to get WPE working (based on http://fmforums.com/forum/topic/89802-109-mavericks/#6 with a few corrections)

 

Open up Terminal

cd /Library/Server/Web/Config/apache2/

Add the following lines to the end of the httpd_server_app.conf file:

Note: they may already be there, commented out, after installing the FileMaker Server

<IfDefine WEBSERVICE_ON>
Include '/Library/FileMaker Server/Admin/admin-helper/WEB-INF/conf/fmi-test.conf'
Include '/Library/FileMaker Server/Admin/admin-helper/WEB-INF/conf/fm-server-status.conf'
LoadModule proxy_ajp_module libexec/apache2/mod_proxy_ajp.so
Include '/Library/FileMaker Server/Admin/admin-helper/WEB-INF/conf/mod_proxy.conf'
</IfDefine>

Some of the modules may be commented out with

#FMI_Configuration_V1

which you will need to uncomment.

 

Restart your Web server

Restart the FileMaker Server

Restart the Web Publishing Engine

Go to http://<your_server_url>:16000 and perform FileMaker Server 12 Technology Tests in the Troubleshooting section.

 

Tested 10.9, 10.9.1, FMS 12.0v4. Not tested with 12.0v5, but it should work as well.

FMS 12v6 is available which should avoid you having to do these workaround:  http://help.filemake...tail/a_id/13131

Update for OS 10.9.2 and Server.app 3.0.3 : after upgrading my server from 10.9.1 and Server 3.0.2 and rebooting, FileMaker server and the Server Admin java apps work fine, but the Web Publishing Engine was disabled.   Applying the fixes as described in the first post of this thread fixed the issues.