The new installer for FMS12 on Mac 10.9 does not work for me. During deployment, the Web Publishing Engine fails to stop and fails to start. I initially tried to deploy with the OS X version of PHP, then tried with the Filemaker-provided PHP and got the same result. Here is the deployment log. You can see the Error: 10007 instances on starting/stopping WPE. I don't know how to solve this problem. Machine is an Intel Core 2 Duo 2.66GHz with 4GB RAM running new install of OS X 10.9.1.

Deployment started at 2/17/14 12:59 PM
---------------------------------
SETUP: PRECONFIGURE WEB SERVER

Stop web server                                             succeeded
Create FMI virtual directory configuration file             succeeded
Add FMI config file to apache httpd.conf: /Library/FileMaker Server/Admin/admin-helper/WEB-INF/conf/fmi-test.conf succeeded
Update WPE deployment configuration                         succeeded
Test web server (http://172.16.19.2:80/fmi-test/test.xml)   succeeded
SETUP: PRECONFIGURE WEB SERVER

Stop web server                                             succeeded
Create FMI virtual directory configuration file             succeeded
Add FMI config file to apache httpd.conf: /Library/FileMaker Server/Admin/admin-helper/WEB-INF/conf/fmi-test.conf succeeded
Update WPE deployment configuration                         succeeded
Test web server (http://172.16.19.2:80/fmi-test/test.xml)   succeeded

UNDO EXISTING DEPLOYMENT

Step 1: Unconfigure Web Publishing Engine (WPE):
Stop WPE                                                    Error: 10007
Set WPE AutoStart to disabled                               succeeded
Update JWPC prefs file                                      succeeded
Update WPE deployment configuration                         succeeded
Step 2: Unconfigure web server:
Stop web server                                             succeeded
Restart web server                                          succeeded
Update WPE deployment configuration                         succeeded

START DEPLOYMENT

Step 3: Save settings:
Save administrative configuration settings                  succeeded
Step 4: Configure Web Publishing Engine (WPE):
Stop WPE                                                    Error: 10007
Set WPE AutoStart to enabled                                succeeded
Add JVM Route to JWPC Server XML file                       succeeded
Create WPC prefs file                                       succeeded
Create CWPC prefs file                                      succeeded
Update JWPC prefs file                                      succeeded
Start WPE                                                   Error: 10007
Update WPE deployment configuration                         succeeded
Update mDNS service info                                    succeeded
Step 5: Configure web server:
Stop web server                                             succeeded
Back up apache configuration file                           succeeded
Create FMI virtual directory configuration file             succeeded
Add FMI config file to apache httpd.conf: /Library/FileMaker Server/Admin/admin-helper/WEB-INF/conf/fmi-test.conf succeeded
Disable line in httpd_server_app.conf: AddDefaultCharset    succeeded
Disable line in httpd_server_app.conf: LoadModule info_module succeeded
Disable line in httpd_server_app.conf: LoadModule status_module succeeded
Disable line in httpd_server_app.conf: AddModule mod_info.c succeeded
Disable line in httpd_server_app.conf: AddModule mod_status.c succeeded
Disable line in httpd_server_app.conf: <Location /server-status> succeeded
Disable line in httpd_server_app.conf: <Location /server-info> succeeded
Add FMI status config file to apache config file            succeeded
Create FMI mod_proxy config file                            succeeded
Disable line in httpd_server_app.conf: LoadModule proxy_ajp_module libexec/apache2/mod_proxy_ajp.so succeeded
Add mod_proxy_ajp to apache conf file                       succeeded
Add FMI config file to apache httpd.conf: /Library/FileMaker Server/Admin/admin-helper/WEB-INF/conf/mod_proxy.conf succeeded
Update FileMaker API PHP setting: : $__FM_CONFIG['hostspec'] = 'http://127.0.0.1'; succeeded
Stop web server                                             succeeded
Start web server                                            succeeded
Update WPE deployment configuration                         succeeded
Disable line in httpd_server_app.conf: LoadModule php4_module succeeded
Disable line in httpd_server_app.conf: LoadModule php5_module succeeded
Disable line in httpd_server_app.conf: AddModule mod_php4.c succeeded
Disable line in httpd_server_app.conf: AddModule mod_php5.c succeeded
Disable line in httpd_server_app.conf: <IfModule mod_php4.c> succeeded
Disable line in httpd_server_app.conf: <IfModule mod_php5.c> succeeded
Disable line in httpd_server_app.conf: AddType application/x-httpd-php .php succeeded
Disable line in httpd_server_app.conf: AddType application/x-httpd-php-source .phps succeeded
Create FileMaker PHP configuration file                     succeeded
Add FMI config file to apache httpd.conf: /Library/FileMaker Server/Web Publishing/publishing-engine/php/mavericks/httpd.fmi.conf.php succeeded
Update FileMaker API PHP setting: : $__FM_CONFIG['locale'] = 'en'; succeeded
Restart web server                                          succeeded
Step 6: Configure Database server (DBS):
Set WPE Authorization Info                                  succeeded
Set XDBC AutoStart to yes                                   succeeded
Update XDBC deployment configuration                        succeeded
Update WPE deployment configuration                         succeeded
Start XDBC                                                  succeeded
Step 7: Configure AdminServer:
Save WPE deployment settings                                succeeded
Save web server deployment settings                         succeeded
Save DBS deployment settings                                succeeded
Restart WPE                                                 Stopping the WPE(wpe1) component failed: error code = 10007
Reindex log files                                           succeeded

Deployment completed at 2/17/14 1:12 PM
 

The installation notes instruct you to first Uninstall the old version, then reboot, then do the installation.   Did you follow those steps?

 

Also, just rebooting the server at least once is probably a good idea.

The installation notes instruct you to first Uninstall the old version, then reboot, then do the installation.   Did you follow those steps?

 

Also, just rebooting the server at least once is probably a good idea.

I did the uninstall first. I rebooted twice. 

Hmm.   Are you running OS X plain or OSX Server?  I just did an upgrade/install from FM12v4 to 12v6 after upgrading 10.8.5 server to 10.9.1 server and it worked (after following the uninstall/reboot instructions).   This thread 

https://discussions.apple.com/thread/5142222

sounds as if FMS may get confused as to which apache configuration file it's editing?

This was an upgrade from Snow Leopard Server to Mtn Lion Server. I'm not having any troubles with the Server app nor with web serving.