OK, so I have this working now and I want to report back what it took to make it work.
This is an installation of FMServer 11 on MacOS 10.8.4.
On both the FileMaker Server 11 host computer and on my laptop, which will be used for remote administration of the FileMaker 11 server-
- I installed Java 7 latest version from Oracle, and enabled Java in the browser. I also adjusted the Java 7 security settings to the lowest possible setting.
- I enabled Java in my web browser.
On the Database host computer-
- I copied the directory /Library/FileMaker Server to a safe location. It turned out that the uninstaller doesn't remove this directory, but I wanted to be sure I didn't lose the database.
- I ran the uninstall from the FMS11 installer.
- I removed the leftover directory /Library/FileMaker Server
- I installed FMS11 from the latest version of the full installer of 11.0v5.
- I allowed the installer to install Java 6 from Apple (even though Java 7 is already installed from Oracle).
- from FileMaker- "When Java 7 is installed on your machine, FileMaker Server 11 and FileMaker Server 11 Advanced will use Java 7 for the Admin Console and will use Java 6 for the Admin Server, and Custom Web Publishing"
The FMS11 installer placed webpage alias on the desktop for Launching the Admin Console, which took me to a console start page. On that webpage, there's a link to "start admin console". Clicking on that link downloaded a .jnlp file to the downloads folder. The file was named "admin_console_webstart.jnlp" Double clicking on this jnlp file launched Java and opened a Security Warning dialog that asks me to accept the risk. It then downloaded a java applet named for the server that allows me to administer the server.