I found this bit of information in the help file included in FileMaker Server 9:
http://localhost:16000/admin-help/en/wwhelp/wwhimpl/js/html/wwhelp.htm
If you enable SSL on the web server, then you must use the Deployment assistant to configure FileMaker Server to use the HTTPS protocol to communicate with the web server. The Deployment assistant tries to detect your web server using the HTTP protocol, so if your web server uses HTTPS, the Deployment assistant fails to detect the web server and prompts you for more information. In the Deployment assistant’s Web Server Test Failed step, select HTTPS and continue. See Changing a FileMaker Server deployment.
So if you already have Web Publishing setup,
1. Open the Admin Console.
2. Select the 'Edit Server Deployment' link.
3. Keep web server turned on with SSL and have the Deployment Assistant 'try' to detect the web server. It will keep searching.
4. Stop the web server to trigger the 'cannot find web server' window.
5. Select the HTTPS radio, enter the server's IP address, and enter the https port (443).
6. Start the web server.
7. If you get errors, restart FileMaker Server.