It is in fact an SSL issue. I fixed the problem by replacing the server.pem SSL certificate file with one from a working system. While this is a full working solution, it comes with a caveat. This will allow you to leave SSL encryption enabled, but having the same cert on multiple client machines is generally a bad idea.
On Win:
-Replace server.pem at the "Program Files > FileMaker > FileMaker " directory.
On Mac:
-Replace server.pem at the "Applications > FileMaker > FileMaker .app > Contents > MacOS" directory. Note: A *.app (application) package can be accessed by right/control clicking on it and choosing "Show Package Contents".
The bug is caused because the FileMaker installer fails to create a valid certificate file on install. Each cert file contains "not before" and "not after" dates to define the scope of the certificate. The "not before" date is the date of the install. The "not after" date is 863913600 seconds (9,999 days) after the install date. If you add 863913600 seconds to any date after September 22th, 2008, the date is incorrectly interpreted by the installer as a date on or after Jan 1st, 1970. This makes the cert unusable because the "not after" date is in 1970 and 2008 is definitely after 1970. This affects multiple versions of FileMaker. Seems like the cause is not a code change or update. Time just had to pass to make this one happen.
The best solution to the problem is to do a reinstall of the application while having your computer's clock set to a date before September 22th, 2008, apply all updates and set the date back to today. The cert that's created should be valid.
We have tried to change the date manually, but the encrypted portion of the certificate relies on the original date set. It did not work.
Loosely related material:
http://www.2038bug.com/
http://en.wikipedia.org/wiki/Year_2038_problem