FMS 13 Custom SSL woes

[tcfitzgerald]

We are working through the steps to bring all of our FileMaker 13 servers to 13.0v9 but are running into a bit of trouble with the custom SSL part.

A little background:

•    We have a process for requesting and obtaining SSL certificates through InCommon (https://www.incommon.org/certificates).  This is our IT Security Office’s preferred way to obtain SSL certificates
•    Technically, Comodo issues the certs, but Comodo is nowhere to be found in the Certification Path of the certs we get from InCommon (they are issued by InCommon RSA Server CA)
•    InCommon is not one of the vendors listed here http://help.filemaker.com/app/answers/detail/a_id/11413

Our IT Security Office and Windows Server Admins asked that we try the InCommon certificates to see if they work.  I generated the certificate request using the “fmsadmin certificate create” command and gave the CSR to our Windows server admins to request an SSL cert.  I am able to import one of the certs using the “fmsadmin certificate import” command, however; after I do this no FileMaker clients are able to connect to the server.  CWP and WebDirect are not able to connect to the databases.  When I use “Open Remote” to try and view the databases on this server I am not able to see any databases.

The Admin console works fine and reports that all of the databases are “Normal”.

I’m not seeing any errors in any of the logs.  I’ve restarted the servers, and have restarted the database server /services several times with no change.

If I turn off SSL, or remove the serverCustom.pem file everything starts working again.

This is a two machine configuration running Windows 2008 R2 Enterprise SP1 currently running 13.0v5.  I’m using FileMaker Pro Advanced 13.0v5.  The certificate is SHA-2, but according to FileMaker that shouldn’t be an issue with the versions I’m running.

Are the symptoms I’m seeing a result of the certificate not being one of the support vendors / types?

Could I have done something wrong in generating the CSR?

Am I importing the wrong certificate?  Here are the options for types of certs I can download:

PKCS#7 Base64 encoded
PKCS#7 Bin encoded
X509, Base64 encoded
X509 Certificate only, Base64 encoded
X509 Intermediates/root only, Base64 encoded
X509 Intermediates/root only Reverse, Base64 encoded

The only one I could actually import was the X509 Certificate only, Base64 encoded cert, all of the rest gave this error:

fmsadmin: This certificate [server.cer] does not match the key file [E:\Program Files\FileMaker\FileMaker Server\CStore\serverKey.pem]

I did notice that the Subject of the resulting certificate does not match the Subject info I supplied when generating the CSR.  It seems the Windows Server Admin who actually did the cert request put in some default values for their group.

Any insight would be helpful and will hopefully help me convince our IT Security Office and Windows Server admins that we actually need to purchase one of the supported SSL certs from FileMaker's list.

Thanks!

[JOrdanMcC]

I found this video very helpful

 

 

I just installed a X509 cert yesterday using Claus' tool.

[tcfitzgerald]

Thanks!  I watched the video and the only difference I can see in my steps is that the FileMaker Server name under "General Settings" was not set to the FQDN of the server.  I updated it to match but it didn't help.

I'm going to try and get a trial / free SSL cert from one of the supported vendors and go from there...

[Vivian M]

Make sure when you copy the certificate and paste it, it is in Plain Text.  I got the certificate on my yahoo account which added formatting to it.  Once we converted the certificate text to plain text, it worked.

 

 

[James Gill]

http://help.filemaker.com/app/answers/detail/a_id/11413/~/list-of-supported-ssl-certificate-types-and-vendors-for-filemaker-platform

 

Unfortunately InCommon is not on the list of approved SSL Vendors (as you apparently already know).  The process you described sounds correct for importing a custom SSL cert, so why not try purchasing from one of the approved vendors to see if that solves your issue?

 

Edit:  I'd also make sure that your firewall rules are allowing secured connections to the Filemaker server.  It's strange that you can't see any connections because last I checked, you can see the databases but will receive an error about the certification if there are issues with it.

Edited April 23, 2015 by James Gill

[Gary Hoffman]

I asked Comodo Support whether the cert I got from InCommon was the same as the Comodo and they told me that it was. The only difference is that there's one more intermediate vendor in the InCommon cert between me and AddTrust. My cert from InCommon seems to work for me and I can get the clients to connect. I have a similar server environment and I just took all my clients to 13v9 after turning on Require Secure Connections on my FMS 13.v9 server.

Handling certs is still a mystery to me.

Gary

[Gary Hoffman]

OK, I am having the same issue with my inCommon cert, which comes from Comodo. I may go through the whole process to get a new cert now with FMS 13v9 or just wait for FMS 14. James may be right that in that the FM approved list is the only certs that work. PITA.

Gary

[Wim Decorte]

James may be right that in that the FM approved list is the only certs that work. PITA.

Gary

​There's no "maybe" about that.  It's explicitly stated.  Even if by some fluke you could make another certificate work, it would still put you outside all support parameters and your IT Security guys would slam you for it, if you ever needed support.

Also: why would you wait for 14? What's your expectation there?

 

[tcfitzgerald]

James:

Thanks for the input!  I'd love to just purchase one of the supported certs, but I think I'm going to need to prove that the InCommon cert just won't work before they let me do that...

It works fine with the default FileMaker supplied cert with SSL on, so I don't think it's a firewall issue.  Using SSL on the server doesn't change the port being used to communicate between the server and clients, it all goes over 5003, SSL or not (at least this is what running package capturing software has indicated, as well as the FileMaker documentation).

I find it odd as well that I don't get an error when trying to list the databases.  I do get an error on the first screen (Connection Failed) if I try to upload a database.

Gary:

A bit confused by your replies...are you saying your InCommon cert worked at first but is no longer working?

At any rate, I was able to get the free trial Quick SSL Premium cert from Geotrust (listed as supported) to work on my personal dev server.  Next step will be to get the same cert for our test/dev server at work and go from there...

[Wim Decorte]

but I think I'm going to need to prove that the InCommon cert just won't work before they let me do that...

​Here's all the proof that you need: http://help.filemaker.com/app/answers/detail/a_id/11413/kw/ssl

If the vendor explicitly states that they only support these then trying an unsupported configuration can potentially compromise security.  I  know off no IT Security department that would insist on going in that direction.

 

 

[tcfitzgerald]

Wim,

I'd love for it to be that easy, however; they look at that list and say, 'InCommon certs are just Comodo certs so it should work fine'.

[Wim Decorte]

NWim,

I'd love for it to be that easy, however; they look at that list and say, 'InCommon certs are just Comodo certs so it should work fine'.

Not sure how to respond at this point.  Not sure if you are talking to one person or multiple at infosec.  Feels to me like you should escalate the issue internally and get past the current point.

From a "due diligence" point of view you could test the certificate just so that you cross your t's and dot your i's.  But I would still put it on record that even if it seems to work that the deployment now strays out of support parameters.  If you ever need support from FMI it is the first thing they will ask you to take out.  You have no guarantee that it will continue to work on the next patch.

To me it feels like this fight is not about the cert, it might be about infosec finally thinking it has a bat to squash FM.  Fight the battle at the level it needs.

 

[tcfitzgerald]

So, just an update on this...

We did end up purchasing third-party SSL certs and they work fine.

However, about two weeks ago, InCommon added the Comodo Elite SSL to the list of certs we could request.  I just tested it out and it worked fine.  Unlink most of the other certs issued by InCommon, the certificate authority in root path is actually Comodo NOT InCommon.