FileVault 2 and FMServer

[cbum]

Our university hospital IT is mandating that all Mac servers that contain PHI be encrypted using FileVault. There is a longstanding and strong recommendation by FMI and posts on this board advising against this for FM server, although there are also some dissenting voices.

The relevant passage on the FM Knowledge base pages (http://help.filemaker.com/app/answers/detail/a_id/9650) reads:

"FileVault:  FileVault is a feature that performs on the fly encryption and decryption of data on your hard drive..  However, this added level of security requires additional processing power.  Because of this, it is recommended that FileVault not be used in conjunction with FileMaker Server and your FileMaker databases." (Last Updated: Jan 13, 2016 01:47 PM PST)

2 points to mention: First, no discussion of actual incompatibility or corruption risk, just a requirement for extra processing power, suggesting this is simply a possible performance issue. Some of the comments I heard at Devcon and read here suggest much more serious issues - I would appreciate any information or discussion on the specifics here.

Second, the way FileVault is described, i.e., "on the fly encryption ... decryption" suggests this entry and general discussions recommending against FV use are based on the old FileVault v1, which was replaced with FileVault2 in Lion (!) quite a few years ago. According to a discussion yesterday with Apple tech support (with escalation) the current FV2 uses encryption at rest, not on the fly, making it similar/equal to FM's own encryption offering, and this has been the case since 2011. (http://www.cnet.com/news/about-filevault-2-in-os-x-10-7-lion/)

Nevertheless, a discussion with FM tech support yesterday revealed that FM's position on FV is unchanged, and they did not answer my question if this was based on testing with FV2 (with encryption at rest, which should have no performance or other impact on FMserver after the initial boot up) or simply reflects that FM has not revisited this problem since FV2 came out in 2011.

I would appreciate some clarity on this issue. My IT security people are not satisfied with the explanations I tried to give them describing FM's position and are not willing to substitute their current requirement with a third party solution they know nothing about, unless I can give them a coherent and documented explanation.

[Wim Decorte]

How about making them responsible instead of you?

Put together a document outlining and referencing FMI's position and making them acknowledge that want to disregard it and assume all responsibility for what happens with the files going forward.

 

Not sure what they think is "3rd party" about FMI's Encryption-at-Rest.  Apple would be just as much a 3rd party as FMi is, no?

[cbum]

Wim,

Apple, as the OS provider, is 1st party for them, and they have vetted and accepted FV as offered by the OS a option for their security requirements. (It's already a win that they even consider Mac as a useful platform, believe me).  For them, FM is simply a SW vendor for that platform, and they don't want to check every other option out there, I assume.

Regardless, I think it's reasonable to want to know what the strong antipathy to using FV with FMS is based on, and particularly if this even applies to the current version available since 2011, and I can't get any useful response from FM. I'm hoping someone can fill in the blanks. There is no "FM position" as you suggest that I can find beyond the quote I included above, which I hope you agree, does not amount to much and sounds like it is out of date and not applicable to the current FV2.

Edited August 27, 2016 by cbum

[Josh Ormond]

What else are you looking for? FMI says explicitly that you shouldn't use it. Beyond performance, there are a lot of unknowns. From the perspective of FMS, the OS actions/processes are third party.

FMI engineers rarely make statements about not doing something as specific without good reason. There is clearly something that causes and issue. I, too, would like to know what it is...however, I have dealt with many vendors that I don't know exactly the reason for the dos and don'ts. I can sometimes get some explanation, but other times not.

It's often noteworthy that "absence of evidence of a problem is not evidence of the abscence of a problem". We are tryin to tell you what to do, we can only go off what info we have. The exact reasons for this position by FMI, is unknown or not publicly available.

[cbum]

Josh,

a simple statement that their recommendation is based on FV2 rather than FV1 would be a good start. As described above, the current info in the knowledge base article strongly suggests it is not relevant to the current version of FV ("FileVault is a feature that performs on the fly encryption", which is clearly wrong), which has been out for over 5 years, and FM tech support is unwilling to explain the current entry or indicate that it relates to FV2. 

As it is, I'm getting NO information. We can't go off what info we have, since there is none.

Edited August 28, 2016 by cbum

[bruceR]

You know enough already; nobody owes you any further answers.

You know the principles behind FileMaker Server's need to have NO other processes messing with its data or read write operations.

[Josh Ormond]

What benefit does FV 2 provide for FMS? The drive is unlocked and data unencrypted when booted up. Seems kind of pointless...especially when you already have the ability to encrypt the FM files at rest and in transit. 

It is my understanding that some of the processes involved in FV 2 CAUSE problems with FMS. Not sure exactly what it effects, but FMI still holds do not using it. 

[Wim Decorte]

9 hours ago, cbum said:

As it is, I'm getting NO information. We can't go off what info we have, since there is none.

 

The info has always been there; and it is the basis of ALL the best practices that have to do with the health of the files and their data:

- FMS requires exclusive control over the files, any and all processes that might interfere with that pose a risk.

- (examples include: AV, indexing, encrypting, OS file sharing, 3rd party backups,…)

- the only way to mitigate the risk is not allow these services

 

Allowing Apple's FV is a big leap of faith; it requires you to believe that Apple FV's routines are smart enough to interpret what FMS is doing and not doing.  That's a risk that no sensible company will take with their data.

Especially since FMI gives you a completely state-of the art AES-256 encryption-at-rest option.

 

 

[cbum]

Wim,

I know about "FMS requires exclusive control over the files" - that's why I don't use Time Machine as backup etc.

But you guys did read my post, right? Where I mention that FV2 does encryption at rest, not on the fly, exactly what is said about the FM solution.

Where do you see "Apple FV's routines ... being ...  smart enough to interpret what FMS is doing" creating a problem if that is true? 

According to Apple, 2 days ago, nothing is happening regarding FV routines while FMS is running. That is what encryption at rest means, after all. What is vague or unclear about that? Or are you just telling me Apple tech support is lying to me?

Josh, re: "drive is unlocked and data unencrypted when booted": How is that different from the FM version of encryption at rest? Isn't that exactly what it does as well?

re: "It is my understanding that some of the processes involved in FV 2 CAUSE problems with FMS. Not sure exactly what it effects..."

Right, that is what FM people tell me (except they haven't specified FV2 vs FV1, to me at least), so why won't they confirm that that is happening with FV2 as well? Tech support refused to be specific on that point. And if that IS the case, why not update the knowledge base section to state that, so I can show that to my IT sec people?

I do not find it intuitive that there are FV2 processes interfering with FMS if it only does anything on boot up and shutdown (that is exactly how the apple tech guy represented it).

If there is an interaction at those 2 times, then first of all, JUST SAY SO, and second, there is already a setting in FMS that prevents the server to start at boot - I use it because some of my disks are external and take time to spin up, so that should be easy to mitigate.

There was an entire session at DevCon about how to help make FMS more easily accepted in institutional environments - "Just use the FM option" without rational explanation is not helping to be responsive to IT sec requiring FV on macs.

Edited August 28, 2016 by cbum

[Wim Decorte]

1 hour ago, cbum said:

Wim,

I know about "FMS requires exclusive control over the files" - that's why I don't use Time Machine as backup etc.

But you guys did read my post, right? Where I mention that FV2 does encryption at rest, not on the fly, exactly what is said about the FM solution.

 

You're missing the point: when does FV2 decide that the file is at rest?  That's where the risk is... same as time machine, dropbox sync,...

[cbum]

According to Apple, is decrypts at boot up, and encrypts at shut down. While the server is running, it's transparent and nothing is happening.

Do you know anything different?

As such, it's simply designed as safety against physical theft, nothing else - IT sec is fine with that, as all network traffic is secured by SSL by FMS.

I'm assuming that's how the FM version of encryption at rest does it as well, but I don't know?

Edited August 28, 2016 by cbum

[Josh Ormond]

The main item with FM's encryption at rest, is that the file is ALWAYS encrypted. If someone takes the file, there is no way to open it. Except to enter the encryption password. 

With FV 2, if someone pulls that file off the server, it is unencrypted. Unless you have FM's EAR turned on. And at that point, what is the point of turning an unsupported process at the OS level. One that you don't really know when the encryption/decryption happens. If the decryption happens before FMS has closed down the files, you may have a problem. Also, while you are reading Apple's marketing piece for FV 2 as being transparent and nothing happening, you don't really know for sure when it does it's thing. Which means you don't know for sure if the data is at risk.

This is your, and the data owner's, risk assessment to make. I usually lean to the side of following supported configurations. We can only sound the warnings based on the info we have, and our own personal experience ( and that of each other ). While FV was running, I've tested FV 2, and I've running into a lot of unexpected behavior and performance issues.

One REALLY important note. That kb article from FM was written in 2011!!! 2011!!! The same time FV 2 was implemented into 10.7 ( Lion ). It has been updated since then...and the recommendation remains the same. So, FMI engineers clearly know something IS happening behind the scenes.

[webko]

My approach is - if I do not know what the behaviour of FileVault (read: Application X ) is going to be, then I will not use it  - if  the *potential* risk of losing my data for my _mission critical_ application is anything > 0, then I will not allow it to happen.

I do have systems where I really do not care about the data within in them, and I might use other applications there to see if they have unintended consequences - but never on my main systems.

[Mike Duncan]

16 hours ago, cbum said:

According to Apple, is decrypts at boot up, and encrypts at shut down. While the server is running, it's transparent and nothing is happening.

Do you know anything different?

As such, it's simply designed as safety against physical theft, nothing else - IT sec is fine with that, as all network traffic is secured by SSL by FMS.

I'm assuming that's how the FM version of encryption at rest does it as well, but I don't know?

I use FileVault on my laptops, and it is rock solid and completely transparent to the end user. My reading of the kb article seems to recommend against it for performance reasons. Basically, it works the same way if you create an encrypted disk image...it happens at the file system level, and it will do this for you one time and them maintain it at the file system level. It does add a little overhead, much like how SSL adds overhead when serving web pages. Otherwise it acts just like any other volume, and that's what it does to your entire disk when you enable it, and mounts the image on boot.

The biggest difference with FM's own EAR is that if you copy the file to a thumb drive, for example, the file itself is also encrypted, and not just the file system on which it sits. If security is a concern for IT, I would also employ FM EAR in addition to filevault. If a user leaves a machine logged in and unattended, someone could theoretically walk up, plug in a USB drive and copy your files. EAR also protects your backups, in case those are copied to other drives.

Most important, in my opinion, is the physical security of your servers. Make sure they are behind locked doors with limited access, and if they are in a rackmount enclosure, keep it locked as well.

[Josh Ormond]

@Mike Duncan - are  you using it on your FMS machine? Or just your laptops? I also use it on my laptop. But not the server. If there are files other than FM files that need encryption, I either put them on a separate machine, or in an encrypted/separate drive.

There is, despite what Apple markets it as, a performance hit. The machine is busy doing something. With FV on, I've seen a lot of strange behavior with open/closing files, schedules running ( and file creation failing ), intermittent beach balls on the client end. In many scenarios, the only change I made was to turn off FileVault and the problems went away. I suppose the performance hit is minimal if you are just talking only the hit from FV. But on a busy server with 20-70 users, the performance hit gets amplified.

[Mike Duncan]

@Josh Ormond I will admit, nothing in production, but then I haven't recommended FMS on OSX for a while now. Kudos to those brave enough to do it, and doubly for those doing it in an enterprise. If IT mandates filevault be turned on, I don't know if you have much recourse. How long ago did you try running it and experience the issues? Earlier versions were associated with more issues, but nowadays it seems pretty solid.

For an OS X server, would it work if you had the OS with filevault enabled on one volume, the FMS application running from a second volume without filevault, and the hosted files with EAR enabled?

[Josh Ormond]

I had last tested it with FMS 14 on both Yosemite ( 10.10 )  and El Capitan ( 10.11 ). I also wouldn't initially recommend OS X as the server either. But there are still many installations out there running a fair amount of users on it. In fact, we just moved of an older MacPro ( 2008-ish ) that is serving 60-70 users to an enterprise grade server.

If the machine has the spare processing power, I could see some configuration working where the install files for FMS and the FM files themselves are not being touched by the process. But that is not something I've tested. In a previous job, we wrote up a proposal to request an exception be made for the FMS. We made it clear that if FV was required, FMS install was not an option. And the quotes to build the system in something other than FM was in the $400k-$700k range, with a development time of 1-2 years.

[cbum]

Josh & Mike,

Thanks for your comments. The issue that files on a logged in mac are unencrypted and subject to getting copied to a thumb drive etc is definitely real, and may be a strong argument against FV in certain situations, but is not relevant here, as we are not concerned about Mr. Robot-style cyberwarfare. My data does not leave the server, users work on it remotely (using SSL). IP sec is just concerned about someone breaking into (locked) offices and grabbing computer hardware to sell. They don't want unencrypted PMI on stuff that can be carried away, so boot or other drives need to be encrypted by an approved method . PMI does not have commercial value of the kind usual thieves would care about, so I don't see anyone breaking in and sticking in a thumb drive to covertly steal PMI from a running server. 

Josh, your experience with FV on 10.10 and 10.11 does not jive with what Apple is saying. I called them again this morning and gave them the link to this thread. They maintain that FV2's EAR only affects boot up and shut down, there should be no processes interfering with the FMS engine once it's running (I'm a bit confused about your point about opening and closing files showing strange effects - I don't usually open or close files, they keep running once FMS is running...?)

They read your posts, and do not see how FV could be involved, but they agreed to get in touch with the engineering guys to discuss possible low level processes etc that are not usually discussed.

I will let you know if they get back to me.

 

 

[Josh Ormond]

To be honest, I never got around to figuring out exactly what was causing the issues ( interference with FMS processes or file issues ). Turning off FV made everything run better. My guess is, and it's only a guess, is that since FMS runs as a service ( the machine doesn't actually have to be logged in ), that a couple processes were overlapping. It could be that the encryption process altered something in the process. Hopefully they can see something. But in 5 years, FMI's engineers stand by the recommendation to not use it. That says something. If it's fine, I'd love to hear that from FMI engineers.

One other possibility that I didn't get a chance to explore while I was there, Is the hard drive failing? That is a very real possibility.

[John May - Point In Space]

Curions what all y'all's reasoning is for not recommending OS X as a host system for FileMaker Server.  We, obviously, run *lots* of OS X systems with FileMaker Server here and rarely have any problems with them.  Historically, we have seen considerably fewer bug reports regarding FileMaker Server on OS X vs Windows as well.

- John

[cbum]

Apple tech support called back with feedback from their engineers, who echoed what they had said: FV2's EAR means that except for boot up and shut down, there is nothing interfering with running apps. FV2 encrypts the drives at the logical volume level using core data at shut down. And indeed files can be copied unencrypted from a logged in account, so be aware of that.

The engineer asked for details about the FMS issues that had been reported, so I told the support guy to copy and paste the most informative passages from this thread to the engineer. I will update if I hear anything back.

[Josh Ormond]

@John May - Point In Space - I'm not not recommending it. The issue for us, specifically, is the hardware support for the number of users we have hammering on the file all day. We have a couple of Xserves that we might have been able use. However, with the amount of time and effort required for us to make the change and setup the configuration and test it, we only really have one shot to get it right. Since we don't have a dedicated server guy, the 2 developers we have here handle that. So we chose to go with a powerful, new server that has plenty of room to grow. Even with that, there are times of the day when we are stressing FMS.

[John May - Point In Space]

We use 2009 Xserves here.  8 core Xeons with hyperthreading for 16 virtual cores, with up to 192GB RAM.  Backended to Fibre Channel SANs.  Definitely not slouches and handle *lots* of traffic.  Processors haven't made it that far since these machines' inception, despite what system manufacturers may tell you.

- John