El Capitan upgrade deleted the entire FileMaker folder along with the databases

[Buckie]

So I saw the new 14.0.4 update popped up on the website and decided I should finally upgrade the server to El Capitan. The instructions said it was "recommended" to uninstall the FM Server first before initiating the upgrade but I decided I could probably go on without uninstalling and went ahead. A couple of hours later right after the 10.10 -> 10.11 upgrade was complete I found out that the entire contents of the /Library/FileMaker Server directory was gone save for an empty Logs folder.

There you go guys. This is what you get for not following the recommendations of your pals at FileMaker Inc.

[Josh Ormond]

Important to follow these steps when upgrading. https://fmhelp.filemaker.com/docs/14/en/fms14_getting_started.pdf#page=62

[Buckie]

The important thing here is that I never even touched FileMaker during the upgrade. Simply initiating the OS upgrade triggered the deletion for unknown reason.

[Josh Ormond]

Did you back them up first?

[Buckie]

This is actually irrelevant. Backed or not, the folder has been erased. I wonder why they didn't also erase my mail storage or web sites folder as well.

No, seriously. Never in my life, and I've been working with OS X servers since version 10.4 with various software suites have I witnessed such an unpredictable and ridiculous deletion. Sure, files got corrupted, websites got hacked, disks got damaged, users misbehaved... but a simple OS upgrade prompting the software to wipe things out with such a deadly precision for no reason? This is something new.

[Josh Ormond]

Oh I wasn't arguing for the rightness of the event. I was  more concerned about you and your files.

I've done dozens of updates and installs and have never seen that happen either. 

[Buckie]

Backup is another thing. Backups were set up to do daily and weekly runs inside that directory that got deleted. That Backups folder in Data. I wanted to set up some minimal backup policy and get to adding a schedule to back up to a separate HDD and to two another machines with this whole server v14/solution being only recently installed. And I also counted on TimeMachine. It worked flawlessly before (on server v10) making copies of both the actual live databases and, more importantly, copies of the Backup folder. This time I didn't bother to double-check its results and it turned out that Backups folder was never copied and copies of live databases did appear in everyday TM sets, however those were older than the actual database. I assume that is because now TM tries to lock the file exclusively before copying and it couldn't do that for a number of days so while the file did appear in every set, its actual date/contents was much older than the date of the set. TM is set to back up to both the internal HDD and another machine and one of those backups had a more recent version than the other.

So the horrible thing here is that the backups inside the FileMaker Server directory were worth s**** - those got deleted without warning along with every other file there and TM just ignored them for reasons unknown.

And man... we're at version 14 now. FM... FM never changes.

[Josh Ormond]

You should never allow TM ( or anything else ) to touch the live files. It DOES result in corruption.

I'm sorry that your update went so poorly. But I can't really blame FM for this. It doesn't sound like you followed the steps provided in the server guide, or that you are following best practices with regards to backups. I would highly recommend that you research some of the best practices regarding backups to save yourself some heartache in the future.

I still don't have any idea why the files or anything would have been deleted. And I feel bad that it happened, but this really isn't the fault of FMI, in my opinion.

[pfroelicher]

Dear Buckie, 

El Capitan introduced SIP. 

Sometimes files not deemed trustful get moved into /library/systemmigration . 

Did you look there for your stuff?

Yours

Pierre

[Buckie]

Pierre,

yes. I've actually searched everywhere and anywhere using find. No trace of fmp12 files. SystemMigration folder contains irrelevant and mostly Apple-made stuff. So unfortunately the files were actually gone. I even tried a recovery pass (trained on several fmp12 files, including an older version of the missing file) on the disk image from the server that I made immediately after noticing the missing data -- nothing had been found. So my guess those were probably deleted somewhere in the middle of the OS setup and got promptly overwritten by new OS files, not during the last stages of setup.

I was reluctant to reply to Josh here, it's a bit irrelevant because Apple seemingly acknowledged it was their bug in the end, but there goes - please don't confuse FM's/Apple's bugs with my carelessness related to backups. I initially lost the data not because I didn't have a backup but due to a bug that triggered a self-destruct mechanism. Now, I could not restore my data later because there were no backups -- that makes sense, but I did not erase the files myself. Imagine a situation where a server destroys your database nightly, Minecraft-style. Using that logic we'd be blaming the user again and again until the last possible reason: you brought it on yourself because you chose FileMaker, so it's all your fault.

I can live with the loss of files, but I don't understand the need to make the saints out of FMI and Apple, they are human just like us here and they don't have a spotless record either.

[Rick Whitelaw]

I know this is no help but I've upgraded my server machine OS several times without uninstalling FMS and never encountered this issue. The SIP idea is a red herring in my opinion. Something strange and heretofore unknown happened with your system. The only time a Mac OS update changes anything in my experience is when it creates an Incompatible Software folder (which is a good thing).

[pfroelicher]

I must say that I upgrade systems with Filemaker Server installed many times without glitches. Always the Library/Filemakerserver/ folder was not influenced. 

[LaRetta]

Nevertheless, it reminds me that any time I perform an upgrade, I should copy my files to an external source just to be safe.  One never know what might happen.  

I too am sorry it did that to you, Buckie.

[Josh Ormond]

On 1/20/2016 at 2:08 PM, Buckie said:

I was reluctant to reply to Josh here, it's a bit irrelevant because Apple seemingly acknowledged it was their bug in the end, but there goes - please don't confuse FM's/Apple's bugs with my carelessness related to backups. I initially lost the data not because I didn't have a backup but due to a bug that triggered a self-destruct mechanism. Now, I could not restore my data later because there were no backups -- that makes sense, but I did not erase the files myself. Imagine a situation where a server destroys your database nightly, Minecraft-style. Using that logic we'd be blaming the user again and again until the last possible reason: you brought it on yourself because you chose FileMaker, so it's all your fault.

I can live with the loss of files, but I don't understand the need to make the saints out of FMI and Apple, they are human just like us here and they don't have a spotless record either.

You can feel free to reply to me any time. I'm not going to be upset. I don't however agree with your reasoning. Yes...it was Apple's bug. But in the end, our best practices are the defense against bugs. Bugs aren't intentional. And the bug you faced, I have rarely seen it happen. Have not 1x had to use plan B to restore the files in all the upgrades I've done over the past 8 years.

If you have a back up...you didn't really lose anything. It is Apple's fault it bugged out. Nothing to do with your files. If they are lost...can't blame anyone else for that.

I'm not saying that to defend FMI or Apple. And I don't think they are perfect. But I won't blame my car manufacture for not putting in a 30 gallon gas tank because I ran out of gas, because I only like to fill up when the gas light comes on...so I won't blame FMI for an OS bug that hosed my files. I simply restore them and nothing is actually lost.

[LaRetta]

We've ALL had very painful things happen to us in this business; whether it is our fault, someone else's fault or simply not planning enough contingency.  Fix the problem not the blame.