FMS18 on Mac randomly crashing at around 11:55 PM

[DataCruncher]

Hello, 

 

We run several FMS18 instances on Mac OSX machines. 

Randomly, more often on Fri / Sat / Sun, Filemaker server will shut down and disconnect all clients - curiously always at the same time, around 11:55 PM - , and I am unable to restart it from the command line and the web admin drops offline. The only way to bring it back to life is to reboot the entire machine. 

 

Is there any scheduled update job at 11:55 PM that I should be aware of that would cause this behavior? 

 

 

Edited October 4, 2020 by DataCruncher

[johnBuckingham]

Same here; FMS18 on Mojave crashed at around 1pm 2 days running. Did you find any reason, or a fix, for your issue?

Edited October 22, 2020 by johnBuckingham

[DataCruncher]

No / not yet. It still keeps crashing at 11:50 PM like clockwork on several machines. I use the term crashing loosely as when I’m connected, I get the disconnect warning message, so it seems to be a clean shutdown. I only can’t start it back up either from the command line or the web interface as that’s also shut down. Terminal responds Error 10007 to restart, start and stop commands

[DataCruncher]

Bumping this topic. It keeps happening, light clockwork, to all three of our FM 18 servers 11:55 PM every Saturday. 

All filemaker functions drop off, and CLI result in 10007 error. What I did observe is when it goes down, the connected FMP clients get this warning message "Filemaker Server is shutting down now' so this appears to be some sort of graceful commanded shutdown that then kills the entire ecosystem for good. We do not have scheduled script tasks or cron jobs running at that time - why would Filemaker Server do that? 

 

Which log should I check for hints why this is happening?

 

Thank you

[johnBuckingham]

Hi Data Cruncher. It seems that the data restoration function may be responsible for causing this in FMS18. I moved to FMS19 and made sure that Data Restoration was not enabled and mostly it seems more stable. That having been said, one file stopped spontaneously last week but it appears it wasn’t damaged. I hope that helps...

[DataCruncher]

Thank you, John - I have data restoration disabled on all my machines as this can really really have things head south, and fast 🙂

 

I was hesitant to move to FMS19 without identifying the cause first, simply because I did not know what else that would break and whether the same error occurs again. But I'll give it a try!

 

Thank you

[johnBuckingham]

I suspect its not actually possible to properly disable Restoration in 18: see 

I was advised by Tech Support to go to 19. And the permissions issues with that, especially on external drives, were formidable but not insurmountable! You may have an interesting Christmas...

JB

[Wim Decorte]

2 hours ago, johnBuckingham said:

I suspect its not actually possible to properly disable Restoration in 18

The link to the community page doesn't work; but you can most certainly completely disable the data restoration in 18 and 19.0.  In fact, 19.0 installs with the feature turned off by default.  The feature is gone in 19.1.

4 hours ago, DataCruncher said:

why would Filemaker Server do that? 

Because it is instructed to do so.  There is most definitely something on your machine that is triggering a shutdown.  And moving up to 19 is probably not going to solve it since this is a non-FMS event.  Use all the logs available to you through the macOS console app and go through them with a fine comb.  Tedious, and easy to get impatient with it, but the clues will be there.

[johnBuckingham]

Hi Wim. Sorry to hear the link is not working. The reason I tried to show it is that it seems to me to say that even if Resolution is disabled, it will still be on at restart for 18.

[Wim Decorte]

We run hundreds of servers: when you turn it off it stays off.

 

[johnBuckingham]

Hi Wim. I bow to your superior expertise and apologise if I misread, misunderstood or misquoted the advice given on the Claris website. What, then, would be your advice to the OP?

With kind regards, JB

[Wim Decorte]

3 hours ago, johnBuckingham said:

What, then, would be your advice to the OP?

I already gave it a few replies back in this thread: use all the system logs available to trace any and all activity on those boxes - the clues will be there.

Since this is a fairly specific time-frame and happens to all of the servers there it has to be some environmental thing.  Or a piece of extra software that gets installed by default on all machines.  Could be as simple as the control software for the UPSes.

This behavior is not induced by FMS, otherwise it would happen to a lot more servers everywhere and it isn't.  So it is something specific to this deployment.

[DataCruncher]

Thank you Wim and John - I will keep looking. Like I said, it's a graceful shutdown as far as FMS is concerned - but after that has shut down, it won't come back on even from fmsadmin CLI. That's what's weird, but I agree it has to be something in the environment. 

 

[DataCruncher]

SOLVED!

Wim was very right. 

On my virtualized Mac server, I looked high and low and couldn't find any cronjobs that would run at that time. 

However, I did not check the launchd plists!!! D'Oh!

 

Turns out, /Library/LaunchDaemons contained two .plist files that invoked letsencrypt renewal scripts at 11:51 PM every Saturday! The StartCalendarInterval was the dead giveaway. 

 

I assume FMS shuts down into some sort of safe hibernation that disallows it to be re-started if someone tries to mess with the SSL certificate while the server is running. 

 

I disabled the letsencrypt plist launchd's and will observe, but I am very confident that will do the trick!

 

Thank you for all the help - this has been a year in the making.