FMS18 All Databases Closing on their own

[jayivan]

One of my FileMaker Server 18 installs has been having a recurring problem for 2 months. It's 18.0.4.428 on macOS 10.14.6 on an older Mac mini (2012) with a 2.3GHz Core i7 and 16GB RAM with 71% free space on the boot volume and 375GB free (of 700GB) free on the external backup volume. Disk First Aid has been run on all volumes. This recurring problem started in late May. Since we were in a work-from-home situation, no updates had been performed on this machine for quite some time prior to this behavior starting.

Essentially, some server process is shutting down all databases at the same time without being prompted to do so; meanwhile the admin console indicates that databases are up and running with users connected (they are not), and server scripts continue to run. The FileMaker Server has been uninstalled and reinstalled. The problem persists. Would love some advice, or recommendations on where to look for the cause.

The symptoms are the same:

• clients cannot connect from any client
• the Event.log logs the request to close the databases and them closing (but no request was made to shut the databases down)
• after the databases are closed, the admin console still indicates that the databases are running and clients are connected (though none are connected)
• after the databases are closed, the event.log shows that server scripts are still running, but logically with error 802 resulting since the files are not hosted
• backup scripts are still running but with errors like 'Schedule "FMS" aborted; no open databases to operate on.'
• a system reboot resolves the issue for between 3 and 10 days
• sometimes, using command line tools to restart just the fms processes will resolve the issue; sometimes, using the command line tools will indicate a fms restart but the symptoms remain (no databases actually hosted, users indicated as being connected for hours though they were not, and could not be since the processes were restarted)
• no crashes are logged by the system
• the timing varies; it is often overnight, but sometimes in the morning or afternoon

 

Here are some sample logs:

020-07-26 21:18:34.767 -0400    Information    150    <redacted>    Schedule "01 Hourly" completed.
2020-07-26 21:18:34.768 -0400    Information    126    <redacted>    Schedule "01 Hourly" scheduled for 07/27/2020 08:05 AM.
2020-07-26 21:42:26.131 -0400    Information    518    <redacted>    Administrator connected: "mexadmin [10.0.87.17]" (fmsadmin).
2020-07-26 21:42:26.136 -0400    Information    520    <redacted>    Administrator disconnected: "mexadmin [10.0.87.17]" (fmsadmin).
2020-07-26 21:42:26.874 -0400    Information    518    <redacted>    Administrator connected: "mexadmin [10.0.87.17]" (fmsadmin).
2020-07-26 21:42:26.880 -0400    Information    140    <redacted>    Closing database "AnthologyEditions"...
2020-07-26 21:42:26.880 -0400    Information    140    <redacted>    Closing database "FMServer_Sample"...
2020-07-26 21:42:26.881 -0400    Information    140    <redacted>    Closing database "Kemado"...
2020-07-26 21:42:26.881 -0400    Information    140    <redacted>    Closing database "TEST"...
2020-07-26 21:42:26.881 -0400    Information    520    <redacted>    Administrator disconnected: "mexadmin [10.0.87.17]" (fmsadmin).
2020-07-26 21:42:27.477 -0400    Information    168    <redacted>    Database "AnthologyEditions" closed.
2020-07-26 21:42:27.497 -0400    Information    168    <redacted>    Database "FMServer_Sample" closed.
2020-07-26 21:42:27.785 -0400    Information    518    <redacted>    Administrator connected: "mexadmin [10.0.87.17]" (fmsadmin).
2020-07-26 21:42:27.798 -0400    Information    520    <redacted>    Administrator disconnected: "mexadmin [10.0.87.17]" (fmsadmin).
2020-07-26 21:42:28.505 -0400    Information    518    <redacted>    Administrator connected: "mexadmin [10.0.87.17]" (fmsadmin).
2020-07-26 21:42:28.507 -0400    Information    520    <redacted>    Administrator disconnected: "mexadmin [10.0.87.17]" (fmsadmin).
2020-07-26 21:42:31.192 -0400    Information    168    <redacted>    Database "Kemado" closed.
2020-07-26 21:42:31.670 -0400    Information    168    <redacted>    Database "TEST" closed.
2020-07-26 22:05:00.104 -0400    Information    148    <redacted>    Schedule "02 Daily" running.
2020-07-26 22:05:00.113 -0400    Error    2    <redacted>    Schedule "02 Daily" aborted; no open databases to operate on.
2020-07-26 22:05:00.113 -0400    Information    126    <redacted>    Schedule "02 Daily" scheduled for 07/27/2020 10:05 PM.
_______

2020-07-09 00:23:16.364 -0400 Information 150 <redacted> Schedule "FMS" completed.

2020-07-09 00:23:16.364 -0400 Information 126 <redacted> Schedule "FMS" scheduled for 07/10/2020 12:00 AM.

2020-07-09 00:28:38.969 -0400 Information 518 <redacted> Administrator connected: "mexadmin [10.0.87.17]" (fmsadmin).

2020-07-09 00:28:38.979 -0400 Information 520 <redacted> Administrator disconnected: "mexadmin [10.0.87.17]" (fmsadmin).

2020-07-09 00:28:39.680 -0400 Information 518 <redacted> Administrator connected: "mexadmin [10.0.87.17]" (fmsadmin).

2020-07-09 00:28:39.687 -0400 Information 140 <redacted> Closing database "AnthologyEditions"...

2020-07-09 00:28:39.687 -0400 Information 140 <redacted> Closing database "FMServer_Sample"...

2020-07-09 00:28:39.687 -0400 Information 140 <redacted> Closing database "Kemado"...

2020-07-09 00:28:39.687 -0400 Information 140 <redacted> Closing database "TEST"...

2020-07-09 00:28:39.688 -0400 Information 520 <redacted> Administrator disconnected: "mexadmin [10.0.87.17]" (fmsadmin).

2020-07-09 00:28:40.244 -0400 Information 168 <redacted> Database "AnthologyEditions" closed.

2020-07-09 00:28:40.256 -0400 Information 168 <redacted> Database "FMServer_Sample" closed.

2020-07-09 00:28:40.506 -0400 Information 518 <redacted> Administrator connected: "mexadmin [10.0.87.17]" (fmsadmin).

2020-07-09 00:28:40.530 -0400 Information 520 <redacted> Administrator disconnected: "mexadmin [10.0.87.17]" (fmsadmin).

2020-07-09 00:28:41.408 -0400 Information 518 <redacted> Administrator connected: "mexadmin [10.0.87.17]" (fmsadmin).

2020-07-09 00:28:41.413 -0400 Information 520 <redacted> Administrator disconnected: "mexadmin [10.0.87.17]" (fmsadmin).

2020-07-09 00:28:42.176 -0400 Information 168 <redacted> Database "Kemado" closed.

2020-07-09 00:28:42.396 -0400 Information 168 <redacted> Database "TEST" closed.

2020-07-09 01:29:43.511 -0400 Information 518 <redacted> Administrator connected: "mexadmin [10.0.87.17]" (fmsadmin).

2020-07-09 01:29:43.515 -0400 Information 520 <redacted> Administrator disconnected: "mexadmin [10.0.87.17]" (fmsadmin