fmserverd crashed on OSX 10.5.4

[mdecheser]

Hey everyone --

I did a little bit of searching around both fmforums as well as the internet at large, but didn't see anything related to this specific issue, so I thought I would start a thread here.

My organization runs FM9 Server Advanced 9.0.3.326 (latest version to the best of my knowledge) on OSX 10.5.4 (1 rev behind) quad core Intel Xeon processor. The server was purchased this year, so I am not yet willing to consider that this is a hardware issue.

Today, I received notification that the server was inaccessible. I launched the Admin Console and noticed that the console couldn't connect to the DB server. I attempted to manually connect without success. Upon inspection of the OSX Activity Monitor, I noticed that one of the FM processes was missing (wasn't sure which one). So I resolved to bounce the server and pray that processes started up successfully, which they did.

After bouncing the server, some databases started up and then paused themselves, while others opened successfully. I attempted to Resume All, without success, so I closed all DBs and then reopened them. This worked successfully. After getting the DBs back online, I verified DB accessibility and verified other production functionality related to FileMaker Server. Finally, after verifying that everything was back afloat productionwise, I set about troubleshooting the cause.

I looked in two OSX logs in the Console, and found the following:

CrashReporter:

Process: fmserverd [143]

Path: /Library/FileMaker Server/Database Server/bin/fmserverd

Identifier: fmserverd

Version: ? ? ? (

Code Type: X86 (Native)

Parent Process: launchd [1]

Date/Time: 2008-09-24 15:43:27.358 -0400

OS Version: Mac OS X Server 10.5.4 (9E17)

Report Version: 6

Exception Type: EXC_ARITHMETIC (SIGFPE)

Exception Codes: EXC_I386_DIV (divide by zero)

Crashed Thread: 25

(skipping to thread 25)

Thread 25 Crashed:

0 ...ilemaker.dbengine.framework 0x00d24019 Draco::HBEngine::HBEngineThread::Main() + 227

1 ...filemaker.support.framework 0x00b6dd08 Draco::Thread::RunThread() + 46

2 ...filemaker.support.framework 0x00b6e957 Draco::Thread::ThreadUnixProc(void*) + 67

3 libSystem.B.dylib 0x94ebd6f5 _pthread_start + 321

4 libSystem.B.dylib 0x94ebd5b2 thread_start + 34

Thread 25 crashed with X86 Thread State (32-bit):

eax: 0x00003fff ebx: 0x00d23f44 ecx: 0x01a24000 edx: 0x00000000

edi: 0x00000000 esi: 0x018d6f20 ebp: 0xb0db4f28 esp: 0xb0db4eb0

ss: 0x0000001f efl: 0x00010246 eip: 0x00d24019 cs: 0x00000017

ds: 0x0000001f es: 0x0000001f fs: 0x0000001f gs: 0x00000037

cr2: 0x05489000

I suppose I could have tried restarting fmserverd or stopping all fm* processes, then reinitializing them, but without being too savvy as to which processes need to be running in order for others to be successfully initialized, I decided to bounce the server and have the processes do their magic that way. The bounce resulted in bringing things back online, but I still don't have any sufficient resolution as to why the fmserver daemon crashed.

Ironically, I was in the process of testing FM9 Server Advanced on a separate machine running OSX 10.5.5 when all this happened. I'm hoping that the resolution is NOT "upgrade to 10.5.5". Updating seems to be Apple's resolution for many things these days.

Anyone here have insight into this issue?

[IdealData]

Were you running the same licence code on each machine?

[mdecheser]

No. My organization owns 2 licenses of FM9 Server Advanced. One is for production and the other is for development/testing.

[Vaughan]

You may have 2 licenses, but unless each installation uses a different code (or a VLA code) only one instance of the program will be able to run.

[mdecheser]

Thanks for your thoughts.

The FileMaker Server License Key applied to each server is distinct. Each License Key has its own VLA. The servers have historically worked perfectly while running simultaneously on the same network.

Any thoughts on the actual issue itself? I know we're just trying to rule out the obvious stuff.

[Vaughan]

Check that nothing tried to open the live hosted FMP files, this can cause problems. Typical things are backup programs, file system indexing, virus checkers.

[mdecheser]

Thanks for the suggestions. I can verify that we are currently not running antivirus software on the server nor any backup services at this time (this includes Time Machine). The one thing I am not 100% sure about is filesystem indexing. I did pore through both Console, /var/log and /private/var/log to locate any logs registering activity within Spotlight, such as MDS or mdimport. There are no logs that I can see in these locations that bare any correlation to these processes, nor any crash logs relating to Spotlight that I'm able to find.

My knowledge of Spotlight is somewhat murky, and have had at least one situation in the past where Spotlight decided to randomly index my filesystem as I was working on my MacBook Pro. When this process starts, it can dominate your CPU. My limited understanding of the process is that it will easily overtake other processes running the the foreground and background. I've only personally witnessed this happen one time, but it's quite possible that Spotlight, MDS or mdimport has done something similar on my FileMaker servers. The strange thing is that I have been managing FileMaker servers for about 2 years now and never had any issue with Spotlight or related processes overtaking CPU utilization and crashing daemons. My organization's deployment of Leopard server is relatively new and, up until now, without incident. As a matter of practice, any backup software that I run on FileMaker servers targets DB backup directories and not the live DB directories themselves.

Can you advise me of a location and name of logs related to Spotlight or related processes so I can do some further investigation? If I can see that an indexing took place at roughly the same time as the server daemon crashing, I would seek to completely disable Spotlight.

I did find software called "Spotless" but unfortunately it's only available for Tiger.