Performance Issues with large dataset

[daveinc]

Hi,

I have a performance conundrum that I have not been able to figure out for the past month and a half or so and am looking here for some guidance. 

We have a 9-year old FM system that we run our entire Production operation on. We have approximately 150 users in 9 locations across the US and Canada for this solution.

We have a Windows Server 2012 VM with 14 CPU Cores, 60GB RAM, and an SSD Array SAN that we run our Production Server on. We have separate Drives configured for the Operating System and Data. All users not in the facility housing the server use Remote Desktop to access the system.

The design of this system is not optimal for performance, but it has been running very well for the last 2 years (until early February 2020).

In early February, it began to stall no matter what we did when we had more than 120 users or so connected and working normally.

We identified several severely taxing actions, mainly Finds on related tables with millions of records, and eliminated those. No luck.

We added unnecessary extra CPUs, RAM, and Disk space to no avail.

We created a completely new VM with a fresh install of FM Server 16. No love.

Finally in desperation, we moved the server to older, slower hardware with non-SSD hard drive array and voila! the system works fine again.

This older VM is similar in every other way: Windows Server 2012, 14(slower) CPU Cores, 60GB(slower)RAM, FastSCSI Disk Array.

As an anti-bonus, in thinking this would be temporary, we have both the OS and the Data on a single C drive on this older hardware.

It works splendidly on this lesser setup.

We had all variety of hardware experts in to make sure the newer/faster setup was performing correctly. We updated all firmware and restarted the whole setup. 

All benchmark tests show the newer system to be considerably faster in all phases, especially Disk.

We have no problems with any other VMs on this newer/faster setup(including some less intensive Filemaker Servers).

The one thing that occurs on the newer/faster machine that DOES NOT happen on the older/slower one, is that it stalls and the Disk is consumed by writing tons of data to FMTEMPFM* files for an extended period of time while there is no increase in data being written to the .fmp12 files.

This particular FM Server and dataset is the only one that this happens to. We have two other FM Servers on there that have 200 plus users 24 hours a day and nary an issue.

Does anyone know what is happening when the Disk Monitor shows the System Process is writing tons of data to FMTEMPFM files and not writing any more data than normal to the .fmp12 files?

Thanks in advance for any guidance.

Dave

 

[Wim Decorte]

4 hours ago, daveinc said:

We added unnecessary extra CPUs, RAM, and Disk space to no avail.

What kind of monitoring did you do on the server?  There's no point in adding resources until you pin down exactly where your bottleneck is.  The FMS regular stats.log, the topcallstats.log and the Windows perfmon counters are critical in this respect.  Any additional monitoring (see https://www.soliantconsulting.com/tag/zabbix/) is going to give you even more information to go on.

What did the numbers tell you?  Not the hardware benchmarks, those are useless since they don't account for the exact nature, design and load of your solution.

Are all the CPU and disk i/o consumed by those temp writes?  If so which of the FMS processes is responsible for it and what does its CPU usage and memory usage look like?

[daveinc]

Hi Wim, thanks for responding. 

I already know what design issues are problematic in this solution. I'm not looking for tips on redesign of the solution, I'm looking for the reason a slower, older, hardware setup performs MUCH better than a newer, faster, hardware setup with the exact same solution, load, hardware settings(# of CPUs, GB of RAM are identical. For Disk Space, the poor performing system has more free space, SSD drives, and Windows on C:(150GB free) Data files on E:(180GB free). The good performing system has far less free disk space, FastSCSI drives, and both Windows and the Data files are on a single drive C:(34GB free) ), and FM server settings.

I have been looking at this in-depth on both sets of hardware for a long time. On the older, slower hardware, the solution does not write 3MB-6MB per second to FMTEMPFM files for 10 minutes to 1/2 hour at a time at the top of the disk stats in Windows Resource Monitor. The older, slower hardware does write to the FMTEMPFM files, but it is in short bursts of up to 1.5MB per second for 30 seconds at most. The CPU usage never rises above 7% on either system and looks very similar on both. The RAM stays very steady on both with about 5GB in use, 28GB on Standby and 26GB Free. 

The solution itself is not the issue, because it performs quite well on the older, slower hardware. I just want to know what is happening when the System Process(not the fmserver Process) is writing tons of data to FMTEMPFM files for a very long time like it does when the new hardware is performing poorly. When the new hardware is performing well, FMTEMPFM files are having very little data written to them, just like on the older, slower hardware.

I'm guessing there is something wrong with either the CPUs or the RAM on the newer hardware that does not show up in the Resource Monitor. Either that, or faster CPU and Disk hardware is not necessarily better for hosting FM Server in certain peculiar situations.

I would not be bothered by this and just leave the solution on the older, slower hardware, but my boss does not like the idea of investing in new hardware only to find it useless for the main reason for buying it in the first place! The old hardware was supposed to be retired and on the scrap heap by now!

[daveinc]

Hi Wim,

Thanks for the Zabbix recommendation. I am using that to monitor all of our well-running servers now(although I never need to look at it because they all run very well!).

Is there anyone who would know what is indicated when the situation I describe above with the FMTEMPFM files is occurring? I am desperately trying to figure this out with no luck so far and I consider you to be the authority(or at least knowing who the authority would be!) on all things FM.

Thanks in advance for any suggestions,

Dave

[Wim Decorte]

It's not a common problem, I certainly haven't seen it and I touch hundreds of FMSes.  So the trick is going to be in using all available monitoring to figure out what the pattern is that causes this.

My gut feeling is that some other process on that server, likely related to the disk sub system is forcing FM to not being able to recognize or use its own temp files, forcing it to create the same files over and over.

So focus on what is different in the disk subsystem between the two systems and focus monitoring on that for now until we either confirm or rule-out that particular potential issue.

 

[daveinc]

Thanks Wim. I will dig around and see if I can identify anything else that is different.