Jump to content
daveinc

Performance Issues with large dataset

Recommended Posts

Hi,

I have a performance conundrum that I have not been able to figure out for the past month and a half or so and am looking here for some guidance. 

We have a 9-year old FM system that we run our entire Production operation on. We have approximately 150 users in 9 locations across the US and Canada for this solution.

We have a Windows Server 2012 VM with 14 CPU Cores, 60GB RAM, and an SSD Array SAN that we run our Production Server on. We have separate Drives configured for the Operating System and Data. All users not in the facility housing the server use Remote Desktop to access the system.

The design of this system is not optimal for performance, but it has been running very well for the last 2 years (until early February 2020).

In early February, it began to stall no matter what we did when we had more than 120 users or so connected and working normally.

We identified several severely taxing actions, mainly Finds on related tables with millions of records, and eliminated those. No luck.

We added unnecessary extra CPUs, RAM, and Disk space to no avail.

We created a completely new VM with a fresh install of FM Server 16. No love.

Finally in desperation, we moved the server to older, slower hardware with non-SSD hard drive array and voila! the system works fine again.

This older VM is similar in every other way: Windows Server 2012, 14(slower) CPU Cores, 60GB(slower)RAM, FastSCSI Disk Array.

As an anti-bonus, in thinking this would be temporary, we have both the OS and the Data on a single C drive on this older hardware.

It works splendidly on this lesser setup.

We had all variety of hardware experts in to make sure the newer/faster setup was performing correctly. We updated all firmware and restarted the whole setup. 

All benchmark tests show the newer system to be considerably faster in all phases, especially Disk.

We have no problems with any other VMs on this newer/faster setup(including some less intensive Filemaker Servers).

The one thing that occurs on the newer/faster machine that DOES NOT happen on the older/slower one, is that it stalls and the Disk is consumed by writing tons of data to FMTEMPFM* files for an extended period of time while there is no increase in data being written to the .fmp12 files.

This particular FM Server and dataset is the only one that this happens to. We have two other FM Servers on there that have 200 plus users 24 hours a day and nary an issue.

Does anyone know what is happening when the Disk Monitor shows the System Process is writing tons of data to FMTEMPFM files and not writing any more data than normal to the .fmp12 files?

Thanks in advance for any guidance.

Dave

 

Share this post


Link to post
Share on other sites
4 hours ago, daveinc said:

We added unnecessary extra CPUs, RAM, and Disk space to no avail.

What kind of monitoring did you do on the server?  There's no point in adding resources until you pin down exactly where your bottleneck is.  The FMS regular stats.log, the topcallstats.log and the Windows perfmon counters are critical in this respect.  Any additional monitoring (see https://www.soliantconsulting.com/tag/zabbix/) is going to give you even more information to go on.

What did the numbers tell you?  Not the hardware benchmarks, those are useless since they don't account for the exact nature, design and load of your solution.

Are all the CPU and disk i/o consumed by those temp writes?  If so which of the FMS processes is responsible for it and what does its CPU usage and memory usage look like?

Share this post


Link to post
Share on other sites

Hi Wim, thanks for responding. 

I already know what design issues are problematic in this solution. I'm not looking for tips on redesign of the solution, I'm looking for the reason a slower, older, hardware setup performs MUCH better than a newer, faster, hardware setup with the exact same solution, load, hardware settings(# of CPUs, GB of RAM are identical. For Disk Space, the poor performing system has more free space, SSD drives, and Windows on C:(150GB free) Data files on E:(180GB free). The good performing system has far less free disk space, FastSCSI drives, and both Windows and the Data files are on a single drive C:(34GB free) ), and FM server settings.

I have been looking at this in-depth on both sets of hardware for a long time. On the older, slower hardware, the solution does not write 3MB-6MB per second to FMTEMPFM files for 10 minutes to 1/2 hour at a time at the top of the disk stats in Windows Resource Monitor. The older, slower hardware does write to the FMTEMPFM files, but it is in short bursts of up to 1.5MB per second for 30 seconds at most. The CPU usage never rises above 7% on either system and looks very similar on both. The RAM stays very steady on both with about 5GB in use, 28GB on Standby and 26GB Free. 

The solution itself is not the issue, because it performs quite well on the older, slower hardware. I just want to know what is happening when the System Process(not the fmserver Process) is writing tons of data to FMTEMPFM files for a very long time like it does when the new hardware is performing poorly. When the new hardware is performing well, FMTEMPFM files are having very little data written to them, just like on the older, slower hardware.

I'm guessing there is something wrong with either the CPUs or the RAM on the newer hardware that does not show up in the Resource Monitor. Either that, or faster CPU and Disk hardware is not necessarily better for hosting FM Server in certain peculiar situations.

I would not be bothered by this and just leave the solution on the older, slower hardware, but my boss does not like the idea of investing in new hardware only to find it useless for the main reason for buying it in the first place! The old hardware was supposed to be retired and on the scrap heap by now!

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


  • Similar Content

    • By DPaquin
      When opining the FMS localhost I try to enable the ScriptMaster Plugin. 
      For some reason I click to enable the plugin, I sometime been able to click on save sometime..
      However, each time I close the host reopen the FMS localhost, the plugin is ALWAYS disabled.
      Would someone have an idea?
      With regards!
      Daniel


    • By DPaquin
      When opining the FMS localhost I try to enable the ScriptMaster Plugin. 
      For some reason I click to enable the plugin, I sometime able to click on save sometime not able.
      However, each time I go back to the FMS localhost the plugin is ALWAYS disabled.
      Would someone have an idea?
      With regards!
      Daniel


    • By Richmilnix
      What's the latest version of FMP / FMPA that gets along well w FMS16?
      I'm in a workplace where I have access to version upgrades, but we're working with files hosted on Server v16.
    • By Dr. Zathras
      We're running FMS 16.0.4 on Windows Server 2012 R2 and Java 8 U181 and everything is fine. Unfortunately our IT security people want me to take Java to v8 U201 and I've been caught by Java updates breaking control of FMS in the past.
      Does anybody have any experience on using FMS 16.0.4 and Java v8 U201?
      Thanks.
      Colin Hunter
    • By jduncan
      Hello,
      We recently upgraded from FMS 14 to FMS16 and one of our databases won't open via WebDirect due to the way it's named:  the database title has an ampersand ("&") in it.  This worked just fine for FMS14, but not for FMS16.  The easiest solution seems to be to change the database title.  I'm a little hesitant to jump into this because of a large number of externally held containers.  How would one safely go about changing the database name (I'd like to change the "&" to "and") without losing the linked containers? 
      In case anyone's wondering, I didn't name the database; I inherited it.
  • Who Viewed the Topic

    10 members have viewed this topic:
    milanm  John May - Point In Space  IdealData  doughemi  Lape  Lee Smith  OlgerDiekstra  ash1474  NLR  Ocean West 
×
×
  • Create New...

Important Information

By using this site, you agree to our Terms of Use.