FM 14 Server and Scale HC3 Architecture

[slywilcox]

So our IT group migrated a Filemaker 14 server with a business application that I support.  Since the migration, the system response has been horrible.  Screens that took about a second to render before are now taking 15 seconds.  I have monitored the server and the most significant statistic seems to be network latency.  Most of the other statistics seem fine, but latency has gone from 20-30 milliseconds to 800 milliseconds at times.  Does anyone have experience with the Scale HC3 architecture running Filemaker solutions?  What has your experience been? Are there words of wisdom to make this better?

[Wim Decorte]

Virtualization can work great but it has to be tuned obviously.  No experience here with Scale HC3.

How did you monitor for latency?  What do the FMS usage statistics say?

[slywilcox]

So, I have some additional info.  Our database is roughly 1.5 gb.  The db server has 12 gb of RAM.  But, sometime is page faults up to 2 gb when 8 gb of memory is still available.  Is there a reason FM Server may do this?  Can I turn off paging to see if this helps?

[Wim Decorte]

1 hour ago, slywilcox said:

But, sometime is page faults up to 2 gb when 8 gb of memory is still available.

'page faults' as in the Windows Performance monitor counter?

Paging is handled by the OS, not FMS.  FMS maintains its own cache and writes to that continuously, but that cache will not grow larger than the setting for it in the admin console.

What problems does it cause?  What does the FMS stats.log say?  What counters are high there when this happens?

 

[slywilcox]

'page faults' as in the Windows Performance monitor counter?

 

True, but FMS interacts with the OS to consume resources, memory-disk-network-cpu.  But, you did give me something to check that I had forgotten about.  I have increased the cache setting in FMS to see if page faults goes down.

 

The problem I am having is poor performance on data display, mostly portals.  This SCALE system is significantly slower than the VMWare system that preceded, 6-7x on average.  The performance monitoring I have done seems to indicate a network card problem.  But, my IT group is not convinced.  So, we are working through the process to develop a common understanding of the problem.

I am gathering more stats today after the cache setting change to see if it helps.

 

[slywilcox]

So, I have more information.

Test system configuration

 

	VMWare	SCALE
OS Name	Microsoft Windows Server 2012 R2 Datacenter	Microsoft Windows Server 2012 R2 Standard
Version	6.3.9600 Build 9600	6.3.9600 Build 9600
Other OS Description	Not Available	Not Available
OS Manufacturer	Microsoft Corporation	Microsoft Corporation
System Name	SIS-TEST-A	SIS-TEST
System Manufacturer	VMware, Inc.	Red Hat
System Model	VMware Virtual Platform	KVM
System Type	x64-based PC	x64-based PC
System SKU
Processor	Intel(R) Xeon(R) CPU           E5606  @ 2.13GHz, 2133 Mhz, 2 Core(s), 2 Logical Processor(s)	Common KVM processor, 2100 Mhz, 4 Core(s), 4 Logical Processor(s)
Processor	Intel(R) Xeon(R) CPU           E5606  @ 2.13GHz, 2133 Mhz, 2 Core(s), 2 Logical Processor(s)
BIOS Version/Date	Phoenix Technologies LTD 6.00, 2/22/2012	Seabios 0.5.1, 1/1/2011
SMBIOS Version	2.4	2.4
Embedded Controller Version	0	255.255
BIOS Mode	Legacy	Legacy
BaseBoard Manufacturer	Intel Corporation
BaseBoard Model	Not Available
BaseBoard Name	Base Board
Platform Role	Desktop	Desktop
Secure Boot State	Unsupported	Unsupported
PCR7 Configuration	Not Available	Not Available
Windows Directory	C:\Windows	C:\Windows
System Directory	C:\Windows\system32	C:\Windows\system32
Boot Device	\Device\HarddiskVolume1	\Device\HarddiskVolume1
Locale	United States	United States
Hardware Abstraction Layer	Version = "6.3.9600.17196"	Version = "6.3.9600.17196"
User Name	--	--
Time Zone	Pacific Daylight Time	Pacific Daylight Time
Installed Physical Memory (RAM)	8.00 GB	12.0 GB
Total Physical Memory	8.00 GB	12.0 GB
Available Physical Memory	2.79 GB	7.02 GB
Total Virtual Memory	9.25 GB	13.8 GB
Available Virtual Memory	4.08 GB	8.82 GB
Page File Space	1.25 GB	1.81 GB
Page File	C:\pagefile.sys	C:\pagefile.sys

 

 

I ran a test of one of the problem forms on each server for 3 minutes and logged the statistics at 5 second intervals.  Here are my averages.

 

Category	VMWare	SCALE
Network KB/sec In	24.97297297	6.447368421
Network KB/sec Out	85.32432432	22.65789474
Disk KB/sec Read	159.7297297	40.76315789
Disk KB/sec Written	102.8648649	53.73684211
Remote Calls/sec	138.4054054	36.05263158
Elapsed Time/call	1217.540541	802.4473684
Wait Time/call	4.216216216	0.578947368
I/O Time/call	186.5945946	5.710526316

 

The same solution on VMWare performs on average 6+ times better in screen response.  The VMWare hardware is 6 years older than the SCALE hardware. 

 

I believe that the problem lies with possibly a generally poorer hardware.   I base this on the Network, Disk, and Remote Call (cpu/memory) performance.  Going back is not really an option, but I can move to actual hardware although I prefer VMs for their ease of management.

 

Any thoughts?

 

 

 

[Wim Decorte]

Yes, 6-year old hardware is very crappy.  Also your VMware has only 2 logical processors which is not enough to run FMS on with any kind of solution.

Not sure about the screen response, that doesn't strike me as relevant at all.  In a server  you're not looking at UI update metrics.

 

In general I wouldn't want to draw any conclusions from those numbers.  while there are some big relative differences the absolute differences are so small in real time that they can be caused by anything.

[slywilcox]

Sorry should have said UI update cycle.

The table I am using for a test has about 45k records.  It sums hours from another table that has 135k records.  The average is about 3 entries in detail table to summary table.  But, the portal shows only about 115 at a time, active associates.

  Symptom: Timecard data for company shown in a portal by date, friday's.  The portal list is about 115 entries long by each friday.  I change the date to view another list.  

    Old hardware environment: UI updated in about 1 second.

    New hardware environment: UI updated in about 6-20 seconds.

 

  Scrolling on the same list:

    Old hardware environment:  UI updated in about .5 - 2 seconds.

    New hardware environment: UI updated in about 6-20 seconds.

 

I keep leaning toward a NIC/network issue due to lower throughput differences on statistics.  There may be other smaller issues but that seems like the dominant factor in my view.

My test environments for these statistics had only me performing operations.  No other users were on these systems.  I ran some stats with not activity for a baseline.  Network output from server was extremely low for each server, 0 or near 0.

Do you still feel that there is something else going on?  Do you have an idea of where to look?

 

Thanks for all of your help.

 

Edited June 28, 2017 by slywilcox

[Wim Decorte]

1 hour ago, slywilcox said:

S    Old hardware environment: UI updated in about 1 second.

    New hardware environment: UI updated in about 6-20 seconds.

 

  Scrolling on the same list:

    Old hardware environment:  UI updated in about .5 - 2 seconds.

    New hardware environment: UI updated in about 6-20 seconds.

 

I keep leaning toward a NIC/network issue due to lower throughput differences on statistics.

I highly doubt it.  Unless there is a faulty NIC or a faulty switch port, or very aggressive QoS on the network that de-prioritizes FM traffic, network throughput is almost never an issue.  Of the 4 traditional bottlenecks (memory, processing, network, disk i/o) invariably the issues tend to be disk i/o and processing power (# of cores, speed of cores) in relation to the design of the solution.

Don't have any more ideas; this would require a much more in-depth analysis of the various logs and a look at the solution to see what could be up.