MAC Server Bogging Down

[datalink]

A client of ours is running with 8-10 seats on v11 and FMS v11 on OSX. At random points during the day the system bogs down with all operators left to sit there with spinning beach balls for a number of minutes. FM doesn't crash. No data is lost. It is just that the system hangs. In a busy environment its untenable.

\

The first step that we took was to upgrade the server hardware and it now has plenty of RAM and a solid state drive etc. I don't suspect the hardware.

Also, when a bog down is happening operators are able to pull down web pages etc at a normal speed, so I'm not inclined to suspect the network.

If I have the activity monitor up on the server at the time that a bog down happens the fmserverd process will show a CPU listing of 175 or more, whereas when things are functioning normally we see CPU listings of ~50.

There is a web application that is being served by this FM system as well, but I don't ever really see FM Web Publishin showing as a CPU hog.

Does anyone have any suggestions?

[Steven H. Blackwell]

A client of ours is running with 8-10 seats on v11 and FMS v11 on OSX. At random points during the day the system bogs down with all operators left to sit there with spinning beach balls for a number of minutes. FM doesn't crash. No data is lost. It is just that the system hangs. In a busy environment its untenable.

\

The first step that we took was to upgrade the server hardware and it now has plenty of RAM and a solid state drive etc. I don't suspect the hardware.

Also, when a bog down is happening operators are able to pull down web pages etc at a normal speed, so I'm not inclined to suspect the network.

If I have the activity monitor up on the server at the time that a bog down happens the fmserverd process will show a CPU listing of 175 or more, whereas when things are functioning normally we see CPU listings of ~50.

There is a web application that is being served by this FM system as well, but I don't ever really see FM Web Publishin showing as a CPU hog.

Does anyone have any suggestions?

So much here to check. First read the logs. Anything else happening at the time of the slowdowns, such as Server Side Scripts, backups, etc. Also are there any external processes happening such as file copying, virus scans (bad bad), disk scan, and so forth?

The first step that we took was to upgrade the server hardware and it now has plenty of RAM and a solid state drive etc. I don't suspect the hardware.

You should suspect it, and you might not have taken the correct course of action here.

How much RAM is installed? How much is allocated to cache in the FMS Console? How are the drives configured and how many of them are there? FileMaker Web Publishing, especially IWP, can be very memory intensive as well. What is the version of OS X Server you are using? What is the exact version of FIleMaker Server you are using?

There are a lot of things to check out here, including NIC cards, switches, the network, etc.

Steven

[datalink]

Hi Steven,

I'm not sure how to do the hand quote function that you do so here is my best copy paste.

You ask:

"So much here to check. First read the logs. Anything else happening at the time of the slowdowns, such as Server Side Scripts, backups, etc. Also are there any external processes happening such as file copying, virus scans (bad bad), disk scan, and so forth?"

There are no sever side scripts. Backups take place after hours and they don't coincide with the bog downs, which seem to be random. No file copying, no virus scans, no disc scan. There isn't anything external going on other that the web activity. In looking at the activity monitor webpublishing does not come up as a process that is running at the time of a bog down, so I'm thinking that that isn't the issue.

You ask:

How much RAM is installed? - 8 gigs

How much is allocated to cache in the FMS Console? - I'm not sure where to check this. I'm sure I'm missing the setting somewhere in the console but just not finding it.

How are the drives configured and how many of them are there? - There is one drive. I'm not sure what you mean by how is it configured. We haven't changed anything from how it shipped when it came out of the box.

What is the version of OS X Server you are using? - We are using OS X 10.5.8 - Not OS X Server

What is the exact version of FIleMaker Server you are using? - The console is v11.0.0.309. I realize that might not be the correct version number for the server software itself, but I'm not sure where to get that.

An odd thing that I have noticed in a review of the logs is that in the Publishing Engine log there are zillions of errors reported all day long. In some cases the error is listed couple times a second. They are listed as Error FM Web Publishing --wpc1 Web Scripting Error: 4, File: "Sprocket", Script: "Gen: WindowAdjust", Script Step: "Adjust Window"

I would imagine that this is the issue. If we turn off web publishing that should cause the server to stop bogging. I'd welcome your thoughts.

[Vaughan]

... the Publishing Engine log there are zillions of errors reported all day long. In some cases the error is listed couple times a second. They are listed as Error FM Web Publishing --wpc1 Web Scripting Error: 4, File: "Sprocket", Script: "Gen: WindowAdjust", Script Step: "Adjust Window"...

It looks as though scripts with web-incompatible steps are being run by the web users. The error specifies the file and script name, and the offending step.

[Steven H. Blackwell]

OK, good. We can make some further presumptions based on this information.

First, the RAM cache setting is in the Admin Console. Configuration-->Database Server-->Databases tab. At the bottom of the screen. Set it as high as it will go, 800 MB. Leave the flush distribution interval at one minute.

I would say to shut off Web Publishing to see what impact that has. I am not sure that it is the WPE that is causing this issue, but it might be.

Do check any server side scripts for incompatible script steps. See the little menu at the bottom of the Manage Scripts UI. Vaughan's suggestion is excellent.

Your server may not be configured optimally from a hardware and OS standpoint. But let's try these other items first.

Steven

[datalink]

Hi Steven and Vaughn,

Thanks so much for your help with this. I have made the RAM Cache settings changes that you suggest. It had been set to 64k rather than the 800k that its now set to.

I'm going give it a couple of hours with the new cache settings to see if we get a bog down, before turning of WPE. If we do bog I'll shut the WPE down.

There are no sever side scripts running on this system and no backups etc. other than the end of day routines that execute after hours of operation, so we are good to go there.

I really appreciate the help and suggestions that you guys are putting forward. Tomorrow is the biggest day of the year for this client with them running around 200 transactions an hour, so there is major pressure to have this issue resolved right away, so thank you so much.

Datalink

[datalink]

I've had another thought:

If the issue is the offending script step being run by WPE (window adjust) could I not just add an If statement in the script and exclude that step if the platform calling the script is not a mac or a pc? In theory that should exclude that repeat error that we are getting every minute all day long. Thoughts?

[IdealData]

I've had another thought:

If the issue is the offending script step being run by WPE (window adjust) could I not just add an If statement in the script and exclude that step if the platform calling the script is not a mac or a pc? In theory that should exclude that repeat error that we are getting every minute all day long. Thoughts?

You can use "Allow User Abort" to conditionally process scripts:

On = unsupported script steps will stop the script (this is the default, if Allow User Abort is not included)

Off = unsupported script steps will be skipped, and the script will continue

The "Off" condition can easily trip you up as your script could easily continue with an operation based on the wrong layout, or context - beware!

If you want to take control yourself then your need to use the Get(ApplicationVersion) to yield the type of client (see FM help).

Get(SystemPlatform) is not based on the client application, it's based on the OS and hardware that is hosting the client, so a web browser would still yield 1, -1 or -2.

[Steven H. Blackwell]

lient (see FM help).

Get(SystemPlatform) is not based on the client application, it's based on the OS and hardware that is hosting the client, so a web browser would still yield 1, -1 or -2.

It's Application Version that will tell you whether a web browser is accessing the file.

Steven

[datalink]

It's Application Version that will tell you whether a web browser is accessing the file.

Steven

By using an If statement in the offending script we have eliminated that error, but we are still having major performance issues. I'm open to any other suggestions/next steps that you might have.

[montek]

By using an If statement in the offending script we have eliminated that error, but we are still having major performance issues. I'm open to any other suggestions/next steps that you might have.

This are simple but important issues to check.

1. See if all you network cards are the same speed (beware of 10 base sometimes they get to remain unnoticed)

2. If you bog downs are the same duration on average definitly is a script.

3. If you server has slow drives any file transfer or upload, or backup will cause a general slow down while the

server finishes that single task involving high network bandwidth and/or massive disk read/write.

4. If the same operation is "triggering" the slow down at the same time every day, isolate it, make the users pay attention

to what they did just before it happens.

5. Check your server drives and network cards speed thoroughly.

6. Check you ethernet cables for assembly errors and impaired wires.

7. Check your hubs and switches, firewall rules can ocurre based on forgoten schedules and conditional events.

8. Check for default backup schedules on the FM server console, remember some schedules are created by default to back up daily, hourly, etc.

9. If available, use a "sniffer" software to monitor you network traffic, it will tell you a lot of things to see how the packages

flow, are rejected or received between all IP devices on your network.

Hope this helps.

Cheers.

[datalink]

Thanks for these ongoing suggestions.

I have just had an unpleasant concern. This solution relies heavily on filtered relationships to check availability of inventory. I'm wondering if as this solution has been used over the years and more data has accumulated if the reliance on these calculations run over these relationships is causing the slow down. If that were the case wouldn't that cause only one terminal to bog down and not all operators on the system. Welcoming any thoughts on this.