WPE crashing despite fresh installs

[Gnurps]

Our Web Publishing Engine keeps crashing. We have a two-machine setup, both running on Mac El Cap, 10.11.6. Yesterday we did fresh downloads of Server 15.02 to both machines. We did the clean uninstall and reinstall procedure on both machines, as called for by FM KB article 15550. Filemaker tech support says that pretty much rules out Filemaker Server as the culprit. 

Today, the problem is actually worse. The WPE crashed three times this afternoon.  

Symptoms:

• We see no problem with Pro access to Filemaker.
• We see no problem with WebDirect. Several people within the office use that method of connection and it continues to work while CWP goes down. Maybe this means the WPE is not really crashed?
• In Activity Monitor, we do not see spikes in CPU or memory usage.
• The WPE log shows nothing unusual.
• In Filemaker Server console, we do see the WPE slider turned off, not showing green. We cannot slide it back to the on position. To fix that, we restart the worker machine. 

We are considering three possibilities:

1. Software conflict... But not much else is running on these machines and they had a clean install of newest El Cap this past summer. They do have CrashPlan, which never interfered before. We will try quitting it. We have Dropbox running on the master machine and have turned it off. 
2. Hardware problem: We will try running for a different machine, or changing to a one-machine deployment. 
3. PHP web pages causing an issue? There have indeed been some significant PHP page/script changes in recent months. But the web site, an e-commerce store,  appears to run fine until the crash, and we see no CPU or memory spikes in Activity Monitor, and Filemaker Server serves to Pro and WebDirect do not slow down, and the logs do not show problem with any page. The only errors we see in the log are 401's, from users or scripts performing searches that find no records. 

Regarding the possibility that a PHP page is the problem, I would like to look at the Apache logs. Where are they? In Console, they should show up under /var/logs/Apache 2. But that is empty on the worker machine, non-existent on the master. Filemaker Server takes over the Apache service, but where does it put the Apache logs? Anything besides the FM Console log viewer?

I'll report back if we find that one of the machines has a hardware problem, or software conflict. Can anyone consider think of another possibility, or diagnosis approach?

 

[Wim Decorte]

10 hours ago, Gnurps said:

• We see no problem with WebDirect. Several people within the office use that method of connection and it continues to work while CWP goes down. Maybe this means the WPE is not really crashed?

That is correct.  If WPE was crashing then WebDirect would be down too.

No mention of WPE quitting in the FMS event log?

When the web publishing slider is off, try this:

- quit out of the admin console

- from Terminal run 'fmsadmin restart adminserver'  (this does not affect the connected users, it just restarts the part of FMS that is responsible for showing an accurate admin console among other things)

- see if the slider is back in its normal slate.

[Gnurps]

When the problem recurred today, we used the command line to try some fixes. First we ran, as Wim suggested, the fmsadmin restart adminserver. That of course restarted that process. The web publishing engine was still shown as off. Then, on the worker machine, we did fmsadmin restart wpe. The machine responded "Error: 10006 (Service already running)". So we tried fmsadmin stop wpe. That changed the server behavior somewhat. Pages started loading fast. If the page had no calls to Filemaker, it loaded correctly. In pages that did call Filemaker, our error checking captured "Communications terror: (22) The requested URL returned error: 503 Service Unavailable." Makes sense. But then we entered fmsadmin start wpe and got the same error 10006 (service already running). And, in the console on the master machine, the WPE switch remained gray, off. Finally, we restarted the worker machine and the WPE resumed running correctly. 

So we are still rather mystified as to what is going on. 

We have carefully examined the logs and cannot see a pattern that would link a PHP page to the problem, nor a user, nor a file. Will keep looking...

[Wim Decorte]

when you stop the wpe from the command line you also kill WebDirect though so be careful with that.

If the issue happens again, try to hit the worker machine with an XML API url to see if you get data back: something like:

 

http://<your server>/fmi/xml/fmresultset.xml?-dbnames

should give you a list of all hosted files that are available to the XML api (that have the fmxml extended priv enabled in a priv set for an active account).  If you don't have an priv set that has it, add it for testing.