Max files limit when running ScriptMaster on Server 13

[Jonathan Perel]

Hi Folks,

 

I'm wondering if anyone out there is having stability issues when running server side scripts which use ScriptMaster on FileMaker 13 Server. Everything seems to run well for a while, but after running for a while, scripts running on FileMaker Server Side start failing.

 

It appears that every time a script using ScriptMaster runs server side, the appopriate JAR files are loaded, but when the script is done those file handles are never released. Eventually, the number of open file handles gets to the maximum allowed for a user/process and the fmsased process can no longer open any files. This means that ScriptMaster can no longer load (even FileMaker functions such as server side sending mail via SMTP won't work), until you reset the server side process via:

fmsadmin restart fmse

You can keep track of the ever growing list of ScriptMaster JAR handles via the following command:

lsof | grep ScriptMaster | grep jar | wc -l

So has anyone out there seen this issue or found a work around? I never saw this issue with FileMaker 12 Server, so I'm not sure if its a FM Server issue or a ScriptMaster issue.

 

I have tried adding an "SMReset" prior to the initialization of ScriptMaster, and when my script is about to end. It does not seem to release the JARs.

 

I'm going to try having my script check the number of open files for the fmsaed process:

lsof -p `pgrep fmsased` | wc -1

and when that number gets too large ( say 100,000) restart the fmse engine with:

fmsadmin -y restart fmse

Cheers,

JP

[john renfrew]

Why load the jars each time??

the jars are loaded into the JVM which instantiated when the server starts (as I understand) so are available on you return if previously loaded...

 

I have taken the route of checking if each jar i need is already loaded and then skipping straight over a load step if it is already in memory, same as on client really. Server is just a headless client.

[Jonathan Perel]

Hey John,

 

I was under the impression that in the Server Side Scripting, a new client session is created every time a scheduled script or a "perform on server" is executed (as each session would have its own global variables). In looking at my logs, I'm seeing that you're correct that the jars (and even the registered ScriptMaster modules) are maintained in the Server Side environment across scheduled script calls. I'll try your suggestion of only loading jars if they aren't loaded already. It still seems like a bug to me that ScriptMaster doesn't release the file handles to when you reload them, or even when you execute the SMReset function. Anyways...

 

I just re-checked my logs and it looks like each time a scheduled server side script is run, there are no loaded jars or registered modules - even if a previous scheduled server side script loaded jars and registered modules. This means that I do have to load jars and register modules every time a scheduled script (or a perform on server script) runs server side, and I will run into the unreleased file handles issue.

 

I made a change in the Load jars for modules("all") function so it checks SMGetLoadedJars to the jar its about to load. This does save me from loading jars mutliple times in a single session, but it doesn't resolve the file handles left open across sessions.

 

Thanks for the info!

JP

Edited February 20, 2015 by Jonathan Perel

[john renfrew]

Jonathan.. thats very interesting and doesn't match my experience..

I have a demo file I use for presentations - creating PDF files using iText, which has three different scripts with one dispatcher script.

 

The dispatcher just runs one of the scripts on server depending on a passed parameter. The OnOpen script just gets out straight away if the file is opened on the server, and in none of them is there any command to load jars or load my ScriptMaster functions... that is handled once with a completely different script to specifically do those two tasks.

Indeed if you just run a script on the server side which sets a field to SMgetloaded jars and SMgetregisteredmodules without any kind of attempt to process, load or use them them then you 'should' find that the contents of the field indeed show persistence.

 

Which log file are you looking in??

[Jonathan Perel]

I created my own event logging database which logs events I generate throughout my code. This screenshot shows two back-to-back scheduled executions of a script "server side - run scripts". I have script "util - Plugin initialize ScriptMaster" call the "Startup" script in the ScriptMaster file. Before that call I check to see if there are any loaded JARs or registered modules, which I repeat at the end of the Scriptmaster initialization.

I took your advice and am skipping reloading previously loaded JARs during the ScriptMaster module registration. You'll see that case where it says "Skipping loaded jar: ..."

The thing to note is that the second time the scheduled script runs, there are no JARs loaded and there are no modules registered. So it looks like each time a scheduled script runs server side (or a "perform script on server" script runs), it runs in a new session (with a clean environment of global variables, plugins, JARs, etc).

 

Cheers,

JP

 

 

[john renfrew]

JP

WhatI tested and observed is that the SMGetLoadedJars and SMGetRegisteredModules do not report correctly on the Server

I created a function which returns the current Groovy version into a text field

On my mac it returns 2.4.0 - as I have the latest ScriptMaster version installed

If I run it on the server I get 2.0.6

If I include the registerGroovy command in the script and also include SMGetRegisteredModules in with the result returned to the field I get '2.0.6 SysDirs' returned as a result

If I then turn off the registerGroovy step, I still get 2.0.6 but NOT SysDirs

BUT the 2.0.6 can only be returned if the function is loaded into the JVM somehow.

Further if I load a jar file and also ask the return to give me SMGetLoadedJars it works as expected the first time when run on the server.

I created a function which needs an external Jar to return a result, and like the registered function it also returns the result I expect each time.

BUT each time after that I get no result from the SMGetLoadedJars - MEANING I can not rely on it as to whether or not a jar is loaded....

This is supported by the demo file I have been using. I have a script I need to run on the server once after the Server has been restarted (it sticks if the file is closed and re-opened), but until the next restart the functions it has created, based on Jar file it has loaded all function as expected WITHOUT needing to be reloaded. The trick seems to be that you need to load the Jar and Create the function in the same PSOS script and then it will stick.

I will post a demo file as soon as I am able...

[Jonathan Perel]

Hi John,

 

I can confirm that there's a problem with SMGetRegisteredModules and SMGetLoadedJars. I tried the following:

 

1. restarting the FMSE

2. registering ScriptMaster with a scheduled script A

3. testing for SMGetRegisteredModules and SMGetLoadedJars with another scheduled script B

4. trying to run ScriptMaster modules with scheduled script B

 

SMGetLoadedJars and SMGetRegisteredModules do not return anything, however, all ScriptMaster modules are actually loaded and working.

 

So something internal to ScriptMaster isn't keeping track of loaded JARS and modules across scheduled script (and perform on server) calls, and so you can end up loading JARs and Modules over and over if you rely on SMGetLoadedJars and SMGetRegisteredModules to tell you what's loaded.

 

JP

[Jonathan Perel]

BTW: I've sent my logs to 360works and they're going to look at this issue.

[Jonathan Perel]

My temporary fix: I create a ScriptMaster module called SMIsRegistered which returns "true". Then in my initialization code, I try to call it and skip initializing ScriptMaster if it returns "true.

 

SMIsRegistered

return ( "true" );

This should keep ScriptMaster from re-initializing when running on Server Side Scripting.

 

JP

[john renfrew]

Like that JP.. simpler version of where was heading

 

What I got is that to correctly load the functions you have to do it right after loading any required jars. Even though they are in memory they apparently can't be seen there next time they might be needed to correctly load the function - so an SMreset would be in order. That's important if you need to update a function or add new new one

 

john

[Jonathan Perel]

You can have a disabled server side script which can run "fmsadmin -y restart fmse", and you can run it manually on the server if you make changes to any of your ScriptMaster modules. That way the next time your server side scheduled (or perform on server) scripts run, then they'll initialize ScriptMaster JARs and Modules when the "SMIsRegistered" call fails.

(or change the SMIsRegistered function to return a serial number, which you check for in your initialization script. Increment it on your initialization script check and in the ScriptMaster module everytime you make a change.)