ScriptMaster, NetBeans, FMS 13, XML Parsing, Multi-Threading: Big boom !

[SteveNadeau]

Hi everyone,

 

I'm trying to parse 13 big XML files. The file sizes range from 5 to 55 MB, and I have to accomplish the parsing in a one hour window. All the XML files represent more than 150MB of data.

 

I've tried using plugins to get the XPath working, but they all came to the same problem: everytime I call the XPath query function with both parameters (the xml itself and the XPath string), the plugin ends up releasing the XML in memory. I had extreme performance issues because every time I wanted to execute an XPath query I had to reload the whole XML in the heap.

 

So I got on with ScriptMaster and NetBeans. I used NetBeans to create a library with one simple class. It has two functions. Both functions are static ones:

- LoadXML, which simply puts the content of the XML in a static String

- XPathQuery, which queries the XPath it recieved in parameter.

 

My performances when up the roof when using this technique. As a matter of fact, I gained more than 80% on the timer, going down from 16 to 2 hours for all the files.

 

I was really happy with my development, but here's when the troubles started happening. To get to that 1 hour window, I had the idea of executing in parallel instead of one after the other.

I made up a simple script that called 13 times the same "Execute Script From Server" with a file in parameter.

 

First problem I had was the heap size. It's been fixed with the memorysize file in the documentation section of ScriptMaster.

Now I'm stuck with this other problem.

 

It all starts out smoothly, but around 10 or 20 seconds after everything is started... everything crashes.

 

I get these error in the log file

CONFIG: === Executing plugin function FF_XML_GetXPathResult on thread Thread-1092 ===
Aug 15, 2014 5:02:35 PM com.prosc.fmkit.PluginBridge pushContext
WARNING: pushContext called with function (PluginFunction{name='FF_XML_GetXPathResult'}) when stack size is 1; this either means it was called re-entrantly, an idle handler was called without waiting for a return from a previous call, or pop was not called from last invocation. Current thread: Thread-1093
Previous calls: [PluginFunction{name='FF_XML_Load'}]
Aug 15, 2014 5:02:35 PM com.prosc.fmkit.PluginBridge$3 run
CONFIG: === Executing plugin function FF_XML_GetXPathResult on thread Thread-1093 ===
Aug 15, 2014 5:02:35 PM com.prosc.fmkit.PluginBridge pushContext
WARNING: pushContext called with function (PluginFunction{name='FF_XML_GetXPathResult'}) when stack size is 1; this either means it was called re-entrantly, an idle handler was called without waiting for a return from a previous call, or pop was not called from last invocation. Current thread: Thread-1094
Previous calls: [PluginFunction{name='FF_XML_Load'}]

I get a bunch of these and everything stops, leaving the scripting engine at a 25-50% CPU usage. I need to restart the scripting engine manually to recover from this.

 

 

My question is this: Is ScriptMaster capable of multi-threading ?

 

 

Thank you for your time !

[Wim Decorte]

CONFIG: === Executing plugin function FF_XML_GetXPathResult on thread Thread-1092 ===

CONFIG: === Executing plugin function FF_XML_GetXPathResult on thread Thread-1093 ===

My question is this: Is ScriptMaster capable of multi-threading ?

 

 

 

The answer to that seems to be in your explanation... the plugin calls clearly run on multiple threads.

 

The question seems to be: "how can I avoid these crashes"?

Does each of the 13 PSoS sessions have these plugin crashes?   Or only some?

What are the specs on the server?

 

With XML file that big, I would be looking at doing the processing outside of FM and just import the results...

[john renfrew]

try the VTD library… its designed for large files

 

at http://vtd-xml.sourceforge.net

[SteveNadeau]

Wim,

 

thanks for the quick reply.

 

It only crashes when I execute more than one parse at the same time.

It's currently running in our test lab, on a Windows 7 Pro VM with 8GB and 4 CPUs allocated to it.

Also, the process never seems to take more than 50% of total CPU...

 

I've tested it on a Windows 2012 server and the crashes still occur.

 

We tried talking about importing Excel files instead of XML ones, but the client insisted on using "new technology". The XMLs are pumped from SAP with a third-party software that they don't understand, so there's a lot of garbage in them.

 

 

 

John, I'm using the jaxen and xom libraries. I'll try with the VTD and give you an update. Thank you for sharing.

[SteveNadeau]

Update on this topic

I ended up using a single-thread server-side script to run all of my parsing in a loop. It takes up more time that ways, but it works.

I ran up to another problem though, which was reported by the client: even though I clean up all my object at the end of the script, the FMSAE.EXE process still holds its memory. I did set the memorysize file to 2048, and everytime we run that script on the server, the process swells a little bit more (around 100-200MB at a time). When it gets to around 1.5GB, it stops swelling, and ScirptMaster seems to be re-using the same heap space.

I even tried calling the garbage collector in my JAVA class. I verified that the static String is no longer available, but the memory doesn't drop.

John, I have gotten a look at the VTD object classes, but I haven't got time to test anything yet.

I'm also starting to look at the XSL import. I'm thinking about writing my own XSL Stylesheet so that the XML could be imported directly into Filemaker without using the XPath technology.

I'll try to keep this thread updated with my findings.

EDIT [september 26th, 13:08]: I've place a call to 360Works and sent them my NetBeans project source code so they can analyse everything. I'm waiting for a feedback.