Web publishing performance issues

[seancwr]

Hi,

We are having trouble with the web publishing engine. It works fine most of the time, then for no apparent reason a request will take ten second instead of less than one - it isn't any particular page, and a page can work fine on minute and then take 10 seconds, and then be fine again.

System is a server with Xeon processor, 2Gb ram running Windows Server 2003. Its sole purpose is as a Filemaker server.

IIS logs simply show an increase in the amount of time required. A trace on the servers shows as follows. The slow requests take about 10000ms , while normal ones are so fast they show up as 0ms. The delay occurs between CALL_ISAPI_EXTENSION and ISAPI_EXTENSION_DONE - on a normal request there will be not be ten seconds between the timestamps!

-----------------------------------------------------------------------------

Request n.57: http://localhost:80/fmi/xml/fmresultset.xml?-db=News_and_Events&-lay=news_web_short_data&published_date=10%2F04%2F2006&published_date.op=lte&type=Splashback&type.op=eq&-max=10&-sortfield.1=published_date&-sortorder.1=descend&-skip=0&-find {00000000-0000-0000-b80a-0060000000fe} 2006-9-4 3:10:41



  IISGeneral: GENERAL_REQUEST_START - IIS starts processing a new request

    SiteId: 1

    AppPoolId: DefaultAppPool

    ConnId: 1073744567

    RawConnId: 0

    RequestURL: http://localhost:80/fmi/xml/fmresultset.xml?-db=News_and_Events&-lay=news_web_short_data&published_date=10%2F04%2F2006&published_date.op=lte&type=Splashback&type.op=eq&-max=10&-sortfield.1=published_date&-sortorder.1=descend&-skip=0&-find

    RequestVerb: GET

    ContextIDSeq: 57

    Timestamp: 03:10:41.509.610900



  IISGeneral: GENERAL_GET_URL_METADATA - IIS gets URL metadata

    PhysicalPath: c:inetpubwwwrootfmixmlfmresultset.xml

    AccessPerms: Read+Exec+Script

    ContextIDSeq: 57

    Timestamp: 03:10:41.509.610900



  IISGeneral: GENERAL_GET_URL_METADATA - IIS gets URL metadata

    PhysicalPath: C:Program FilesFileMakerFileMaker ServerWeb Publishingweb-configurationweb-server-moduleisapi_redirect.dll

    AccessPerms: Read+Exec+Script

    ContextIDSeq: 57

    Timestamp: 03:10:41.509.610900



  IISISAPI: ISAPI_START - IIS starts processing an ISAPI Request

    ContextIDSeq: 57

    Timestamp: 03:10:41.509.610900



  IISGeneral: GENERAL_ISAPI_HANDLER - IIS processes an in-proc ISAPI request

    ContextIDSeq: 57

    Timestamp: 03:10:41.509.610900



  W3Isapi: CALL_ISAPI_EXTENSION - Call Isapi Extension

    DllName: C:Program FilesFileMakerFileMaker ServerWeb Publishingweb-configurationweb-server-moduleisapi_redirect.dll

    ContextIDSeq: 57

    Timestamp: 03:10:41.509.610900



  W3Isapi: ISAPI_EXTENSION_DONE - Isapi Extension Done

    ContextIDSeq: 57

    Timestamp: 03:10:51.588.251900



  IISISAPI: ISAPI_END - IIS ends processing an ISAPI Request

    ContextIDSeq: 57

    Timestamp: 03:10:51.588.251900



  IISGeneral: GENERAL_REQUEST_END - IIS ends processing a request

    HttpStatus: 200

    HttpSubStatus: 0

    BytesSent: 5421

    BytesReceived: 316

    ContextIDSeq: 57

    Timestamp: 03:10:51.588.251900



 Total time: 10000 msecs

-------------------------------------

I set up logging on the isapi_redirect to see if it could shed any light on the situation.  The full log for a working request and a slow request follows, but it seems to get hung up on the following....

[Wed Oct 04 12:05:35 2006]  [jk_ajp_common.c (884)]: ajp_send_request 2: request body to send 0 - request body to resend 0

[Wed Oct 04 12:05:45 2006]  [jk_ajp_common.c (729)]: received from ajp13 #319

Does this look like a problem with Filemaker?  Any suggestions you have would be appreciated.

Many thanks,

Sean

[Wed Oct 04 12:04:26 2006]  [jk_isapi_plugin.c (696)]: HttpFilterProc started

[Wed Oct 04 12:04:26 2006]  [jk_isapi_plugin.c (759)]: In HttpFilterProc Virtual Host redirection of /localhost/fmi/xml/fmresultset.dtd

[Wed Oct 04 12:04:26 2006]  [jk_uri_worker_map.c (460)]: Into jk_uri_worker_map_t::map_uri_to_worker

[Wed Oct 04 12:04:26 2006]  [jk_uri_worker_map.c (477)]: Attempting to map URI '/localhost/fmi/xml/fmresultset.dtd'

[Wed Oct 04 12:04:26 2006]  [jk_uri_worker_map.c (599)]: jk_uri_worker_map_t::map_uri_to_worker, done without a match

[Wed Oct 04 12:04:26 2006]  [jk_isapi_plugin.c (765)]: In HttpFilterProc test Default redirection of /fmi/xml/fmresultset.dtd

[Wed Oct 04 12:04:26 2006]  [jk_uri_worker_map.c (460)]: Into jk_uri_worker_map_t::map_uri_to_worker

[Wed Oct 04 12:04:26 2006]  [jk_uri_worker_map.c (477)]: Attempting to map URI '/fmi/xml/fmresultset.dtd'

[Wed Oct 04 12:04:26 2006]  [jk_uri_worker_map.c (502)]: jk_uri_worker_map_t::map_uri_to_worker, Found a context match wpc -> /fmi/xml/

[Wed Oct 04 12:04:26 2006]  [jk_isapi_plugin.c (775)]: HttpFilterProc [/fmi/xml/fmresultset.dtd] is a servlet url - should redirect to wpc

[Wed Oct 04 12:04:26 2006]  [jk_isapi_plugin.c (838)]: HttpFilterProc check if [/fmi/xml/fmresultset.dtd] is points to the web-inf directory

[Wed Oct 04 12:04:26 2006]  [jk_isapi_plugin.c (878)]: HttpExtensionProc started

[Wed Oct 04 12:04:26 2006]  [jk_worker.c (132)]: Into wc_get_worker_for_name wpc

[Wed Oct 04 12:04:26 2006]  [jk_worker.c (136)]: wc_get_worker_for_name, done  found a worker

[Wed Oct 04 12:04:26 2006]  [jk_isapi_plugin.c (913)]: HttpExtensionProc got a worker for name wpc

[Wed Oct 04 12:04:26 2006]  [jk_ajp_common.c (1404)]: Into jk_worker_t::get_endpoint

[Wed Oct 04 12:04:26 2006]  [jk_ajp_common.c (1448)]: In jk_endpoint_t::ajp_get_endpoint, time elapsed since last request = 4 seconds

[Wed Oct 04 12:04:26 2006]  [jk_ajp_common.c (1116)]: Into jk_endpoint_t::service

[Wed Oct 04 12:04:26 2006]  [jk_ajp_common.c (295)]: Into ajp_marshal_into_msgb

[Wed Oct 04 12:04:26 2006]  [jk_ajp_common.c (432)]: ajp_marshal_into_msgb - Done

[Wed Oct 04 12:04:26 2006]  [jk_ajp_common.c (642)]: sending to ajp13 #187

[Wed Oct 04 12:04:26 2006]  [jk_ajp_common.c (884)]: ajp_send_request 2: request body to send 0 - request body to resend 0

[Wed Oct 04 12:04:26 2006]  [jk_ajp_common.c (729)]: received from ajp13 #321

[Wed Oct 04 12:04:26 2006]  [jk_ajp_common.c (483)]: ajp_unmarshal_response: status = 200

[Wed Oct 04 12:04:26 2006]  [jk_ajp_common.c (488)]: ajp_unmarshal_response: Number of headers is = 9

[Wed Oct 04 12:04:26 2006]  [jk_ajp_common.c (532)]: ajp_unmarshal_response: Header[0] [Date] = [Wed, 04 Oct 2006 04:04:26 GMT]

[Wed Oct 04 12:04:26 2006]  [jk_ajp_common.c (532)]: ajp_unmarshal_response: Header[1] [server] = [localhost]

[Wed Oct 04 12:04:26 2006]  [jk_ajp_common.c (532)]: ajp_unmarshal_response: Header[2] [MIME-Version] = [1.0]

[Wed Oct 04 12:04:26 2006]  [jk_ajp_common.c (532)]: ajp_unmarshal_response: Header[3] [Cache-control] = [no-cache="set-cookie"]

[Wed Oct 04 12:04:26 2006]  [jk_ajp_common.c (532)]: ajp_unmarshal_response: Header[4] [Cache-control] = [must-revalidate]

[Wed Oct 04 12:04:26 2006]  [jk_ajp_common.c (532)]: ajp_unmarshal_response: Header[5] [set-Cookie] = [fmi-cookie=fmi-cookie; Path=/; Version=1]

[Wed Oct 04 12:04:26 2006]  [jk_ajp_common.c (532)]: ajp_unmarshal_response: Header[6] [Connection] = [Close]

[Wed Oct 04 12:04:26 2006]  [jk_ajp_common.c (532)]: ajp_unmarshal_response: Header[7] [Content-Type] = [text/plain; charset=utf-8]

[Wed Oct 04 12:04:26 2006]  [jk_ajp_common.c (532)]: ajp_unmarshal_response: Header[8] [Expires] = [Tue, 03 May 1988 14:40:00 GMT]

[Wed Oct 04 12:04:26 2006]  [jk_isapi_plugin.c (432)]: Into jk_ws_service_t::start_response

[Wed Oct 04 12:04:26 2006]  [jk_ajp_common.c (729)]: received from ajp13 #1027

[Wed Oct 04 12:04:26 2006]  [jk_isapi_plugin.c (566)]: Into jk_ws_service_t::write

[Wed Oct 04 12:04:26 2006]  [jk_ajp_common.c (729)]: received from ajp13 #744

[Wed Oct 04 12:04:26 2006]  [jk_isapi_plugin.c (566)]: Into jk_ws_service_t::write

[Wed Oct 04 12:04:26 2006]  [jk_ajp_common.c (729)]: received from ajp13 #2

[Wed Oct 04 12:04:26 2006]  [jk_isapi_plugin.c (925)]: HttpExtensionProc service() returned OK

[Wed Oct 04 12:04:26 2006]  [jk_ajp_common.c (1382)]: Into jk_endpoint_t::done, recycling connection



[Wed Oct 04 12:05:35 2006]  [jk_isapi_plugin.c (696)]: HttpFilterProc started

[Wed Oct 04 12:05:35 2006]  [jk_isapi_plugin.c (759)]: In HttpFilterProc Virtual Host redirection of /localhost/fmi/xml/fmresultset.xml

[Wed Oct 04 12:05:35 2006]  [jk_uri_worker_map.c (460)]: Into jk_uri_worker_map_t::map_uri_to_worker

[Wed Oct 04 12:05:35 2006]  [jk_uri_worker_map.c (477)]: Attempting to map URI '/localhost/fmi/xml/fmresultset.xml'

[Wed Oct 04 12:05:35 2006]  [jk_uri_worker_map.c (599)]: jk_uri_worker_map_t::map_uri_to_worker, done without a match

[Wed Oct 04 12:05:35 2006]  [jk_isapi_plugin.c (765)]: In HttpFilterProc test Default redirection of /fmi/xml/fmresultset.xml

[Wed Oct 04 12:05:35 2006]  [jk_uri_worker_map.c (460)]: Into jk_uri_worker_map_t::map_uri_to_worker

[Wed Oct 04 12:05:35 2006]  [jk_uri_worker_map.c (477)]: Attempting to map URI '/fmi/xml/fmresultset.xml'

[Wed Oct 04 12:05:35 2006]  [jk_uri_worker_map.c (502)]: jk_uri_worker_map_t::map_uri_to_worker, Found a context match wpc -> /fmi/xml/

[Wed Oct 04 12:05:35 2006]  [jk_isapi_plugin.c (775)]: HttpFilterProc [/fmi/xml/fmresultset.xml] is a servlet url - should redirect to wpc

[Wed Oct 04 12:05:35 2006]  [jk_isapi_plugin.c (838)]: HttpFilterProc check if [/fmi/xml/fmresultset.xml] is points to the web-inf directory

[Wed Oct 04 12:05:35 2006]  [jk_isapi_plugin.c (878)]: HttpExtensionProc started

[Wed Oct 04 12:05:35 2006]  [jk_worker.c (132)]: Into wc_get_worker_for_name wpc

[Wed Oct 04 12:05:35 2006]  [jk_worker.c (136)]: wc_get_worker_for_name, done  found a worker

[Wed Oct 04 12:05:35 2006]  [jk_isapi_plugin.c (913)]: HttpExtensionProc got a worker for name wpc

[Wed Oct 04 12:05:35 2006]  [jk_ajp_common.c (1404)]: Into jk_worker_t::get_endpoint

[Wed Oct 04 12:05:35 2006]  [jk_ajp_common.c (1448)]: In jk_endpoint_t::ajp_get_endpoint, time elapsed since last request = 73 seconds

[Wed Oct 04 12:05:35 2006]  [jk_ajp_common.c (1116)]: Into jk_endpoint_t::service

[Wed Oct 04 12:05:35 2006]  [jk_ajp_common.c (295)]: Into ajp_marshal_into_msgb

[Wed Oct 04 12:05:35 2006]  [jk_ajp_common.c (432)]: ajp_marshal_into_msgb - Done

[Wed Oct 04 12:05:35 2006]  [jk_ajp_common.c (642)]: sending to ajp13 #336

[Wed Oct 04 12:05:35 2006]  [jk_ajp_common.c (884)]: ajp_send_request 2: request body to send 0 - request body to resend 0

[Wed Oct 04 12:05:45 2006]  [jk_ajp_common.c (729)]: received from ajp13 #319

[Wed Oct 04 12:05:45 2006]  [jk_ajp_common.c (483)]: ajp_unmarshal_response: status = 200

[Wed Oct 04 12:05:45 2006]  [jk_ajp_common.c (488)]: ajp_unmarshal_response: Number of headers is = 9

[Wed Oct 04 12:05:45 2006]  [jk_ajp_common.c (532)]: ajp_unmarshal_response: Header[0] [Date] = [Wed, 04 Oct 2006 04:05:35 GMT]

[Wed Oct 04 12:05:45 2006]  [jk_ajp_common.c (532)]: ajp_unmarshal_response: Header[1] [server] = [localhost]

[Wed Oct 04 12:05:45 2006]  [jk_ajp_common.c (532)]: ajp_unmarshal_response: Header[2] [MIME-Version] = [1.0]

[Wed Oct 04 12:05:45 2006]  [jk_ajp_common.c (532)]: ajp_unmarshal_response: Header[3] [Cache-control] = [no-cache="set-cookie"]

[Wed Oct 04 12:05:45 2006]  [jk_ajp_common.c (532)]: ajp_unmarshal_response: Header[4] [Cache-control] = [must-revalidate]

[Wed Oct 04 12:05:45 2006]  [jk_ajp_common.c (532)]: ajp_unmarshal_response: Header[5] [set-Cookie] = [fmi-cookie=fmi-cookie; Path=/; Version=1]

[Wed Oct 04 12:05:45 2006]  [jk_ajp_common.c (532)]: ajp_unmarshal_response: Header[6] [Connection] = [Close]

[Wed Oct 04 12:05:45 2006]  [jk_ajp_common.c (532)]: ajp_unmarshal_response: Header[7] [Content-Type] = [text/xml; charset=utf-8]

[Wed Oct 04 12:05:45 2006]  [jk_ajp_common.c (532)]: ajp_unmarshal_response: Header[8] [Expires] = [Tue, 03 May 1988 14:40:00 GMT]

[Wed Oct 04 12:05:45 2006]  [jk_isapi_plugin.c (432)]: Into jk_ws_service_t::start_response

[Wed Oct 04 12:05:45 2006]  [jk_ajp_common.c (729)]: received from ajp13 #1027

[Wed Oct 04 12:05:45 2006]  [jk_isapi_plugin.c (566)]: Into jk_ws_service_t::write

[Wed Oct 04 12:05:45 2006]  [jk_ajp_common.c (729)]: received from ajp13 #1027

[Wed Oct 04 12:05:45 2006]  [jk_isapi_plugin.c (566)]: Into jk_ws_service_t::write

[Wed Oct 04 12:05:45 2006]  [jk_ajp_common.c (729)]: received from ajp13 #1027

[Wed Oct 04 12:05:45 2006]  [jk_isapi_plugin.c (566)]: Into jk_ws_service_t::write

[Wed Oct 04 12:05:45 2006]  [jk_ajp_common.c (729)]: received from ajp13 #736

[Wed Oct 04 12:05:45 2006]  [jk_isapi_plugin.c (566)]: Into jk_ws_service_t::write

[Wed Oct 04 12:05:45 2006]  [jk_ajp_common.c (729)]: received from ajp13 #2

[Wed Oct 04 12:05:45 2006]  [jk_isapi_plugin.c (925)]: HttpExtensionProc service() returned OK

[Wed Oct 04 12:05:45 2006]  [jk_ajp_common.c (1382)]: Into jk_endpoint_t::done, recycling connection

Edited October 5, 2006 by Guest

[Martin Brändle]

Since you have so detailed reports, I suggest that you send them to http://www.filemaker.com/company/problems.html

Another comment: I would never ever enable the fmxml privilege with guest access, but assign an account and password to it. With a little knowledge of how to write XML queries, I and others are able to read out the complete database structure and all the data of your databases that are CWP enabled.

[seancwr]

Hi Martin, thanks for your help, I have submitted a problem report with FileMaker and will post any updates here .

As for security...thanks for the tip. (We do already have it set up to not allow anonymous access, it just doesn't show in the trace logs!)

I was considering restricting access to IIS on the Filemaker server to the webserver that serves the site - would this prevent unauthorised access from FileMaker experts, or is there another backdoor to the Filemaker fmxml data that we need to close?

Cheers,

Sean

[Mr F]

I am experiencing this exact problem. We found that as soon as the load increases on the FMSA server, the response times to queries go up MASSIVELY. ie: a 10% increase in load causes a ~580% drop in performance.

We have extensive logging and graphing of CPU utilization/load/etc vs query times, and the end result is that if ANYTHING significant is happening on the FM server, the WPE basically hangs.

This has been replicated and reported to FileMaker (Australia in my case) and has been escalated to engineering in the US.

No response as yet....

[Steven H. Blackwell]

What amount of RAM do you fellows who are having this problem have installed on your Servers?

Steven

[Martin Brändle]

Steven,

I don't think this is a problem of RAM. The WPE has huge problems if several queries which result in large answer sets come in at a time, although it should not and according to the specifications should be able to serve 100 concurrent connections. We are now able to reproduce this behavior, and will send in a problem report soon.

[Wim Decorte]

Not making excuses for FMI here because I do believe we need to press FMI for better performance.

But keep in mind that the WPE really behaves like a copy of FMP. So to maximize performance we really should look hard at the specs on the machine that runs WPE and on the connections (bandwidth) between the machine running WPE and the machines running the Web server and FMS.

Having said that, only by collecting good monitoring data and sending that to FMI will we be able to convince them to go where we want to go.

[seancwr]

Just an update...no response from FileMaker so far.

We have 2 gigabytes of ram on our server, 3.2Ghz Xeon processor, and it is dedicated to being a FileMaker server. IIS is installed with the tomcat installation that FileMaker WPE creates, but it doesn't do anything else...all requests come from a different webserver on our network which passes the FileMaker query in an HTTP request to IIS and expects an XML document back as a result.

I still haven't been able to diagnose the problem, but there is no doubt in my mind that this is not about the specs of the machine that it is installed on, unless FileMaker has unpublished or unidentified compatibility issues.

The trace logs on the FileMaker machine show a ten second wait at ISAPI level, and the Tomcat logs show a ten second delay at the AJP request level. At this level the only communication that is going on is tomcat<->filemaker, both of which are FileMaker installations. Whether the problem is with the IWP Tomcat setup or FileMaker itself I don't know, but the problem is definitely in communications between sofware that is installed as part of FileMaker and FileMaker IWP.

One thing that I have considered is to install Apache on the server with a tomcat connector and use that to connect to FileMaker. FileMaker don't support this so I don't even know if it's possible or what configuration will be necessary to make it work...before I reinvent the wheel, has anybody else tried Apache/Tomcat/FileMaker combination on a windows machine?

Cheers,

Sean

Edited October 16, 2006 by Guest

[Wim Decorte]

Apache on Windows won't work at all. Good info on the trace logs! That's the kind of info we need to send to FMI and let them explain it. And work on it.

[Mr F]

Our machines (DB server & WPE) both have 8GB of RAM, of which only 4GB is addressable by windows (hah!).

They are both quad CPU 3.2Ghz Xeons with GigE between them.

They (IMHO) should be well and truly massively overspec'ed for what we are attempting to do.

Interestingly, once the WPE slows down, the size of the resultsets becomes irrelevent. You can execute a query that results in a single integer being returned, and the performance is still absolutely atrocious (ie: 2-3 minutes).

This is a very real and severe issue.

We have had no response from FileMaker (US) yet either.

[Martin Brändle]

Wim,

I used an understatement. The WPE can be reproducibly crashed with our procedure (only 5-10 bigger queries in a fast series). Hardware in our setup is state of the art. E.g., our servers are on a 1 Gigabit ethernet, cable connection between the FMS and the WPE server is 40 cm and they communicate between each other with dedicated NICs.

Edited October 16, 2006 by Guest

[seancwr]

An update for those experiencing this problem..

I decided to try reinstalling to rule out faulty windows installation...

I created a temporary server to transfer our databases to while wiping the server, with a clean windows install, clean filemaker install, and no other sofware at all. Got everything hooked up to this and it was quite snappy. This machine is a P4 desktop with 1Gb of ram.

I wiped the server, and did exactly the same thing, clean windows install, clean filemaker install, no other software at all. Hooked everything back up to this, and guess what? Ten second queries.

What's different I wonder? The server is on the domain but the desktop isn't. I add the desktop to our domain, no apparent problems. Server has the v4 updates installed, desktop doesnt. I ugrade the desktop FM server to v4, no apparent problems. I upgrade the desktop WPE to v4 and it seems OK, but I start getting 10 second queries.

I didn't test thoroughly enough to be certain that there were no problems between the updates, but I'm fairly certain the problem started with the WPE update. I'm going to try an installation without applying the WPE update. Will keep you posted on how it goes...

[Mr F]

This is interesting stuff (nice work by the way)!

I'll be very interested in the results you get here.

That said, are the performance issues you experience easy to replicate just by increasing load slightly (ie: submitting multiple queries at once)?

The reason I ask is that my setup is fairly snappy in a test environment (and for a portion of the time in production, for that matter), it just all falls apart horribly when load increases even slightly.

Cheers again for the efforts.

[seancwr]

We didn't even get to the point of testing under load! But it seemed to happen after a number of requests rather than straight away, so I wonder if perhaps there is something unhealthy going on in the way Tomcat is trying to reuse its workers, which would mean load would cause the problem too.

Set everything up again on the server this morning with no V4 updates, and it's fine so far. Too early to be confident about it being fixed, but its looking hopeful.

[Steven H. Blackwell]

Do you have an official Tech Support case with a case number open at FMI?

Regarding RAM, as discussed earlier in this thread, the reason I asked that question originally, is that your symptoms are very similar, although not identical, to an earlier issue with IWP under load. That issue was addressed by adding RAM. Now, you are not running IWP, and now you note 2 GB of installed RAM. So, let's see what happens. But please do keep us posted.

Steven

[seancwr]

Hi Steven,

No we don't have a support case at FMI. I've filed a problem report, but we don't have any sort of support contract with them - 99% of problems I can sort out without any help, and the 1% are the impossible things like this where no one knows what is going on!

As far as the setup is going... the configuration is exactly the same is it was when we were having trouble, IWP is still enabled with the same four databases accessible, the same seven are used for XML access vie the WPE, other databases on the server are used (sparingly) by client machines for office management tasks. There has always been 2GB of ram. The only thing different is we haven't applied the v4 updates, and it is still working fine.

Monitoring TDI traffic to look at the tomcat/filemaker communication causes FileMaker to have all sorts of problems, which is making me think that there is something seriously wrong with the way Tomcat gets data from FileMaker at a very low level. If I was working at FileMaker, it be starting to look very hard at what was changed between v1 and v4.

Cheers,

Sean

[seancwr]

Hi Mr F,

I don't know how much trouble it would be for you, but a reinstall with no V4 updates to FileMAker server or the WPE does seem to have fixed the problem so far. If you want to try it, it might help to know that tried on the test machine doing an uninstall and reinstall of just the WPE and it didn't fix the problem.

If you aren;t looking urgently for things to try, I'll keep an eye on it for the next week and post back whether it has had any problems.

Cheers,

Sean

[Steven H. Blackwell]

If I was working at FileMaker, it be starting to look very hard at what was changed between v1 and v4.

That's why I am trying to find out if you have a Tech Support case number. It might be a good idea to open a Tech Support case here.

Steven

[Mr F]

I have a tech support case open and have spoken to the head engineer at FileMaker Asia-Pacific.

So far, they don't know what causes this, but it's been escalated to US Engineering.

I am currently working through that process with them.

I'll keep you guys updated as to how I go, but so far, the progress has been quite slow. The indication/feeling I got from speaking with them (the Asia-Pac engineering were very helpful/thorough) is that it could take a long time to solve (ie: many months).

Any info that we're able to work out independently would be greatly appreciated.

I'll keep you guys updated as to how I go.

[seancwr]

I don't know how to open a Tech Support case from here, but if someone points me in the right directions I am happy to do so.

Its still running with no problems. Load testing will come next...

Cheers,

Sean

[Steven H. Blackwell]

It's either 800-325-2747 or 408-727-9004, I don't rememebr which. I believe there is a card in your installation information with more details.

Steven

[seancwr]

Steven, I don't know if it works differently here (Australia) but we get one tech support case for free, which we used trying to solve this issue. I've filed a problem report, to which they responded with a "can you tell use more information about your installation" email, but I'm not going to pay for the privilege of helping them troubleshoot their software!

Sean

[Steven H. Blackwell]

I think that's the way it works in the US as well. I am usually on the receiving end of Tech Support calls, so I am not 100% sure. I had originally asked you for your Case Number so I could make some inquiries towards solving your issue.

Best of luck.

Steven

[angelleye]

I've got some questions of my own about the performance of CWP. I've got some pages setup to display products from our database. One set uses the XML API in FM and the other set uses ODBC. ODBC wins the speed battle more often than not, which was surprising to me.

In any case, with both methods I notice that when a query hits the CPU on the server jumps to about 25% - 35% for a second or 2 and then goes back to normal. Is this to be expected..?? I'm the only one hitting the database that way right now and if it's jumping to 30% with just me doing a single query I can't imagine it would be able to handle web traffic...would it?

If I have 100+ on there at any given time all browsing through product pages..well I don't know that I'd even get 100+ people on there like this.

Will it be ok? Is that a normal spike? Any information would be greatly appreciated. Thanks!

[Wim Decorte]

That's a question with no easy answer: you need to do baselining for that. Hit it with 5-10-20-50-100 whatever requests and monitor all of the system. See what parts fluke out and how you can counter it. A lot depends on how your system is built. Another big part is that FMSA is not a speed daemon. But it's only with proper monitoring and then working with FMI that we can get them convinced to do something about it.

For you personally: you do *not* want to commit to going live without having tested this and found the proper hardware combination that will give you the best result.

[Mr F]

Hey DrewAngell,

This sort of CPU spike is the sort of performance issues we've seen (see previous posts by myself).

In answer to your question - from our experience, the XML interface does NOT scale well, particularly with parallelizing requests.

We found we had to implement write-back caching and write all data syncronously just to keep performance even slightly vaguely acceptable.

Combining this with very heavy caching of reads has been the only way we've been able to maintain stable (if fairly awful) performance.

Keep in mind, this is running on two very high end quad-CPU xeons with 4GB RAM each and you can barely maintain a few simultaneous requests.

We have a support process happening with FM, and I'll post my results/experiences as I move forwards.

Cheers!

Mr F.

[angelleye]

Would it be feasible to create one giant XML that contains all of the product data and then just parse from that? Then I could just run a script to generate that XML for me however often and whenever it can stand it. Wouldn't be completely real-time for displaying current products, but then I could probably get away will running inserts and updates directly since there won't be near as many.

Does that sound like something that could work? Or, maybe there's a way one could create that giant XML that contains everything, and then sync that XML with the FM server so that it would receive small updates in real-time...: Pulling from a static XML would likely be faster and take less server load since it's not actually querying everytime.

I'm gonna be spending some time on experts-exchange.com trying to figure this one out. : Anybody care to join the project?

Or...even better...maybe FM will fix the problem..??

[Mr F]

Hi there,

If you have no major requirement for data to be realtime, then you'd be better off doing a batch export of data from FileMaker to flat XML (then into a MySQL DB or somesuch), or to propegate it via ODBC.

Alternately, FX.php does contain a caching module (PHPCACHE which is fairly undocumented), which you could very possibly use to get around most problems.

Unfortunately, for our application, a lot of the data MUST be realtime, which is what causes all our problems.

You might be OK, but I certainly wouldn't attempt to run any sort of website that relied on FMP queries via XML to serve pages to give any sort of reasonable performance.

Cheers,

Mr F.

[angelleye]

Well, I've tried the same pages using ODBC and I get the same 25% - 35% spike on the CPU for those queries as well. Should that not be happening?

[angelleye]

Do you have the Web Publishing Engine installed on it's own server beside your FM Server or do you have both installed on the same machine? According to this FM book I have you'll gain performance by putting the WPE on it's own dedicated server instead of being on the same machine as FM Server.

Have you tried that?

[angelleye]

I went ahead and created a script that generates static xml's for each of my product categories and then I can pull from those separately.

I'm actually using CWP to generate these XML's, so my CPU takes about a 50% hit for a good 2 min. or so while it's generating these XML's. It creates the new file to a temp directory and then moves it to the live location overwriting the previous so hopefully I won't have too much of an issue with people trying to browse while the file is locked.

We'll have to play around and see how often we can run the updates.

It'd sure be nice to be able to use the XML API the way its intended. As it stands now I don't know what I get from Advanced Server..??

[Martin Brändle]

DrewAngell, can you first describe your hardware setup?

[angelleye]

AMD Opteron 254 2.81GHz w/ 2GB RAM. I saw somebody else that mentioned they're running dual Xeon's though, and 4GB RAM and they have the same problems. Our hard disk is a SCSI 15k rpm.

[seancwr]

Well, its been a while now and the database is still holding up without the v4 updates and without the long query times. I don't know if it will fix anyone elses problems, but it seems to have fixed mine.

Though we have a slightly different problem now...sometimes it will just stop sorting results by fields passed to it and sort by record id instead, so our "recent news" suddenly starts being the news from 2001. Ho hum.

Cheers,

Sean

[Steven H. Blackwell]

Though we have a slightly different problem now...sometimes it will just stop sorting results by fields passed to it and sort by record id instead, so our "recent news" suddenly starts being the news from 2001. Ho hum.

I believe this is a known issue. I will check to see something about this and report back.

Steven