Record creation performance

[Layton Duncan]

This really needs to be addressed.

Coming from a Server Advanced perspective, record creation is extremely slow. (Like around 20-30 records per second) on our xServe clusternode / xRAID configuration.

The unusual thing is that if you setup your records as a text file, then import it from a filemaker client application, we get performance of almost 1,000 records per second.

BUT, with Server Advanced 8 there is no web context for importing files, like there was under FM Pro Unlimited.

Why does individual record creation take so long?

Layton

[xochi]

Things to check:

1. Cache sizes (on both server & client). I've seen really slow performance when the index(es) of the file you are importing into are larger than the client cache.

2. Auto-enter calc fields, and indexed fields -- these slow it down a lot. Consider changing these to unstored calcs?

3. network performance -- any problems?

[Layton Duncan]

Nope, None of the above apply to the testing and results above.

The testing I did to get those numbers involed a new database, one table, with one text field.

Using fxphp classes to interface with server advanced. (fxphp is not a bottle neck)

Gigabit ethernet (certified cabling) fully switched.

Record creation performance is just plain pathetic. I don't know how it can be so slow. I've verified this with other (Mac) hardware configs, all show (lack of) performance similar to this.

[Søren Dyhr]

What if you turn you import source into a filemaker file and reference it in the filereferences, perhaps deleting the old references before attempting??

--sd

[Layton Duncan]

The real speed problem isn't importing (you can't import with a web based solution, as there is no filesystem context in FMSA 8 like there was in FM Unlimited).

The problem is individual record creation, (which is the only way to create records using a web interface)

The issue i've had is with an online ordering system. Basically when an order is despatched, a new record needs to be created for each item. Sometimes there are up to 100 records that need to be created for an order. With other load on the database this can take up to 60 to 100 seconds.

I've been extremely careful to try and optimise the layout, looking for all the usual sources of slowdowns. But then, even my "best case" testing results above, gives woeful performance.

The performance issue is real, and I think is something which needs to be looked at.

[Søren Dyhr]

The problem is individual record creation, (which is the only way to create records using a web interface)

Nah! I didn't see that - I appologise!

But the company/client behind this ordering system isn't a broker of arbitrary items from whatever source. The problem is usually solved by having no separate jointable but instead a multiline keyfield, although the lookups might prove a slight problem here.

Setting values in a multiline keyfield is blinding fast compared to generation of full records. But some kind of workaround needs indeed to be invisioned here ...I will think a little further, to see if I can come up with something.

--sd

[xoomaster]

I have had similar problem in the past, and found out that the problem was one of the "Relationships" . For whatever reason it would take up to almost a minute to creat a record, but after I decided to remove one of the relationships ( I may add relation to an external file ) the problem resolved !

Also try the "optimizing your File".

Xoomaster

[Layton Duncan]

Did you guys read this:

Nope, None of the above apply to the testing and results above.

The testing I did to get those numbers involed a new database, one table, with one text field.

Using fxphp classes to interface with server advanced. (fxphp is not a bottle neck)

Gigabit ethernet (certified cabling) fully switched.

Basically as minimal file as you can possibly get (so minimal it's useless) and record creation performance is still astoundingly poor.

[Søren Dyhr]

But if the ordering is one of each, could the jointable be eliminated - the writing of ID's in a textfield does largely exceed the creation of records, we have found that a combination of multicriteria relations and CF'ing multiline keys for these relations, beats any kind of scripting or sole CF'ing when creating recuring events.

So when it comes to it is your questions scope too general, when you consider other approaches might make it up for the purchase items you wish to record. This might not be what you think you wish to, but someone once said filemaker is 20% straight programming and 80% workarounds. If you need speed should you turn to mySQL or Valentina when being on a macintosh, sacrificing the RAD for another say Runtime Revolution for dealing with interfaces.

--sd

[Layton Duncan]

I think the performance problem is more fundamental than you think.

I am currently migrating this particular solution to PostgreSQL.

Layton

[Søren Dyhr]

No/yes but isn't it just down to picking the right tool, for the right purpose ...the benfits from using filemaker is laying elsewhere. Utilize filemakers webfeatures where they suit, or perhaps start tinkering with the produced HTML ...all one size fit's all are compromises!

--sd

[Layton Duncan]

Well, there may be a bit of that in it. But for the record, i've created a few other FM web solutions which operate perfectly fine performance wise. But this solution has some different requirements for which there are currently no acceptable work arounds for. (Basically it needs to create up to a couple of hundred records at the click of the button)

But this is the "Wants and Wishes" section. So my wish is that individual record creation performance be significantly improved from the 20-30 records per second I've clocked on a dp xServe G5 xServe RAID combo. Which by ANY standard is abysmal.

OR alternatively provide a file system context for FM Server Advanced 8, so one can take advantage of the record creation performance of importing text files (clocked at over 1000 records per second), like you could in the days of FM Unlimited.

[comment]

Why don't you import from a Filemaker file (as shown here)?

[Layton Duncan]

Like i've said, because this is a web driven solution, and there is no longer any filesystem context for importing any sort of file in FMSA, this potential workaround is not possible.

It did work under the old FM web architecture using FM Unlimited. But it is no longer possible to trigger an import of a file from the web under FMSA 7/8.

[Søren Dyhr]

Yes but I seem to remember a plugin which apparently, can revitalize the grayed outs ...do anyone remember this??

But then is there a robot... a client version logged in at all times that either makes these imports between tables with timed intervals or get activated by tool like this:

http://www.troi.com/software/activatorplugin.html

I have similar heard of cron taking looks at the logfile, and acting upon the gathered details, while java virtual machines unfortunately hog all availiable processing power availiable, not willing to give it back later.

--sd

Edited December 10, 2006 by Guest