CPU Usage When Running Scripts

[RPM765]

I run a daily data import procedure that involves several calculation fields, sorting, data mapping, etc. on 65,000+ records. I have found that when Filemaker is doing these calculations (which take several minutes), it is only using 50% of the CPU. I deternined this by opening the Windows Task Manager and looking at the "Performance" tab. The CPU Usage History maintains a very flat line at 50% during this calculation time. In looking at "Processes" I have confirmed that FM is using only 48% to 50% of the CPU power, never any more.

Is it possible to tell Filemaker to use 100% of the CPU when it needs it? Other program routinely use 100% processor but FM never does.

I am assuming that the calculations would go a little faster if FM could use 100% CPU time.

[FruitFul]

My initial thought is to right click (within Task Manager) and 'set priority' to a higher level.

[RPM]

Thanks for the suggestion. I tried this and it didn't change anything.

[Søren Dyhr]

I run a daily data import procedure that involves several calculation fields, sorting, data mapping, etc. on 65,000+ records.

Could you tell us why such a task is necessary ... if you're getting the data from another application, should you perhaps seek another "vehicle" since filemaker isn't a database in the strictest sense:

http://www.proofgroup.com/articles/2006/jun/filemakery_part_i

The interface part is standing in the way for some real crunching! Could it really be so that Gerald M. Weinberg is right after all - it's always a people problem, not matter how technical it gets, such as picking the right tool for the wrong purpose or opposite...

--sd

[RPM]

The data is coming via FTP from a main frame application as fixed-field-length textual data. Within FM I parse the data, map it to fields and do some calculations on it for a daily report. I have written code in FM to format the reports so they can be easily read and quickly understood.

I may have given you the wrong impression by stating there are 65,000 records. While there are that many TOTAL records in the database now, the daily data import in question here involves only a hundred or so new records each day.

[Søren Dyhr]

And the import of those roughly takes how much time, perhaps it where we should address you problem from, you're probably using calc'fields or calc's in autoenters to truncate, my guess is that once imported could the trimming be much faster by utilizing:

http://www.filemaker.com/help/Script-Steps36.html

But perhaps you should use the command line interface of this:

http://www.softinterface.com/Convert-XLS/Features/Convert-Fixed-Width-Text-File-To-CSV.htm

...as preparation of the data before import, instead of letting autoenter options for the fields be in charge of the trimmings.

It is claimed that the conversion is 10-15 times faster with this tool than letting excel do the conversion, what the ratio might be with the craftiest filemaker methods applied, might land somewhere similar!

On mac's shouldn't OS X command line be ignored as an utterly fast method, when stream-editing:

http://www.student.northpark.edu/pemente/sed/sedfaq4.html

--sd