Speeding up update matching import

[Hiroshi]

Hello,

I was wondering if someone can help me with an import I do daily.

I currently import records from a web page (that I've been given access to as a courtesy). What I've been doing is "Select All" on the web page and paste this into a text file which then turns into a tab-separated text file. All I had to do then was set the "First row is a header row" option and import using the "Update Matching (add new)" option.

This has worked fine until the number of records started to grow. It seems to take almost a minute per 100 records (I use FMP8 on a 667MHz G4 PB).

I don't have the "perform auto-enter options during import" option selected.

I've turned indexing off for all of the fields I import to, but I do have calcs that are based on some of the imported fields.

What I wanted ask are:

a) How can I speed up the "update matching" import process?

What is the best way to get the data from the web page without bothering the owner? Is there some kind of plug-in that I can use to make FMP go read the page?

I have some scripting experience. Thank you in advance for your help.

Cheers,

Hiroshi

[xochi]

I have run into some situations where "import / update matching records" is horrendously slow. For example, I had two 25000+ record tables and I needed to update one field in table1 from macthing records / values in table2. It was crawling along at about 1-2 records per second!

As a work-around, I imported the 2nd table in its entirety, then set up a relationship between table 1 and table 2 based on the match fields, then did a "replace field contents" on the proper field in table 1. It was 100x faster.

An advantage of this method is that you can also make sure the matching is working -- create a layout with the fields from table 1, and the related field from table 2. You can then quickly eyeball the data to make sure records that should match do indeed match.

A downside is that this requires a lot more work the first time. However, if you are doing this update more than once, having it scripted makes it much faster the 2nd+ times.