2GB Limit

[nelliott]

I have a database with 150,000+ resumes/cvs on and it is fast approaching the 2GB limit. Ideally I don't want to split the records into two separate databases as it will make searching cumbersome and slow. Any ideas on how I can reduce the amount of memory that is currently being used?

Thanks

Nick

[Anatoli]

RE: as it will make searching cumbersome and slow.

WHY?

Good relational design is always faster.

BTW, database with 2+ million records has only over 700MB

Redesign is due soon. Just ask question here

[Charles Delfs]

Find out what field(s) are you space hogs and just relate them, this way there is minor changes. Usually there is only one field that will take up 80% or the space eg: a container with a picture etc.

[RodinBangkok]

One issue you may have is container fields with imported rather than referenced information. For instance if you import a photo versus referencing the photo's location during import this can cause problems with size of files. If you have only 150K records with nothing but text then there is something going on that does not equate and justifys further investigation.....Basically seconding Anatoli's response, more details on whats in there are needed.

[nelliott]

I was assuming that it was taking up so much memory due to the size of the resume field which is indexed to allow faster searching and contains people's full resume (therefore a pretty big field)

I was also under the impression that if I had the resumes in a separate database then searching in 2 different databases at once would be really slow

Your help is appreciated

[Johds]

nelliott said:

I was assuming that it was taking up so much memory due to the size of the resume field which is indexed to allow faster searching and contains people's full resume (therefore a pretty big field)

I was also under the impression that if I had the resumes in a separate database then searching in 2 different databases at once would be really slow

Your help is appreciated

Make sure you have an unique ID field for the person. Export that ID field, and the field containing the Resume to a new Filemaker file.

Make a relation from the original Filemaker file, relating the unique ID field in the original Filemaker file, to the unique ID field in the new file.

On the layout where you search, change the Resume field from the original file to the related file.

Then delete the Resume field in the original file.

It's a 1-to-1 relation (One person to One resume) so you don't need any portal.

The search should be just as fast, and now you have 2 GB just for the resume, and 2 GB for the other data that you need.

Attached are a zip file containing two small FMP5 files with a quick'n'dirty example.

Best regards

Johds

resume.zip

[nelliott]

Hi there!

I tried what you said but unfortunately having the resume field in a separate database is causing the searches (which take long enough as it is!) to take 10 times as long (

Any other ideas?

Nick

[Anatoli]

It cannot be!

Or you are searching in wrong database. Are you searching in Resume database?

[Paolo]

maybe you have forgotten to index the resume field in the related file

[nelliott]

Resume field has been indexed. It took about 10 hours to complete!

How I understood it was to have the resume field located on the original database via a relationship ie. resumedatabase::resumefield and then perform the search on that field from within the original database.

If this is wrong, please help

Cheers

Nick

[Anatoli]

To have relation is good for decreasing size of files.

I would test the searches, but my guess is that search through portal will be in this case much slower. Search in Resume file.

HTW

[Johds]

Don't think it will be slower, as there is no reason to use a portal, based on the given information.

One resume pr record = 1-to-1 relationship.

No reason to do a scripted find in the resumefile.

Did some tests with the q'n'd examplefiles I posted earlier, where I imported 8000 textbased log files with various amounts of data ranging from 8 kb to 64 kb text (Perl is THE great tool to concatenate so many files into one csv file )

As long as the resumefile::resumefield was indexed, I didn't see any slowdowns in searching.

Just drop the resumefile::resumefield field on the search layout, and it should work fine IMHO..

YMMV,

Johds

[nelliott]

Johds,

This is exactly as I set it up but unfortunately it resulted in a much slower find

Nick

[Johds]

Something else must be going on..

Could you possible do some screendumps of the relation dialog, of the Search layout in layout mode, and without example data, so the fieldnames are shown and of the Define fields dialog in both the main and the resume database, and post them.

Also please indicate which platform you are running, which version of FMP, and if you are working locally on the files, or if they are hosted on a FMP server or via a client ?

Need more info to find out what is going on.

Just my 0.02