Puzzled by database size

[K1200]

Database size just doesn't add up!

I'm using the separation model with Filemaker and have a modest database.fp7 that totals about 2,650,000 bytes (after a Save As Compact). One of my tables has a container field that's used for PNGs and PDFs. Here are some measurements I've made:

When I place a single PNG of 187,211 bytes into a record, the database size jumps by 339,968. Adding two more images totaling 216,571 bumps the total size by another 425,984.

But the real killer was when I dragged/dropped a single-page PDF (allegedly only 25,068 bytes on disk) and the database size jumped by 4,567,040 bytes! Dropping the same PDF into a second record added another 4,562,944 !! With just a handful of images and documents, the database size ballooned to over 25,000,000 bytes.

Is FileMaker really that inefficient at storing images? If yes, then is the practical solution to always store only file references? And what's up with the PDFs? Are there published guidelines anywhere on these things? Is this a well-understood phenomena? I'm trying to make some capacity projections but feel I'm at a real loss for any valid basis.

Thanks for any help or advice.

[John Mark Osborne]

I believe the reason is that FileMaker stores the graphic in a Macintosh and Windows format for cross-platform compatibility. I'm not sure about the PDF since a PDF should already be cross-platform compatible.

[K1200]

Thanks for your response.

I guess that's understandable, although it would seem that a "native mode" option with a companion on-the-fly conversion could save hundreds of megabytes for applications like photo databases.

Regarding the PDFs, I would certainly appreciate someone testing a couple of documents to see if they get the kind of results I'm seeing. They're simply astounding to the point of making PDF storage just about impossible to support. It's as if the PDF is including Acrobat Reader with the document. Could that be?

[John Mark Osborne]

There used to be a "store compatible graphics" option in FileMaker 6 but so many people forgot to select it and had to go back and reimport graphics for cross-platform compatibility or didn't know it even existed that FMI made it automatic in FileMaker 7.

I tested your PDF scenario by creating a new file in 8.5 with a single container field. When I inserted a PDF document, the FileMaker file was increased by exactly the same amount as the PDF. I made to sure to use a PDF document containing text and graphics.

Then, I thought to myself, what if the PDF were inserted as a picture rather than a file? When I tried this, the file increased by double the PDF document. Maybe this explain what happened to you.

[K1200]

Thanks for the test results.

Unfortunately, my results are hugely greater. A 25Kbyte PDF increased my database by 4.5MEGAbytes. I won't be able to do additonal testing for a couple of weeks, but I'd certainly like to hear what others are seeing.

[pjdodd]

In order to keep the size of the database small, especially with PDFs in my experience, store the file as a reference i.e. DO NOT STORE WITHIN. And as your example shows inserting as a picture is a bad idea.

My database currently has roughly 2000 PDFs files all linked with thumbnails visible in FIlemaker. The size of each PDF varies from 20K to 2.5 MEG. The current size (after optimization) is 10.1 Meg (the size before varies from 11 to 15 meg). I checked and stripping out the link (well deleting the container field itself from the table, I got lazy here) only saves about 700K space.

I think part of the problem is that all PDFs files are not the same. For example, a PDF saved from Photoshop is different to a PDF saved from OS X Output option is different to PDF from Distiller (using any number of settings) is different from PDF saved out of Acrobat Professional is different from PDF saved using Adobe PDF plugin from Quark and so on. More recently I have witnessed the mess of a PDF saved out of InDesign and have begun to think the format has not become the safe standard it purports to be.

I hope I am not confusing you but making you aware to avoid the traps of thinking that there is only one kind of PDF and each one 100% ok. There is a great deal of labour saving and stress avoidance with PDF but it is not perfect.

Anyway, I have be unable to find out what exactly Filemaker does when a file is imported to a container field, specifically with a PDF. Perhaps this is dependent on what type of PDF it encounters, whether or not there is an thumbnail for it to use (all of the PDFs for my solution were from Distiller and have thumbnails enabled), maybe font information brings to bear no it. Who knows. I am unable to devote the necessary time to create PDF from various sources (on Windows and Mac) and test them on both platforms and so can't offer much illumination.

[K1200]

Thanks for the response.

I'll certainly look into the practicality of only storing file references where I am able to control the operation. But, unfortunately, I have instances where a container field must accept a file from a drag and drop operation with the user in control. Do you know of any way to "filter" a drag and drop so that only the file reference ends up in the field?

And just to confirm that it's not just my PC environment that's introducing the problem, I would sure like to hear if anyone else has observed the kind of 100-fold increase I've observed with simple PDFs.

[pjdodd]

Don't allow drag and drop. This might get up some developers noses because it is potentially a useful feature. However, and someone please confirm or refute this, drag and drop control cannot be controlled without an external plugin.

Garbage in, garbage out is the design prinicpal and drag and drop (as far i know) does not have any barriers as to what you drag in. In other words, for correct control of how data is entered into your solution, use scripts. That way you can perform other checks and balances and refuse say large files and other undesirables.

[bdonelson]

I agree with pjdodd

That way you can perform other checks and balances and refuse say large files and other undesirables

Not only are you not able to limit the size, but are you also open to any type of object being placed in the container?

What about the "Store only as reference to file" option on the "Insert File" Dialog. Can this be set automatically?

Edited November 30, 2006 by Guest
Second Question

[pjdodd]

First the easy question - after inserting something into a container field once the "store as a reference" is checked it will stay checked after that. Sadly I dont know of a way to have this option automatically set, or even removed/disabled. Would be nice for future versions of Filemaker.

Scripting an insertion in to a container fields means that with some script steps and calcuations you can deduce the file size very easily, as well as the full file name and therefore the extension. To do this though I've only found that you must allow the file to be inserted, then deduce path, then decide to commit records or clear the field offending file. On Macs applescript can do the donkey work PRIOR to insertion, as the scripting allows for easier and more accurate identification of file type.

If

Length ( container_field ) > 102400

Clear container_field

Else

Commit Records

End If

That will block any file greater than 100K.

Edited December 1, 2006 by Guest
code added.

[bdonelson]

Even that's better then nothing at all.

Thanks,

[K1200]

For anyone following or finding this thread, here's a link to a recent post from FileMaker on the subject:

Statement on Large PDFs

To a large degree, it explains what I and others have experienced. With respect to container fields, "groomed" PDFs might be usable, but ones of unknown origin are probably best avoided.

[Genx]

Lol, I managed to generate a PDF 238 MB in size that had two containers referencing 1MB images in them.