General ideas for archiving data/records

[BrentHedden]

I'm looking for some general ideas and suggestions here, rather than direct answers. Although those would be appreciated too

My company has had a customized FM solution running for several years now. It's been a policy to never delete/remove any data. But that has caught up to use with bloated filesizes and slow performance levels. Using SyncDeK, we've set up a second server that will not only mimic all of the records/changes of the production system, but will also hold all of these 'archived' records. We've setup rules to delete 'archive' records from the production system, but never from the archive system. We're not too concerned with the performance/size of this archive system, as it won't be used very often.

The big concern is updating the schema on the archive system. Since our business is in a constant state of change, we have to change/add procedures with the FileMaker solution all of the time. SyncDeK doesn't handle items like layout and script changes, so these updates have to be done manually. We want to make sure that users will be able to use all of the same features. It's not practical to make the same change in two different system. Neither is it to change to a data separation model, although that would make this problem a heck of a lot easier!

What I'm looking for is some ideas on how to keep this archive system updated. Since I'll have to manually do this update, it'll probably happen only once a month or even longer. To biggest concern besides reliability is the amount of time these updates will take. Considering the amount of records, I anticipate the update will take several hours, but I wanted to minimize the overall update time as much as possible. The records that are 'archived' have a specific field flag just for this purpose, so it's easy to tell what's what.

Ideally, the only difference between the two system will be these archive records, so my initial thought was this - Stop Archive system, export all of the 'archive' records into separate files, then import them into a recent backup from the production system. Take these new files and replace the ones currently on the archive server. The problem with this method is the amount of time it takes for the overall process.

If you've got any idea that will help with the overall time of this process, I'll certainly appreciate it. Sorry about the long explanation, but it seemed necessary to prevent future questions that may show up.

Thanks!

[Matthew F]

You might want to check out RefreshFM. http://www.goya.com.au

[dansmith65]

Have you checked out fmDataGuard? http://www.syncdek.com/fmDataGuard/

It may provide a way to approach your problem from a different angle. I don't think it does exactly what you're asking, but if you try to see how it might work for you, maybe you'll see that it's a viable alternative.

[BrentHedden]

I'm actually evaluating RefreshFM as a possibility. It's especially handy for vertical markets. You can certanly program the same thing as what this does, but the time and effort isn't worth it once you consider how inexpensive this software is. It's more designed to update a blank copy of a system, but from what I've seen so far there is no reason why it can't have records in it before updating, you just gotta be careful with duplicates.

Unless there is something that I'm missing, I don't see how fmDataGuard would help in this at all. We actually own an older license of this tool, and it's been great for what it's designed for, which is reliable and fast audit logging.