FM9: X.fp7 is damaged

[GalainHH]

Hello,

I have a big FMP-Solution with 30 files. I downloaded the FMP9 Trial Version to inspect my FMP-solution, after I read in another forum, that some users have problems to open some (not all) files that work flawlessly in FMP 8.5.

And for 3 of my 30 files, I got the following message:

"x.fp7" is damaged and cannot be opened. Use the recover command to recover this file.

These files work without any problem under FMP 8.5

When I recover the "damaged" files, the sumary shows 0 Errors, but I read several warnings [color:red]NOT TO WORK with recovered files.

So, what can I do?

- Sell my solution with FMP 8.5? But it won´t be long until the first customer buys FMP9 and run into this error.

- Throw away the "damaged" files and develop them completely new?? That is impossible

So, what is your opinion?

- Is this a problem of localized FMP9?

- Would you use recoverd files?

Torsten

[Blaze]

I have the exact same problem - only my recovery hangs up at step 6 of 16 and stays there for hours - never completely recovering.

Mine is a single file but it has 4.x gigs of data

Anybody with any ideas on this one?

Edited August 8, 2007 by Guest

[Fenton]

I was just going on a bit about Recover, in another post; it bears repeating:

Recover is not meant to be used on files you will continue using. It is meant to aggressively stablize the file so that you can get the data out; it can remove bits of the structure if it considers them suspect. In other words, it's the worst thing you can do if you suspect structural problems on a file you have no clean backup of.

It is especially inappropriate to use to try and fix a file to use in FileMaker 9. Since you can get the data by opening the file in FileMaker 8.5; they're the same file format.

Save a Copy as Compacted is really the only safe thing to do, and it seems to work in many cases of the "won't open in 9" files. If it doesn't work after that, then you're in trouble. You may want to try backup copies until you find one that works.

[Blaze]

Hey Fenton,

Agreed on the use of recovery - the issue is why is 9 treating larger files or solutions as needing repair.

[Fenton]

FileMaker Pro 9 has added, to the regular application, the more intense file tests for opening that used to only be run by FileMaker Server 8 (remember when people had problems then?). They are perhaps even a little pickier, as some people whose files were running fine on Server 8 also got messages. In most cases Save a Copy as Compacted will fix files that have only minor issues, like indexing. I don't really know the technical details beyond that.

[Sergeant Ron]

Just a thought. If you needed to rebuild this solution from scratch, is it safe to copy and paste the tables / fields from a damaged file, then import the layouts, rebuild the relationships, then import the scripts. Once thats done then import the data? Or would the same problems still be present since the tables were imported in from the damaged file?

Ron

[Vaughan]

"If you needed to rebuild this solution from scratch, is it safe to copy and paste the tables / fields from a damaged file, then import the layouts, rebuild the relationships, then import the scripts"

Reports have been made of corruption coming in to the new file from copying damaged layout objects and scripts.

[LaRetta]

I can confirm that in spades, Vaughan. I was wonderfully privileged to work on a system with inferior specs which crashed 18 times in two years. For fun (and to satiate my curiosity), I played with various attempts to recover them and pull objects and scripts from them. But I did not for a moment plan to ever use these files again - they were retired into my Weird-Wonky file.

Scripts may LOOK fine but they are not. They can eventually throw a message saying they don't exist but they DO and they DID work fine for awhile after the crash. Layout objects can begin to dance around, disappear at random and cause further crashing. And I have a few files where a layout number 285 says it's there but ... you can't get there from anywhere ... it exists only in the twilight zone and no combination of rescue work saves it (or any of these files).

I trusted those on FM Forums who told me; so I'm passing it along - NEVER EVER EVER trust a file which has crashed. If you do, and it later crashes for good on you, you'll have to trash ALL THE WORK you did between your first crash and the final one. It can cost you months and months of work. Bite the bullet and pull out your last good backup; better to lose a few hours (or days) of work than months.

I couldn't be more serious - notice I don't even have a smiley face.

LaRetta

[Vaughan]

Thanks for confirming that LaRetta.

[xochi]

You might want to try FMDiff (http://www.fmdiff.com/) on the original file, and then on the recovered file. Supposedly it will give you a pretty good report of what's different between the two, it may help you narrow down the issue...

[GalainHH]

I tested my files a while:

I have 1 "damaged" DATA file

and

3 "damaged" files that contain the scripts and layouts (I call them Layout-Files).

Most important to me is the DATA file for one reason:

The DATA File contains the Data of my customers. Whenever I release an update, the DATA file remains untouched. Only the "Layout-Files" are replaced by new ones.

[color:red]MY DATA File is considered to be "damaged" due to FM9. But [color:red]MY DATA File contains no import Data, only Demo Records.

I tried two things:

1) I stored a clone without DATA: It works perfectly, but some of you suggest not to use such a cloned file

2) I deleted 3000 Demo records in DATA file: The file works without problems in FM9

So I think: There are different "damages" that a file can have:

- bad Structure and/or bad Data

- only bad DATA

LaRetta spoke of [color:green]recovered Files, not [color:blue]cloned Files.

Here my questions:

1) Would you think, that [color:blue]cloned files can be used for further usages and development, when FM9 says, they are "good"?

2) Would you continue working with the DATA-File, that is "damaged" when containing my Demo DATA, but that is "good" when I simply delete the Demo Data?

[LaRetta]

Whoa there ... what we ALL are saying is that your files might be fine. FM9 is simply more sensitive and ALL files first opened in FM9 appear to produce scanning message and MANY report being damanged. If you have never even gotten a scanning message in prior version, I wouldn't worry until AFTER I've saved all files first as compacted clone.

Then if you open them in vs. 9 and they say they are damaged, THEN panic. What we are saying is that Recover aggressively repairs files even if they aren't broken but just have questionable spots. This aggressive behavior actually destroys a file and should only be used to recover data to move it to another good backup copy.

Again, take your good files and save each one as compacted clone, then rename them to their original names, then open them in vs. 9. Don't panic until vs. 9 again produces 'file damanged'. I wish FileMaker would have addressed this a bit more - you are not the only person worrying. We all, when first opening our 8.5 files in 9, received similar surprises.

LaRetta

[LaRetta]

Well, The FileMaker Pro 9 Consistency Check is a week old.

I notice it says to use either Save As Compacted (if in lesser versions) or RECOVWER in vs. 9!! Good grief! I've heard that vs. 9 has improved Recover but enough to trust it??? Yikes! Where is confirmation on this?

I hope the Developers at DevCon are getting answers for us. I'm reading on multiple forums that people are trashing their files or incorrectly handling these errors. Come on, FileMaker!!

LaRetta

[LaRetta]

If vs. 9 is supposed to handle errors better, it sure isn't documented that I can find. I've been crawling all over their website and tech info. Nowhere does it say that vs. 9 has improved the Recover function or handles errors any better.

I would not trust using Recover in vs. 9 (like it is indicated in that link I provided) without someone like Steven Blackwell saying so. I am quite disappointed in FileMaker for not making this (the consistency check and damaged file issue) clear to everyone who opens prior versions in vs. 9. You can't tell me that they weren't aware of the issue.

:soap:

[AudioFreak]

Laretta,

I agree with you 100%. Filemaker should have made people more aware of what to expect when upgrading. I also had files that didn't pass and YES I did Panic too.

From the link.

In FileMaker Pro 9 or FileMaker Pro 9 Advanced, Choose File menu > Recover and select the damaged file, and click Open - Recover is the next best option for less complicated solutions. Recover will fix many problems, however it can occasionally remove some damaged records/objects if necessary to get the file back in working order.

This is kinda confusing..lol

So now we can recover files and it's reusable???

[AudioFreak]

You can't tell me that they weren't aware of the issue.

I'm sure some of their converted files didn't pass either....lol

[Fenton]

some of you suggest not to use such a cloned file

No one has suggested that here, and I have never seen it said. You are the first person to use the word "clone" in this thread.

A clone is just the file without the data (or indexes). So if you get no error message from a clone, then import data and try again. I would not use the data from the file that got the error, unless you ran it through a rigorous test for weird characters. (I use BBEdit's Zap Gremlins... command.) If FileMaker 9 opens the file fine then you're good to go (for now).

It is almost impossible for we developers to say anything definitive about what Recover actually does. We are not FileMaker engineers. I think the idea of running FMDiff on the two files (and maybe some earlier backups) is a good one. At least it would give you some idea.

Just to blow everyone's mind, I did some searching in my archived messages, and found this, from Jimmy Jones, who worked on recovered files within FileMaker in earlier years; this is from Sept. 1998:

I don't know why you think you should never use a file that has been recovered but it ain't true. A recovered file is just as good as an unrecovered file. The only difference is a recovered file may have stuff that was damaged removed from a layout etc thereby making it less than complete, which is to be expected if it was damaged.

However, a file that has been recovered and validated for completeness is just as good as a file that has never been recovered.

The main thing this tells me is: we don't really know. Also, if it removed some damaged stuff, how would I know what exactly it was. It could be something critical to functionality which would go unnoticed until data was badly mangled.

But I do know that if FileMaker included a really serious tool to fix damaged files in FileMaker Pro Advanced I'd be willing to pay another $100 for it, in a second.

Apparently they have some of the means to do this, as they said it was possible to remove corruption when they converted a file from .fp5 to .fp7, rebuilding the structure. Why not include the ability to do this to "same format" files also? But would it be 100% effective? I think I'd still want my uncrashed master copy.

Edited August 9, 2007 by Guest

[LaRetta]

Uhm, Fenton? I don't recall saying this at all!

In response to LaRetta ...

Quote:some of you suggest not to use such a cloned file

No one has suggested that here, and I have never seen it said. You are the first person to use the word "clone" in this thread.

I know I used the phrase 'compacted clone' by accident (I meant Save As Compacted only). But that was my only reference. And I didn't imply that ANYONE said anything of the sort so I'm a bit confused.

Heck, Fenton, it was you who taught me to trash a questionable file and go back to good backups. Maybe that lesson was planted too deeply but what is the risk worth? As for Recover being safe to keep using a file? Not THIS girl! Never! Ever!

If Recover just ONCE told me what it did, then maybe. But I've seen the resulting trashed file and it never says it found a SINGLE error and never says it corrected anything (even while I'm staring at it's destruction)! I DON'T TRUST IT ONE BIT!! Should those portions of the solution been removed? Maybe; even probably. But without being told they happened, I'd rather not take the chance of discovering them later. You've taught me well, sir. :wink2:

[Fenton]

Oops, sorry. I am guilty of just hitting the Reply button at the end of the last message in the thread, rather than going back to the message I was actually responding to. It wasn't to you LaRetta. I know you would never say such a thing. It was in reply to the original poster, who did say it. And it would make a big difference in what action he should take now.

Also, the reason I included Jimmy Jones's quote was just to show that we developers do not have definitive answers about this. We can only offer anecdotal experiences. It is my personal opinion that corruption in FileMaker is a big problem, not widespread, but something they should focus on improving. I know of at least one large deployment (not mine fortunately) that is going through hell because of this.

But you, LaRetta, already know all about that.

[Lee Smith]

Hi Fenton,

Jimmy said that! That's incredible. It is the first time I have ever heard that a Recovered File is okay to use.

A recovered file is just as good as an unrecovered file. The only difference is a recovered file may have stuff that was damaged removed from a layout etc thereby making it less than complete, which is to be expected if it was damaged.

So the removal of damaged stuff isn't important to the future use of the file

However, a file that has been recovered and validated for completeness is just as good as a file that has never been recovered.

Minus maybe some missing parts, that could be important to you, such as data.

I guess in theory this would be okay. But my clients don't work in theory.

:soap:

Edited August 9, 2007 by Guest

[Fenton]

I guess we should consider that Jimmy said that in 1998, on a mailing list; perhaps he would now reconsider. But he said that AFTER he worked recovering files for FileMaker.

As you say, "missing parts"? What parts?!? And, and you also say, what is the impact on clients of missing structure? It could be fairly benign; oh, the field is no longer on the layout. Or it could be disastrous to the data, with no error or warning, possibly not even immediately noticed.

It would be much more likely that it would be structure which was removed rather than data. Recover is designed with a focus to restore access to data. For a good reason. If you've been good, and have a master clone, then all you really need is the data. I don't know that Recover ever removes data. I don't know what it does if it encounters bad data (unexpected control characters, etc.). But that's not really "data."

[LaRetta]

Hi Xochi!

Thank you for the information on FMDiff! OMG - how does it know so much??? I don't see where it says it'll work on vs. 9 ... I would assume it will but I suppose only Winfried Huslik could say for sure. I would LOVE to run comparative tests with Recover and FMDiff to see what is produced, missed or caught. Too bad it doesn't produce the FileMaker database for us as well - I want to take any two files - spit each DDR into a FileMaker file (or table) and be able to relate them on table/file and field name for comparisons. And yes, dear Santa, I want it all. I'm currently working on XML (and XSLT) to turn a DDR into an FM file . It's a labor of love for two reasons, 1) learn XML/XSLT and 2) be able to produce FM files from ANY solution. I can't justify purchasing a program such as FMDiff AND Inspector AND ... etc - just to get what I need.

Hi Fenton!

Oh, sorry I misinterpreted. I too would pay easily $100 for a good, dependable method of identifying corruption; either in the design OR in the data. Corruption in the data may seem simpler, but it can bring a served solution down in a heartbeat when script hits bad data in a field - like bringing down a house of cards! Both types of corruption are very serious. Heck, I'd pay $500 easily - no joke!! I envision a separate FM Fixer file ... cost of $500. A solution could be ran through it. It wouldn't need to increase the cost of EVERY client copy!

UPDATE! Heck, if I was good enough, I'd write such a program. And people could pay me whenever they wished to have their solution analyzed and they'd have the FileMaker file of the results. $$$$$$$$$$ ^)

LaRetta

Edited August 9, 2007 by Guest

[LaRetta]

Oh, ha ha, I wouldn't charge $500 - that's if FileMaker wanted to provide a solution without increasing FMPro's cost. I'd only charge $50 a whack or something small - maybe $25 each time. Because I'd just start it running and go hiking. It'd be (once built) ongoing residual income. Ha ha. ^)

[tgilders]

Hi LaRetta,

I just got back from Dev Con and I attended an "Under the Hood: Recovery and File Maintenance" session.

My take away was this:

Yes, FileMaker has added the consistency check that will run every time a file is opened in v.9.

The first line of defense is the "save as compacted" --- if 9 will let you.

They didn't say this part, but I would assume you could do this in 8.5 first, then try it in 9 to see if whatever is causing the error is now gone.

They did say that stored images or files in the database are particularly troublesome. Reason being is FileMaker can't look at the data and determine if it's corrupted or not. It can only find the start and stop of the blocks and hope everything in the middle is fine. These problems usually only surface once the database is open and attempting to access/display the image or file however. They said this can vary by platform, and even by graphics card on the machine.

They then went into detail about other things that fail (indexes, corrupt data maps, block counts, etc...), but knowing this isn’t of any particular help really because we still are only left with the option to run save a compacted copy or recover.

They did say you CAN run a recovered file IF YOU HAVE NO OTHER OPTION. Recover will delete certain data from the file EVERY time, no questions asked, which are things like found set data, printer settings, export mappings, etc.. (can’t remember the others). This is, however, only pertinent to the data from the last (local?) users setting. All settings stored in scripts ARE maintained (so long as it wasn't corrupt in the first place).

The preferred practice was to use a backup and compact it to hopefully remove whatever caused the failure in the first place (doesn’t pertain if it was a hardware crash of course). If the file is TRUELY damaged and must be recovered, do the recovery, then go to a backup and import new/changed data into the backup from the recovered file. Then save as compacted. Now, the catch here is the deleted records. How to find those and delete? -- They suggested creating a relationship between the two and compare. Then mark / delete based on found set (use a script or a find on == in a field).

Yes, this is a LOT of work, and I'm sure there are people thinking about how LONG this would take if it happens in the middle of the day (especially if you have millions of records within lots of tables and gigabytes of data like I do). FM's suggestion for this was put up the backups immediately and let people get back to work. Then do the WORK (yes, they said YOU will have to do some hard WORK to bring in the records from the recovered file. Then they said the obvious, make sure you have auto-entered timestamps in every table otherwise you’ll never know what had changed since the backup!

One other thing that was particularly interesting was the fact that they said the "Maintenance Tools" that come with FM Advanced are basically not helpful (Optimize & Compact), and they don't recommend using them. Why? First is because they can actually CAUSE corruption in your file because it works on the ACTIVE database (and hopefully not your only copy!), whereas save a copy and recover only READS the active database and then creates a brand new file with the changes.

Oh, one more thing I learned, we all know (hopefully!) not to backup the live databases, but don’t forget the virus-scanner! Make sure it’s not scanning the database. I did know that, but did you know about the temp folder? Don’t let the virus scanner or the backup operate on the temp folder! C:WindowsTemp – Don’t know what it is on Mac. They said that FM uses the temp actively, and if something disrupts it’s read/writes to the files it creates… boom, crash or corruption.

I’m hoping I can just exclude the files that FileMaker is using, because there’s an awful lot of potential mischief going on in the temp folder!

So, I’m thinking prevention. I don’t want to have to go through all this. I’m doing what I can to prevent the dreaded corrupted database. I’m even thinking it might be a good idea to have some scripts in there that can help look for changed records based on a timestamp I can input that I can then migrate if need be.

Sorry for the long post, hopefully it had some info that made it worth your while!

[Lee Smith]

It was not too long.

Thanks for the report. They covered some interesting points. I hope they are going to follow with a Tech note, or white paper soon.

Thanks again,

Lee

[LaRetta]

Thank you, tgilders!

I am dismayed that Compact and Optimize are still in vs. 9 then. It is good to have them finally state some of the things we Developers have known for awhile. But vs. 9 has been turned loose and how many copies have been sold? And there is NOTHING ON THEIR WEBSITE about these issues yet today!!

As Lee says, I hope something is published soon; however, vs. 9 has been publically released ALREADY and how many thousands of copies have been sold? This information should be public and not only presented at DevCon in ONE track. Truly, it makes me fuming mad.

Get your act together, FileMaker. Again, an issue which wastes thousands of Developer hours (worldwide) is being swept under the rug. The data corruption issue has been ONGOING FOR YEARS!! And people STILL aren't clear how to handle it. Shame on you, FM.

L

[mr_vodka]

Pretty good synopsis of that session which a few others of us attended as well...

Here is the breakdown of Recover vs Save as compacted...

ALWAYS USE SAVE A COPY AS COMPACTED FIRST. Only if there is absolutely no choice at all would you perform a recover and cross your fingers that you could retrieve all your data from it. Their example was, "if your plane was having failure, you could always use a parachute, but wouldnt you rather be able to land the plane instead".

With a recover, it will try and verify each block; if there is an inconsistency, it will ignore it. THEREFORE, a recovered file is UNRELIABLE for a 100% backup.

That is the reason why it is preached to use a fresh backup copy and import the recovered data if you have no other choice. If wise, you will have many backups, a good recovery plan, and spent the extra few bucks on good hardware; especially the hard drive (the single most important factor of your server).