Has Recover recovered in vs. 9?

[LaRetta]

The debate continues... If one needs to use the Recover command after a crash, to pull data from a solution for importing into a new solution, we have been told over and over that we should NEVER EVER use a file which has had Recover performed on it.

And now, this has been pointed out to me.

We are told that vs. 9 is MUCH BETTER at identifying problems within a solution because of its refined scanning; but does that mean a file is safe to continue using if it has been served and it crashes? What if we get NO scanning message on re-open ... is it THEN safe? Is it EVER safe to use a file which has been improperly closed? Is it ever safe to use a file which has had Recover performed on it?

[aldipalo]

I have had this happen several times and was unaware that a recovered file should not be used again. I thought the whole idea of recovering a file was so you could use it again and not have to go to a back up that may be a few days old? I just had that happen this morning and recovered my development file and have been merrily continuing on my way. Are you saying I should have immediately scraped the file and gone back to the last backup?

I guess I'd better set up a better backup system, like several times a day!

Al

[Quartekoen]

I wouldn't trust working on a file that's has recovery performed on it. It might actually be just fine, but I think this is a fine example of "better safe than sorry."

As for backup frequency, the company I work for has data that's constantly changing, and a daily backup could be potentially devastating. We backup our data every hour. Daily backups might be okay for some situations, but if you're worried about it, I'd say backup more :

[Colin Keefe]

Are you saying I should have immediately scraped the file and gone back to the last backup?

She's saying that in the past Recovered files have been viewed as potentially unstable. Basically the Recover process uses aggressive methods to extract as much data as it can, privileging data over any design elements in the file. It'll drop field calculations, layout elements, layouts, table occurences etc.

Here's an explanation of the practices to date:

Knowledgebase article

Most favor the conservative approach outlined in the linked article 1580.

As to what's up with the knowledge base article you posted LaRetta, I'm curious too, but suspect it wasn't vetted properly, and I wonder if more recent FM revs have dealt with the issue it's supposed to fix - because I've never experienced it.

[LaRetta]

Hey, thanks. Don't get me wrong ... I have a long history with crashed files, corruption and data recovery. But it has been my impression that some feel vs. 9 is now worth the risk. Even FileMaker itself, and I can't show the link it came from their website, finally acknowledged that a file shouldn't be used after Recovery was performed admitting that Recovery actively and aggresively scours the file for any possible signs of problem and deletes them.

So I was shocked to see the FM article; particularly since it was updated just this fall. I don't trust ANY recovered file even if it burps. Period. But we are giving out mixed messages. The two posts prior to mine is prime example.

I confess that it makes me fume.

[LaRetta]

Hi Colin, thank you for joining in. I truly don't care what the Script Maker problem was; I haven't had it either. I am just twirped because it was updated as early as last fall and it STILL indicates using Recover. Misinformation continues to follow us everywhere and, as indicated by two very contrasting responses above yours ... it causes a lot of confusion and headache.

Edited July 8, 2008 by Guest

[Vaughan]

The Recover command's job is to recover *data*. That is its primary objective, and it will recover data at the expense of file structure if necessary.

This means that a recovered file *may* have bits of structure missing: layouts, scripts, and so on.

Which is why the recovered data should be imported into a clean, known-good backup.

Perhaps FMI needs to rename the command to "recover data" and make a "recover file" command that does the opposite: cleans up the structure at the expense of the data if necessary. But I can see that becoming *very* confusing for users. Imagine if they got them mixed up, it'd be the worst of all scenarios.

[aldipalo]

I'm working with the UI part of a separation model. That's the side that has needed recovery, several times, due to system crashes and power surges. I live in Tampa Bay. The lightning capital of the world! So, we get storms every afternoon. We call it fireworks by God. However, once I run the recover everything appears to be working fine. There are no data files only layouts in the UI section. Even my globals are on the Data side. Are you guys saying I need to rebuild all my layouts in a new UI file?

[LaRetta]

Your repeated crashes can be BECAUSE you have damage. Once a file is damaged, it will crash over and over, unexpectedly. And you will not (usually) be able to find the damage. I have found layouts which were corrupted and deleted them; but they wouldn't delete. Scripts that are corrupt won't let you delete them; custom functions take a nose-dive and are one of the first pieces to corrupt.

To answer your question: Yes, particularly if it keeps crashing on you. You should go back to a good design copy that you know has never crashed. The problem is ... the more time you put into continuing to develop in the file only to finally realize it is trashed ... the more time you will have lost. And NOTHING is more painful than having put a hundred hours into a design and having to go back months because you took the risk that, since it APPEARED okay, it was. And it probably ISN'T. I have set aside probably 20 files which have suffered crashes (most while being served but one wasn't served). And I have identified many wonkies in them (after much hunting). They appeared fine.

I bring this issue up again because we are told that vs. 9 is more stable, that vs. 9 has advanced scanning abilities etc. So I wanted to check:

If a file crashes while being served, should we ALWAYS run Recover on the data and pull it into a clean clone? Can we just replace the UI and assume the data file is okay? If we don't get the scanning error on opening, does that mean the file wasn't damaged? How much of a risk are we taking if we continue to use a file which was improperly closed? Is the risk greater with the UI file (layouts, scripts etc) or the data file?

[fabriceN]

Sorry to disturb, that's for a survey :

Did anyone lose critical data recently ?

I mean, did you experience a situation where you had proper backups scheduled, and where file recovery couldn't restore your most recent data ?

If so, with what version of FM/FMS ?

My idea here is that recovery is not a big issue as I have always been able to recover all the data. What is an issue on the contrary is that FileMaker doesn't provide us a simple way to import data to a healthy structure file.

And this doesn't apply only to recovery situations, but also to our solutions upgrades.

What we would need to do so would be

- import by matching field IDs (some fields can have been deleted or renamed)

- set next serial value that would work -at least- like Set Field, that is to say that we would not have to specify a field (current field would be set if no field specified). This to initialise serial numbers of course

Now, I can be wrong, maybe some of you have lost data that couldn't be recovered. This is why my question occurs.

Thanks,

Fabrice

PS : by the way, look at the warranty terms. It says "doesn't work in Tampa Bay" ???

Edited July 9, 2008 by Guest

[Colin Keefe]

Hear, hear! Though it would also be wonderful to for either function to accept a variable as a specification.

Imagine all the people, living in a world that's context-free...

...

What we would need to do so would be

- import by matching field IDs (some fields can have been deleted or renamed)

- set next serial value that would work -at least- like Set Field, that is to say that we would not have to specify a field (current field would be set if no field specified). This to initialise serial numbers of course

[fabriceN]

Not a single testimonial?

[Colin Keefe]

Perhaps set the question up as a poll in a new topic, so it's not buried in this thread? I'd be interested as well.

I haven't had any issues with lost data on recovery recently, for the record.

But absence of proof isn't proof of absence :

[LaRetta]

Now, I can be wrong, maybe some of you have lost data that couldn't be recovered.

Well I was just told by FileMaker yesterday that recovery doesn't do a THING to (or for) the data; that it's only job is recovering the layouts, scripts etc.

Again ... misinformation is everywhere. You can pull your question out and place it by itself but at least this is the correct forum for it. I'm glad Stephen set up a specific forum for such a misunderstood topic.

My response, Fabrice ... have I lost data that couldn't be recovered? Recovered in the sense of using Recovery? Or recovered in the sense of being able to export/import without data loss? I agree WHOLEHARTEDLY with your suggestion of improved importing and setting next serials in case of corruption ... we just (this morning) finished such a recovery process.

UPDATE: I guess it is terminology which is confusing ... phrases such as "once you recover the data" and everyone thinking that means one should always run Recover on a file before exporting the data. It's just like people saying "server" and others not knowing whether they mean their network server or FileMaker server.

Edited July 10, 2008 by Guest
Added update

[fabriceN]

Yes, you're right about terminology.

So let's formulate clearly : "did you permanently lose data?"

Cheers,

Fabrice

[LaRetta]

I've only had one crash on vs. 9 and we didn't lose any data that we know of (yet). I HAVE lost data in vs. 7 and 8 although rarely; sometimes records with portions of words or a single field would be blank. But I've never lost entire records ever. It has been MY understanding that one should ALWAYS run Recover to repair any possible problems in the DATA and then export that data and pull into clean clone - that it would be the ONLY way of being sure the data was totally clean and repaired). And, I have files which kept crashing whenever script would put cursor into a single field (of a single record) that, once I ran Recover, didn't crash any more but had 3 characters missing in that field. So 'Recover doesn't touch the data' is big shock to me and I STILL don't know if it is true; it certainly hasn't been my experience.

Our recent crashed files had two tests performed: Using a file that FileMaker gave us that their techs use and we also ran Recover on them (just to see if it identified corruption). We KNEW we had corruption and trashed indexes throughout. Neither produced the slightest indication of a problem. Recover showed all zeros (nothing identified as even close to bad). I know older versions (7,8) would trash portions if even suspected. So I still don't trust any file which has crashed, even with vs. 9's reputation for being tighter at scanning etc.

[mr_vodka]

This issue with recovered files is that while it may seem all is well, it actually may not. The recovery process tries to verify every block of data. If it can not verify, it will just ignore it. Therefore it should only be a last resort to get the data out of it and bring it into a known good backup. It can come bite you months from now when you have a lot more into the file. Its better to err on the side of caution.

However, even recover sometimes can not find all the hidden corruption such as corrupt characters in a particular record's field.

[aldipalo]

When I upgraded from 8.5 to 9 every file I had, even test files were required to run Recover. After running it I did not have further problems until this week and it was on the UI side of the separation model.

Now LaRetta says

Well I was just told by FileMaker yesterday that recovery doesn't do a THING to (or for) the data; that it's only job is recovering the layouts, scripts etc.

So, if you run recover and it tells you everything was recovered, no lost data, or whatever the message says, what then?

Since I've been using the files for some time now and have not rebuilt my files (Because I thought they had been "Recovered") I guess I will have to go along until I get bitten. It would be impossible to figure out when it happened and it would costs me 100's of hours. I will start making backups every time I make any major changes and hope for the best. Good thing I have a 500gb external HD to save all these backed up files.

I will also make copies of my files before I upgrade in the future. At my age you just keep learning that experience IS the best teacher. :(

[mr_vodka]

At some point, you may want to optimize and recreate your system. I know that is sucks but its always better to be safe than sorry. I know that you had thought about reselling your solution.

A good practice is to regularly save a copy as a clone and have that as your clean baseline for versioning.

[aldipalo]

On my main app I create 2 copies every night on two different drives. As far as I can tell most things are working correctly. On the new app I'm building the data side of the app is fine and has not had to be recovered. The UI side has been recovered twice in the last two weeks. That's about 40 hours of development down the drain. Again, I have not seen any irregularities in the UI and any scripts. I've run them all several times and have tested every scenario I could think of. I guess I'll have to go back and make new copies of all my layouts and script. The B--ch is all my TO's. Oh well!

[LaRetta]

So, if you run recover and it tells you everything was recovered, no lost data, or whatever the message says, what then?

At least as of the current vs. 9.0v3, do NOT trust what it says. If a file has closed improperly, replace it immediately with a clean empty clone, no matter how tempting it may be to keep it; not matter how much work you may lose in the process. It simply is NOT worth the risk.

Well I was just told by FileMaker yesterday that recovery doesn't do a THING to (or for) the data; that it's only job is recovering the layouts, scripts etc.

I have NOW been told that what we were told (by a FileMakaer system engineer) is not correct and that Recover DOES assist in repair and recovery of the data (and this was by the top FM engineer who specializes in corruption). I believe we have hope on the horizon and that FileMaker is listening to our needs. I sure hope so because I've performed 9 recoveries in the past two weeks (and counting). And yes, the reasons are being tracked down ... it is network (probably hardware) issues.

LaRetta

[fabriceN]

LaRetta, did you recently move to Tempa Bay ? :

[LaRetta]

Hi Fabrice!! I don't know 'Tempa Bay' ... is that in reference to a TV show or movie, like in a joke? Or did you mean Tampa Bay as in Florida? Nope, I moved 15 months ago but not recently. Did I say something funny in my post that makes you think of something called Tempa Bay? :laugh2:

[fabriceN]

Hi!

no, it was just above in the thread. :

http://fmforums.com/forum/showpost.php?post/296399/

[LaRetta]

Ah! Now it makes perfect sense! Nah, ours isn't power outages from storms; they say it's network issue ... a bad switch, cable, port or thingy. They are working down the lines to isolate it. :king:

[Cortical]

You should be using a UPS (recommend APC).

Personally I prefer to put all my globals, aka. volatile data in my user file.

[Cortical]

After spending all last week re-building a user/data file pair, from scratch, that had corrupted, and which recover, recovered; it took 3 days to identify a field corruption in the recovered data file; it was still there.

Recover facilitates data recovery. How much one trusts recovered data is another issue.

Yes one could have resorted to a trusted clone; but this particular solution had been in need of a ground up review/scrub/rebirth for some time anyway...

Edited August 8, 2008 by Guest

[LaRetta]

...it took 3 days to identify a field corruption in the recovered data file; it was still there.

My passion is to see if I can find residual corruption from a recovered file. It is like an easter egg hunt. I never use a recovered file again but I keep them to play with. I usually identify corruption in them somewhere and 40% of the time, it is when the Recovery dialog says all zeros (nothing needed to be rebuilt) Bahhh.

How much one trusts recovered data is another issue.

What are our current options, Chris? I've even had data Recovered and exported as .MER import back into a file (clean clone) bringing lower ASCII with it. It kept crashing us every time script hit that record (and only that one field). Export/import is still better (I think) than importing directly from Recovered file into new clone.

Normally I'm an optimist but when it comes to crashing, I trust NOTHING. As of finishing a rebuild tonight, I'll have performed 46 recoveries in past 3 1/2 years (but who's counting). ROFLMAO!

[Cortical]

I too think export to individual files, then import these into the new db, is the way.

As far as current options re trusting the recovered data; if one were really|really|really concerned about the data, it is easy enough to create match rels/calcs/scripting to flag discrepancies between original data and the exported/imported data; somewhat time consuming yes, but in the first instance only the primary data needs to be crossmatched; i.e. non calc fields...

46 recovers? good grief. I would be looking seriously at my OS/RAM.

regards

Chris

[LaRetta]

46 recovers? good grief. I would be looking seriously at my OS/RAM.

This is different businesses; different OS; different solutions. Truth? I think I'm quite lucky. :wink2:

[gdurniak]

Unfortunately, FM9 Recover is only slightly improved (it is now supposed to "fix" the stuck scriptmaker problem)

Pretty much everything I posted in 2003 is still true

http://nyfmp.org/1/57

FileMaker explained at DevCon 2008 that they are looking at ways to make Recover more useful

greg

>I have NOW been told that what we were told (by a FileMakaer system engineer) is not correct and that Recover DOES assist in repair and recovery of the data (and this was by the top FM engineer who specializes in corruption).

LaRetta

[gdurniak]

(cont'd from above)

I have collected info on Recover here:

http://fileshoppe.com/recover.htm

greg