Corrupt File/Data disappeared

[dylan214u]

I have to be the dumbest guy on earth. I recently had a power surge and all my data disappeared. The data is still there because I can do various find requests and get a record count that should meet the criteria but I cannot see it. The file still shows the size it was before the surge I've tried all kinds of recovery techniques. Recovering under Mac and Windows environments. I've tried to import data into clone. The catcher is I don't have a recent backup(dum dumb). Anyone got an idea?? I've sent the file to FM recovery but it'll take several weeks. The file will open with no errors. Any suggestions out there would be greatly appreciated. I have also checked to see if under get info anything is unusually checked and isn't Thanks

[LaRetta]

Hi Dylan,

Oh. I've had this happen - more than once. Bite the bullet, trash your existing design and use a good backup copy (as best you can). Your data is 'probably' fine and can be imported into new design copy after running recovery on it. If import won't work, try exporting as .MER or .CSV. But FM will probably never identify those records again in your crashed file. It seems to lose it's index forever and won't spot those records. Even if you create a new table, the records will not appear.

Even if they did, it is highly likely that your design has suffered. And if you kept using it, you may not discover the breaks until further into the project - after putting in many more hours. I sympathize greatly. It has happened to all of us at one time or another. Take your loss now. It will save you heartache later. And get a UPS. An Uninterruptible Power Supply (and frequent backups) will save you from these terror moments - I learned the hard way.

LaRetta

[Adrian Simpson]

The scary thing is that this will continue to happen, especially if you have this being served on FileMaker Pro Server. I posted this issue a few months ago. My clients have experienced no crashes whatsover. The files were simply converted from 6 - 7 and then to 8. Both versions have the same problem. Just by being used, and sometimes just overnight, the data in a table would simply disappear. You can still search on it, but it is gone. I have tried all sorts of ways to get the data back. But the only way is from a backup. The corruption to the file is permanent. I even tried giving it to FileMaker but they couldn't figure it out. They just said that someone must have caused it to crash. This is a FileMaker 7/8 issue that they will never admit to. All we did with a file that had been used for 3 years in version 5 & 6 was to convert it to 7 and host it on Server 7, and within weeks, the data in numerous tables started going in this fashion.

[xochi]

Dylan,

I had a corrupted database that FM7 Developer would crash on, while trying to recover. However, FM8 would recover the file just fine. I would advise that before you give up on a file, try to recover it using FM 8 Advanced, as it seems to have improved algorithms.

Whether you dare to use this file after recovery (or just use it to get raw data) seems to be a matter of religious dispute. Personally, I'm agnostic as to what the risks / benefits are under FM8 of doing this.

[xochi]

Also, I forgot to add -- try both "File Maintenance" commands under the Tool menu in FM8 Advanced. I've seen some cases where Recovery + File Maintenance did a better job than just Recovery alone.

[tv_kid]

I've also experienced the same problem with files in a complex solution, served by Server 7. The files were converted from v6 and 'rationalised'. Although there were minor issues in the old files, since conversion the problems have kept on coming.

As described above, for no apparent reason all the data from one table simply disappeared from view one day. The records were still there, and they appeared to sort too, they just couldn't be viewed.

Following the advice of these forums and FM knowledge base, I went to a backup, exported the data to a text or csv file, saved as a clone and reimported. This appeared to solve the problem, except that when the file was put back into the mix (six files in all with approx 50 tables in total) it created a kind of 'domino effect' with more and more bizarre problems appearing in hitherto 'good' files. On three seperate occasions I have deleted then completely recreated an entire table in a file to try to avoid this problem. It seems to work but a problem then pops up in a different table. Worse still, the new tables do not appear to be corruption clear, as (different) problems occur.

It culminated today. After investigating why information from related records wasn't been pulled through, I realised that the index of the primary key in the related table was virtually (but not totally) empty. After consulting the forums (again!, and may I just add what a great resource they are), I turned off the indexing on this field in the Define Fields dialogue, (intending to close the file, open it again and turn indexing back on). When I clicked ok to exit Define Fields, the file crashed. I had to force quit FM, and then found that not only was that file now severly corrupted (trying the recovery process just crashed FM), but another file that was also open and in the foreground at the time of the crash was also corrupted beyond repair (luckily it was an interface file with no data).

To top it all, FM Server (which was hosting the files at the time) was now unresponsive and had to be shut down from the Activity Monitor (in OS X Server).

I'm at a bit of a loss to know how to proceed. I fear to work on any of the relatively minor issues that are cropping up in case I cause something far worse. I'm being paid to investigate this, but somehow I don't think the company who are hiring me will be very happy if I suggest that we scrap the entire lot and rebuild.

Any thoughts (or even just sympathy)?

Edited November 24, 2005 by Guest

[xochi]

That level of recurring corruption seems unusual to me. I'm wondering if you are running into the old "someone is opening the file via file sharing" bug or if something else is seriously wrong with your system?

My suggestions:

1. Make 100% absolutely completely totally for friggin-sure that there is no possibility that someone is opening the file directly via file sharing, or from a client on the same machine as on the server.

2. Check for bad RAM or other hardware / software problem -- if on Mac, run the Apple Hardware test CD, boot from the install CD and repair permissions and repair your boot drive, etc.

3. Make sure you are updated to the most recent versions of FM server and client. Check the version #s in the "about this program" file menu to make sure the installations actually worked.

[xochi]

One more thing:

4. Make sure your server is running on a UPS that has a fresh battery. About 2 years ago, I had recurring corruption that kept happening every few days on a brand new PowerMac G5. I finally realized that it was being caused by power spikes and brownouts.

The reason I didn't realize that the power was so bad at my office was that I'd been running exclusively on powerbooks, which are relatively immune to power spikes due to their built in battery.

[tv_kid]

Hi xochi

Thanks for taking the time to offer some suggestions:

1. Make 100% absolutely completely totally for friggin-sure that there is no possibility that someone is opening the file directly via file sharing, or from a client on the same machine as on the server.

I am 100% certain that this is not the case. I set up the Server myself. FM Client is not on the machine and there is no way the individual files can be accessed via the network, only via FM net.

2. Check for bad RAM or other hardware / software problem -- if on Mac, run the Apple Hardware test CD, boot from the install CD and repair permissions and repair your boot drive, etc.

I hadn't considered that hardware might be an issue, it is a relatively old (5 years) server so it could be due for a bit of an overhaul. It only has 250MB RAM, which is the minimum for running Server, so it may be that it could do with a bit of a boost. I'll check it out.

3. Make sure you are updated to the most recent versions of FM server and client. Check the version #s in the "about this program" file menu to make sure the installations actually worked.

I had thought about this. FM client is on 7v3 and Server on 7.02. Have seen some bad reports about 7.04 though (http://www.fmforums.com/forum/showtopic.php?tid/171343/post/184797/#184797/)

so might leave that one for now!

4. Make sure your server is running on a UPS that has a fresh battery. About 2 years ago, I had recurring corruption that kept happening every few days on a brand new PowerMac G5. I finally realized that it was being caused by power spikes and brownouts.

The reason I didn't realize that the power was so bad at my office was that I'd been running exclusively on powerbooks, which are relatively immune to power spikes due to their built in battery.

That's a good idea too. The UPS is also pretty old, although not as old as the Server. I'll look into this. Any easy way to check the capacity? Put a meter across it I suppose?

Thanks again for your suggestions. I'll report back with an update

Matt

[xochi]

If I'm remembering correctly, server 7.0v2 was problematic for me. I used it to run some stress tests (creating a million records) and could routinely cause it to crash and corrupt my database files after 100,000 or 200,000 records. Upgrading to server 7.0v3 fixed that problem.

I had another problem -- recurring lockups using server 7.0v3 and OSX 10.4.2 that would happen about twice per month. I'm running server 7.0v4 and OSX 10.4.3 now, and it's been over two weeks of uptime, so I'm cautiously optimistic that the lockup has been fixed. I'll probably know for sure in another week or two.

Edit: I just re-read your original post where you said " I recently had a power surge..." so I'm gonna place my money on the Bad Power theory as the one thing that explains it all.

Edited November 30, 2005 by Guest

[xochi]

One more thing : If you need to recover a file, try to get a copy of FM 8 Advanced. I found that it would recover corrupted files that FM7 could not.

The usual caveats apply about how just how stupid it may be to then use that recovered file for anything..

[tv_kid]

Thanks, I'll give the upgrades a try, and also get some more memory for the server.

Edit: I just re-read your original post where you said " I recently had a power surge..." so I'm gonna place my money on the Bad Power theory as the one thing that explains it all.

That wasn't my post (I jumped in the discussion later) but I'll certainly check that out too.

Thanks also for the suggestion about FM 8 Advanced. I'd made sure that a rock solid backup procedure was in place so virtually nothing was lost, but I'll bear that in mind for the future.

Thanks again

Matt

[gtorok]

I have seen this corruption in a complex, migrated solution. This has been a recurring problem for the last year since migration to 7.

I am currently running the solution on FMS 7v3 with approximately 80 connected clients. Every few weeks a random file (ie. different file every time) hosted on FMS will 'dump' all or most of its data. The file will still show the correct number of records, but every record from a certain point on is completely blank. Finds and relationships still work, showing blank records in portals and find results, providing evidence that the index is intact and the incident was not the result of a runaway cascading delete or other programming error. I have seen the issue in large, heavily used files as well as small, seldom changed reference files.

When the file is closed on the server the actual file size shrinks, indicating real data loss as opposed to inaccessible data due to file corruption. Recovering the file, if successful, will remove all of the blank records from the total record count, leaving behind only the records that were not blanked out.

I logged this as a product problem with FMI and I STRONGLY urge others that have seen this issue to log it with FileMaker. They seem to think that I am the only one seeing this issue and it makes it too easy to dismiss as a client-specific problem.

Please log your issue at:

http://www.filemaker.com/company/product/problems.html

Also, I am very interested if anyone has seen this issue on FMS 8. I have not been able to move our production server to 8 until server advanced is available and I have not been able to reproduce the issue in FMS8 albeit with a minimal client load (ie. not a good real-world stress test).

[LaRetta]

Once a file has crashed, whether damage is evident or not, it should be tossed (ie, recover and import into good clean clone). Once it has crashed it will be prone to crash again; each time taking parts of itself down with it until it finally won't recover at all. How do I know for sure? Because I have several crashed files that I use for testing this very thing. Some of them look perfect and work perfect. But they are damaged. And each time the file is opened and closed, the damage spreads like a creeping fungus.

It would not surprise me at all that a file which crashed now continues to crash. Even if you have to re-write half your solution (by going back to a prior un-crashed copy from a year ago), it's better than what you are experiencing. I have been tempted to trust a file is fine ... but I've never given in to the temptation because the risk is too great and the consequence too high.

I also believe that vs. 7 is more sensitive to crashing. It's new, after all. I hold a strong belief that 8 will be much more stable and I'll be happy when our move up is complete (we're half converted now). And running a file on the correct hardware/software helps also. But again ... if a file crashes, toss it. Period. No matter how good it still looks ...

I will also notify FM of my concerns - thanks for the link. We must do what we can from this side ... backup backup backup ... close properly ... no file sharing and the list goes on. My list is getting quite long. I'm going to try praying to my server next.

LaRetta

[Bailey Kessing]

I am getting corrupted indexed fields all the time. You can't search in these fields anymore. To fix it, I simple turn off indexing and then turn it back on and the issue is gone. This is very frustrating and new to 7/8.

Edited December 27, 2005 by Guest

[xochi]

Are you seeing corrupted index fields on a Server/Client setup? If so, what versions of Server & Client?

[Bailey Kessing]

Server 8 and client 8

[onthebass]

A YEAR AND A HALF LATER AND THE PROBLEM IS STILL THERE!!!

I looked for a thread like this a few days ago and didn't find it, so I posted this topic:

SDK Bind Failures and Deleted Records

There I detail weird results when I tried to create an SDK from a solution that I migrated from FM6 to 8.5. When I sent a bind that finally seemed okay to my client he later reported the disappearance of all his records in one of the files; this has occurred twice for him. Now I have been just sitting on his files for a week, scared as hell to send them back to him. This is a big deal because I'm in the middle of a sale, and if he bails then all his friends that are interested will bail too! And I just sunk all that money into the new FM 8.5 Advanced, not to mention hours and hours of development time.

Anyway, I'm going to rebuild the problem file from scratch (it's not very complicated), and then rebind the solution. Send it to my client and tell him, what?, that he should back up 20 times a day because the files I'm trying to sell him may make all his data disappear?

-OTB

[onthebass]

Oh, FYI, the problem seems to be happening in the Windows bind. I haven't binded the solution on the MAC.

Edited May 10, 2007 by Guest