Layout seems to be corrupted

[BAleiHi]

Help! I've been working on a file very intensively for the past month or so and today I tried, for the first time, to share it peer-to-peer.  When I opened it on the remote machine I noticed that my main layout was acting strangely - fields that were supposed to be hidden were not, one field is empty, and so forth.  So I looked at the layout on the remote machine and see a whole lot of crap!  None of the unknown objects show up on the host machine.  So I deleted the unknown objects on the remote machine and when I went back to the host the file was totally bonkers.  I tried to copy and paste an object between layouts and it won't paste.  When I go from layout mode to browse mode, the toolbar stays in Layout mode.  And worst, I can't close the file or Filemaker (Exit is disabled in the File menu and the Window close buttons don't do anything).  Finally I used Task Manager to Force a close.  Ughhh!  Here's what I have tried so far:

1. Saving a compacted file - get the same behavior
2. Recovering the file using all the Advanced options - the recovery crashes Filemaker in step 14 - recovering Layouts and Themes.
3. Going back to old backups - so far they all look OK until I open them remotely at which point the layouts show the same crap.  There may be some really old ones that don't do that, but I'm loathe to go back that far as I have made a *huge* number of improvements recently - in layouts, scripts, and data! The most recent backup, which I made just before deleting the weird layout objects has everything I need in it and works fine on the host computer but I can't recover it (FM hangs at the same place - step 14 of 17).

Lastly, as I began researching this, I realize that I am doing essentially everything wrong to avoid corruption issues:

1. The main file is located on a OneDrive folder that is constantly being synced (slowly)
2. The host machine is running FM13 Advanced while the remote one I'm testing with has FM12 Advanced
3. A couple days ago (maybe it was just yesterday?) while debugging a script something hung and I had to force a close.  It wasn't a power failure, but FM was acting as if it were paused on some running script step. The script debugger was blank...

If you've read this far - thank you!  What I'm hoping for is some process to recover things, preferably from the corrupted file rather than a backup.  I can get to all the data and scripts - I suspect the issue is in the layouts. I have some ideas about how to rebuild from scratch, but would appreciate some guidance on how to best do that...

[Lee Smith]

I moved your topic to be its own thread.

[BAleiHi]

Thank you Lee,

I'm not sure when to start a new thread...

A little more poking around and I have concluded that, indeed, much of the problem, perhaps all of the original problem, was due to the fact that FM12 does not support Popovers or HideWhen, two features I have used extensively.  When I modified the layout in FM12 and saved it, that seems to have corrupted things in the file, no matter which version I open it in.  Uggg!  But at least I have a bead on the problem....

[Kris M]

BAleiHi said "I have some ideas about how to rebuild from scratch, but would appreciate some guidance on how to best do that..."

Correct your profile to show what FM version you are using. It shows 8.5 advanced.

Fix the deployment issues or be ready to repeat this process again at some perfectly inconvenient time in the future.

Start with a new, blank .fmp12 file.

Build the solution over again. Do not copy anything from the old, damaged file... not even graphics.

Run a recover on the damaged file

Export the data after recover.

Import the old, recovered data into the new file.

Scour the data for errors.

If you use fmp 13 features set the minimum version to open as 13.

Make sure your conducting backups as you develop.

After every major development iteration save as clone.

Develop on hosted on FMS only if possible.

 

Edited July 21, 2015 by Kris M

[flyjar]

One other note...you should never open and work in a FileMaker file that is in a OneDrive/Dropbox/Google Drive folder as this is a set up for corruption. Since FileMaker is always writing changes to the file as you work and OneDrive is always syncing these changes, you can really make a mess of the file. It is ok to save backup copies in a OneDrive folder, but working copies should be in a local folder or hosted on FMS.

[Josh Ormond]

One other note...you should never open and work in a FileMaker file that is in a OneDrive/Dropbox/Google Drive folder as this is a set up for corruption. Since FileMaker is always writing changes to the file as you work and OneDrive is always syncing these changes, you can really make a mess of the file. It is ok to save backup copies in a OneDrive folder, but working copies should be in a local folder or hosted on FMS.

This can't be emphasized enough. Those sync programs alter the file when they are syncing...that WILL be the source of corruption. Not an "if", it is a "when" it gets corrupted.

Technically the file lock should prevent it, but it doesn't always. It's asking for trouble.

On another note, is there a reason you are not using FileMaker Server to host the file?

[BAleiHi]

Thanks to you all for the help.  I'm regularly working on the file from both home and school so having it on OneDrive (I formerly had it on Dropbox) is a huge convenience.  So far I have had no corruption issues and even in this case I have concluded that the problem was modifying the file in FM12 when it had FM13 features.  But I (reluctantly) accept your advice and will try to find a way to deal (can you say laptop?).

The reason I don't use FMServer is because I am a teacher in a small private (not rich, ...private) school.  That means that any piece of software that costs more than about $100 (on TechSoup) or requires a network administrator is a huge stretch.  I'm a math teacher who used to write software...way back when it was FORTRAN..., not really an IT guy.  The plan was (is?) to host this file on a FM hosting service.  But the 90MB file takes about 20 minutes to upload and sorting 3000 records when first I open the file on the server takes so long that I've never let it finish.  There's no way I can imagine developing at that speed.  I thought about changing all my scripts to "Perform on Server" which helps the speed a lot.  But it seems that command does not work at all unless you're running FMServer so I would either have to do *everything* under the server or write some wrapper function for every "Perform Script" that determines whether we're on a server and performs the script accordingly.  Either way seems a big hassle and performance hit.  And there will always be the bandwidth problem - we're talking DSL speeds here - no T1 in my world.  So I was trying out the peer-to-peer process.  We won't need more than 5 simultaneous users so it seems to be a better low budget solution.  Perhaps you can shed some light on my thinking which seems to just get more muddled the farther I get into this....

[BAleiHi]

Another question - are there corruption problems using something like DropBox if I limit to one user at a time?  I'm hoping that as long as I don't do any peer-to-peer stuff off a Dropbox share, I can still use it for my development copy.  I have a feeling I'm stuck in the pre-network days and really have to use FMServer to do this right.

[Rick Whitelaw]

No way a 90MB file should take that long to load over DSL. And sorting the small number of records should be nearly instant, laptop or not.

[Josh Ormond]

Opening a live file from dropbox is dangerous. File locking isn't guaranteed. And dropbox will alter your file, sometimes making it so you can't even open or recover it.

Move copy to your desktop and open. Use the 'Save a copy as' so save a copy of the file to dropbox.

[OlgerDiekstra]

we're talking DSL speeds here - no T1 in my world.

That's not a correct statement. T1 is 1.544 kbps, DSL varies from 128kbps to 20Mbps. Then there's synchronous (where upload speed is the same as download) and asynchronous (where download speed is higher than upload). with ADSL2+, you can get way higher upload speeds than T1. ADSL2 and ADSL2+ are all DSL services.

One thing to consider with any internet service is the latency, ie how long it takes for a packet of data to travel from A to B. You will get better performance on a 512/512kbps service with 20ms latency, than a T1 with 100ms latency. It will take 5 times longer on that T1 link to send the same packet compared to the 512/512 link, which is a third of the speed.

In fact, there's a point where a bigger link is simply not going to make a difference any more, when latency gets too high. Latency between Australia and US is about 250ms. A 1Mbps link will perform the same as a 10Mbps link. You have to start applying tricks to improve performance. Adjust TCP window sizes, use bittorrent technology to create multiple data streams, use WAN optimizers, etc to increase throughput.

It's not all about size... :-P

[OlgerDiekstra]

Opening a live file from dropbox is dangerous. File locking isn't guaranteed. And dropbox will alter your file, sometimes making it so you can't even open or recover it.

Move copy to your desktop and open. Use the 'Save a copy as' so save a copy of the file to dropbox.

In the case of the OP file locking would be guaranteed. He's working on Windows and that doesn't let you sync a file via dropbox whilst it's open. Dropbox while start syncing the file as soon as its closed. So if you're developing a FM file from home and work, that can be perfectly save as long as you close the file when you go to work, or go home.

File locking on Windows works differently than on a Mac.

That's not to say it's good practice. If you open the synced file on both computers at the same time, and start making changes at both ends, then once you close both ends, dropbox will save two files as it won't be able to determine which is the master. Or, it will sync one file over the other, loosing the changes made from the other computer.

[Josh Ormond]

In the case of the OP file locking would be guaranteed. He's working on Windows and that doesn't let you sync a file via dropbox whilst it's open. Dropbox while start syncing the file as soon as its closed. So if you're developing a FM file from home and work, that can be perfectly save as long as you close the file when you go to work, or go home.

File locking on Windows works differently than on a Mac.

That's not to say it's good practice. If you open the synced file on both computers at the same time, and start making changes at both ends, then once you close both ends, dropbox will save two files as it won't be able to determine which is the master. Or, it will sync one file over the other, loosing the changes made from the other computer.

I will disagree with you, based on personal experience. There are a number of scenarios where Windows doesn't lock the file properly and Dropbox begins the process of syncing.  The 2 most common scenarios, an OS glitch/error that causes the file not to appear locked to Dropbox, and trying to open the file while Dropbox it already syncing the file. That changes the file, while it's syncing, causing a 2nd sync call. The file can be easily hosed. And you end up with a file that will not open because the file is part way through 2 syncs.

I have had multiple files get corrupted because of opening the file in Dropbox, or any sync software folder ( Google Drive is the worst of them that doesn't respect file lock, or has gaps in the process ).

Edited July 25, 2015 by Josh Ormond

[Josh Ormond]

BAleiHi,

I was reminded of this article today and thought you may want to take a look at it. In addition to a lot of good info ( most of which comes from FMI regarding data corruption ), there is a mention in there about peer-to-peer sharing.

http://sixfriedrice.com/wp/dealing-with-damaged-filemaker-files/