S L O W Server Pt. II

[Peter Payne]

Thanks for everyone's help on my other post. (I had databases that ran about 1/10 the speed they should on FM9 Server Advanced). Here's what I did to try to make this work again.

a) I did the export/import thing, exporting my data to MER or FP7

I saved clones of the database

c) I recovered those clones

d) using the clones, I imported the data again

e) I did NOT then save a compressed copy, as I read I should do somewhere, if this matters

I also dropped down from FM Server 9 Advanced to 8 Normal, and things were great for a while. For a while. Then I started getting crashing again, and that's where I am now -- the database will go down 1-2 times per day, and certainly overnight, and everything is basically the Vietnam War around here. No one can work at all.

I am at this point going to try to rebuild my files. I am avoiding doing the "rebuild in one file" because I am *sure* that future problems will happen, and having one megafile to recover cannot be good for someone in this situation.

What my plan is to rebuild the databases is basically

a) open, say, INVOICES.FP7 renamed INVOICES OLD

: make a new database

c) copy and past the table fields keeping all names the same (IS THIS OKAY?)

d) rebuild the relationships

e) copy and paste the layout elements keeping all names the same (IS THIS OKAY?)

f) import the data

So, my question is mainly, is rebuilding by copy and pasting okay, or should I assume that this will bring over corruption? I know the answer will be "maybe" so why am I asking? : If anyone can tell me their thoughts I'd appreciate it.

Second, is there no tool to "scrub" a MER file of corruption, of any data other than ASCII and tabs and stuff, so I can know that I don't have any control X's in my data that is causing all of this?If the files were small I'd open in BBEdit and zap gremlins, but the data here is huge of course, 9 gb for the exported file.

Also, is XML better for exporting/importing? I'd choose it but occasionally I have Japanese text in my database, which breaks the import. (Unless these can be cleaned with some tool).

PLEASE help, thank you!

[Peter Payne]

I did find a good .jar tool to scrub the XML code, called atlassian-xml-cleaner.0.1.jar which seems to clean them up nicely. (Actually, it said my files were already clean, but whatever.) Any feedback on the other issues would be great.

[IdealData]

Have you considered you may have a dodgy hard drive?

I recently had an Xserve HD go down, admittedly it wasn't running FMS but it is my primary file server - things were starting to slow down and I had system crashes. A new HD solved it.

Another idea - put the databases on another machine and run them using FMP - not FMS and see how it goes.

[Steven H. Blackwell]

did the export/import thing, exporting my data to MER or FP7

I saved clones of the database

c) I recovered those clones

d) using the clones, I imported the data again

e) I did NOT then save a compressed copy, as I read I should do somewhere, if this matters

Why did you do these things, especially recovering a clone?

This sounds as if there are hardware issues or possibly network items in play here.

What are the hardware specs and the OS here?

Steven

[Peter Payne]

Thanks for the replies. The server is running on a G5 2.5 ghz with 2 gb of memory. Having had a server go "bad" before, I thought of this and have tried running FMServer on various other machines, including two Mac Pros that were brand new. The original machine started out with a Power Mac G5 2.0 running on a Raid, and I moved everything to the 2.5 ghz machine to see if that would fix things. While hardware might be a problem, I've switched things around enough that it *seems* to not be the case.

So here's an update:

I *completely* rebuilt all files from scratch using the instructions from this page: http://www.dbservices.com/articles/filemaker-pro/filemaker-conversion-tips.html. It took two days and wasn't fun, but I got it mostly working, although there were little problems like scripts that pointed at the wrong place. I exported all my data to XML format, cleaned it with atlassian-xml-cleaner-0.1.jar, and imported it. I also removed a lot of bad code along the way.

Things worked great yesterday...but when I was testing at night, I noticed the server going slow as before, and I thought something must be wrong. Sure enough, it crashed during a backup overnight. I brought it up again, but it just crashed a few hours later, making me ready to go live on a mountaintop as a hermit.

So. An old database solution from 1996 that was ultimately based on the ancient Invoice and Products solution that came with Filemaker 3. (The first relational version!) Files have been completely renewed from scratch with old/unnecessary scripts and data sources removed, but the exact same problems exist. File works great for 1-2 days then problems start to build up. I've changed the server machine twice, so I don't think it's straight bad hardware. I switched out the ethernet to another switch, and also (thinking by this time that it might be "bad" electricity) plugged the machine into a different wall socket. Currently I'm running from Filemaker 8.5 directly, no more server for me until I can figure this out.

Can anyone give me more suggestions? Fundamentally bad data pointers that made it over into the new files which are causing data to build up? "Corrupted" scripts that cause problems in the new file when I copy/paste them? "Corrupted" layouts that also bring problems over when I copy them in?

I have to ask in agony, WHY is Filemaker not a better tool? Why is there no exact method of getting to the source of problems like this? Sifting through the "folklore" of Filemaker database development these past weeks has been amazingly complex. If it were possible to move to another platform I'd do it, but of course that wouldn't be possible. Filemaker is so easy to use, like the Mac itself, until something like this happens. Can anyone tell me if they'll *really* be addressing corruption issues in FM10?

Any other help you can give would be greatly appreciated. My entire company is on the line here.

Edited November 18, 2008 by Guest

[IdealData]

Peter

You appear to have hit "headless chicken zone" - slow down, "half the pace, twice the speed"

Let's get some basics in place:

It's not clear which version of OS X or FMS you are using. The symptoms you describe have been seen before with bug ridden version of FMS.

If you completely re-built your files from scratch in only 2 days then this is not a complex solution, and I doubt that you have any data corruption.

I advised you to try running under FMP to eliminate FMS from the problem - hey presto it's seems to run. So, logical conclusion, its FMS that is your problem - not your data.

Please let us know what versions you are running - DO NOT SKIMP on detail - for example OS X 10.5.5 - not just 10.5!

[gdurniak]

yes, the previous post has the right idea. Since this is a trial & error process of elimination, you need to step back, simplify, and "host" with plain FileMaker. Have just one "user", then two.

The hardest part is *making* it crash. Until that happens, you don't know what to fix. With my first corrupted FM4 file, this took a full 7 days of anguish. It was a script that crashed only when *two* users were on the same layout. One user was fine. Go figure.

yes, copy / paste / import can carry "corruption" from one file to the next. Although this is hotly debated, I have seen it happen (an imported script), so I avoid it, when time & money permits.

something as simple as a "bad" graphic Logo can crash FileMaker. "Print" script steps can also be problematic.

I have posted tips here (as you know) http://nyfmp.org/1/57

also, please post more details, e.g. OS, number of files, file sizes, FM versions, etc

make sure there is no other backup, or anti virus software trying to access the "live" files

Since you say (backchannel) that "backups will eventually start to give verification errors", you probably have a hardware problem, memory / hard drive / network card, on host or client

ping your network multiple times. any errors? (a managed switch makes this easier)

greg

[Peter Payne]

Thanks for the help. Yes, I am kind of losing it there, sorry about that ;;

Well, there are about 12 files in the set, including invoices, products, line items, shipping log and so on. The system had run for the longest time on a G5 2.0 running Leopard (latest updates) and Filemaker Server 9. Around 5 weeks who things started slowing down and the system would go down, requiring that I verify the files and sometimes work from backed up versions in the event of files that needed to be recovered.

I've tried many things over the past month, including running the files on other machines (two Mac Pros) and moving it from the G5 2.0 to a similar unit (2.5 mhz), all while completely reformatting everything when I start. I also bought a new 10K rpm hard drive to run everything off, with no difference in the issue. I had no luck getting the slowness and crashing in FMS9 to stop, so I switched to Tiger (latest updated version) and Filemaker Server 8, the previous version we'd used. It seemed fine, but then crashing started 1-2 days later. (No slowness, as with FMS9, but overall less stability.) That's when I moved everything to new files, copy/pasting where applicable, and re-importing all (cleaned) XML data. It ran for two days before getting slow and then going down overnight last night.

Currently I'm running the files on normal FMA 9 (newest version), with the clients connecting directly quitting 3-4 times a day to manually save backups. So far no problems, and I plan to keep with this for another week while I see how things shake out.

So, on the issue of corruption, can anyone tell me if there's a "best way" to find what the corruption, if any, might be? Can I reset scripts that print? Should I delete and recreate all layout elements other than fields? It's been really hard to see what happens when the crashes happen, and it's nothing like, clicking on this field brings an instant crash. (Although backing up seemed to be a big point of crashing, at least with Filemaker Server 9.)

One possible issue, I noticed I had FMP 8.5 v1 on most of my client machines. I've upgraded everything to v2 over the past week, with the last machines I'd missed done now. (My two Filemaker Pro 9 Advanced installs are at v3). Is FM8.5 v1 known for corruption like some of the versions of 8 were?

Thanks in advance for any other options you can offer.

[gdurniak]

check your activity monitor

e.g. a recent post on Tech Talk "Do **NOT** keep Admin Console running on OS X"

"I launched Activity Monitor and watched as the Console consistently consumed an average of 25% of my processes (dual G5, so really 12% overall) and upwards of 40%"

greg

[Peter Payne]

Urg, just got a crash a few minutes after starting with clean files that I'd backed up last night. This fills me with fear since crashing without backups = fail. I'm currently doing a consistency check on the files that just crashed.

I don't know what Admin Console is, but I'm running Activity Monitor to see if I can see anything if I get another crash. Is there any way to get log updates with normal filemaker, such as running Server with tail -f /Library/Filemaker Server/Logs/Event.log with server?

Edited November 19, 2008 by Guest

[IdealData]

Peter

You are still not giving the info we need to help you.

OS and FMS EXACT version numbers please. In the mean time run tour system using FMP to host - at least the users can still work.

[Peter Payne]

Sorry. All Macs are running 10.5.5 build 9F33. The machine hosting the databases is running this, and Filemaker Pro Advanced 9.0v3.

The (current install of) the server that was giving us problems with the slow files that crashed during backup is a Mac G5 tower 2.5 ghz running 10.5.5, with Filemaker Server 9 Advanced, version 0.3.3.326.

All clients are using Filemaker Pro 8.5v2. I did NOT have these all updated properly, most of them were set to v1 before I found it fairly recently. Is that that much of a danger?

Also, my mother (who has been in databases since the Wang days) swears that I must have a bad cable, since that was always the solution for her. For the record I did try running the machine that crashed on 2-3 different ethernet ports, and in a different switch (our sub-switch as opposed to our main one), but it didn't seem to have an effect. Anyone think this could be a cause?

Currently the idea of hosting the files off normal Filemaker is working well. I'll see if it crashes overnight.

[Peter Payne]

I had another crash, this time while importing some invoices that I'd left out of by rebuilt files. Same type of crash as before (computer is active but color wheeled).

Does anyone have any good suggestions about what to be looking for? Is there a checklist of ways to check for corruption in scripts or layouts? Has anyone heard of a bad network switch/ethernet cables causing this kind of issue?

[Vaughan]

I'd be clearing the slate:

Get a different computer, erase the hard disk, install operating system and FM Server, make sure everything is patched and correct and by-the-book.

Get a set of the files, create compressed copies and host these.

[IdealData]

I'm with you on that Vaughan. I'm still suspicious that the file references are not right - maybe they are corrupted.

Having read back through this posting problems started around 5 or 6 weeks ago.

Think - what happened then, and not just about the server, but possibly network setup etc..

[Peter Payne]

Thanks for the advice. So one thing to start with files saved as compressed? This clears out indexed data and causes it to be rebuilt when needed, right? Is this the same thing as using the File Maintenance to remove unnecessary information from open files?

I can certainly try this and see what happens. So I should

a) save a copy of my files w/ compression

do a fresh wipe of my server, in Leopard, using the other hard drive

c) see if the slowness returns and stop using it if it does since it will start crashing/causing problems soon.

Currently I've been hosting the files in "normal" Filemaker Pro, manually quitting to save a backup 3 times a day, and it's been stable, if inconvenient. Since the files seem to crash most every day overnight (frozen server/color wheel from hell), it seems another approach might be to run all files but one (say, products this night, or invoices the next) and see if the crashing goes away. This would (?) help me figure out which file the problem is in maybe? Does this sound like a good idea?

[IdealData]

Are your files not related to each other?

How can you not run one of the files if it is related?

If you can do this then certainly it will be a good test.

[Kris M]

Outside thought here. If your hosting files via "normal" filemaker pro (not server) are the database files subject to automated virusscan while open? If yes this may be causing corruption.

K

[Peter Payne]

No, no virus scanning or anything is running. I also have the folder the files are hosted from set so that Time Machine doesn't try to back them up, in case that also is a no-no. (I figure it would be.)

This is funny. The files ran fine all day, and I just got done compressing them according to the advice here, to see if they better/differently. 30 minutes later, the server is down again. I hate walking to work at 1 am to reboot it, but I guess I will, to get more data on if it crashes again for the rest of the night.

I presume there's no way to get information on what Filemaker is doing at all when the system goes down, like viewing log files?

[Peter Payne]

Brief update. I wiped one of the G5's again, reinstalled everything, and plugged it in on the other side of our work space, into a different electric plug and ethernet switch, and problems seemed t have magically disappeared. Hence it appears to be ethernet related, and I'll try replacing all the wires in that part of the company when I move the server back. Thanks for everyone's help, it was the most frustrating thing I've been trough, but also educational.

[Steven H. Blackwell]

o one thing to start with files saved as compressed? This clears out indexed data and causes it to be rebuilt when needed, right? Is this the same thing as using the File Maintenance to remove unnecessary information from open files?

[color:red]Absolutely, positiviely, not. Do not ever use FIle Mainetenance on files that are thought to be damaged. If you do, you will likely damage them more. From the FIle...menu, select Save a Copy...as compacted.

If I had a magic wand I could wave over FileMaker Pro 9 Advanced, those File Maintenance options would disappear.

Steven

[Joe King]

Several ideas:

1. check into Beezwax Inspector and dotcom FMDiff. I've found they can be valuable tools.

2. Send me the files and I will host them temporarily on my Windows Server 2003 based Server with FileMaker 9 Server Advanced.

3. Like they others my first impression is not a file corruption or hardware; so what about the network? What happens when you run 'bad' copy of the database on FMP Client without hosting.

4. The fact that backup failed is a sign. Did it only fail once?

5. I know that you stated that most of your data was 'text' but I would be looking real close at any binary data...

[Thomas Seidler]

you mentioned turning time machine off, you also obviously have to tell [color:blue]Spotlight not to index that drive/area folder etc. - add it to private sections.

Good to read all that detail, and resolution - since I'm just in process of converting 48 file FMS5.5 solution, 2.6GB of data to FMS10 via MetaData etc,

Blessings,

T

EDIT: i should have started a new thread re: server backup since this is not discussed on any thread i can see in server forum... so i did...

Edited March 26, 2009 by Guest
new thread needed for part of post

[Peter Payne]

I wanted to update this one more time. I am pretty sure the issue was, I had a bloody crossover cable plugged into my server, instead of an Ethernet cable. I of course changed cables when I started getting the problems, but presumably the data was in a corrupted state when I started doing this, and it wouldn't have happened if I hadn't have done the crossover cable thing in the first place. Word to the wise!