It happened again: Lost host on network

[wga79403]

Zardoz!!

I can confirm exactly what you have posted. I wish I would have found your posting a year ago. I have been wresting with this issue for almost 2 years on and off. A real pain in the caboose!

I have a win2k server fully patched from MS. I host maybe 20 files to about 60 users. Users are both local and remote. I have the remotes trained to select the server by IP. The locals panic when they don't see the server immediatley after clicking on "hosts". This problem hit randomly as you describe from the workstation level.

My nightly scripts on the server stop and start the FM server service. A bandaid and a terrible one at that. I have FM7 Server ready and plan to upgrade within the next 30 days. I haven't seen any listings of a ver7 system with these problems yet.

I have opened trouble tickets with FM on this issue. And still there are "no known issues" with this. Great SUPPORT!

[Ted S]

Zardoz mentioned at the end of his last post that he had a theory as to what is happening on the client to make this problem occur. I'll be curious to see what he comes up with. Zardoz has done excellent work so far!

We had an outage again as recently as last week when one of our traveling salesman tried to fire up FMP while he was both physically plugged into our network and had his wireless adapter enabled. In this scenario were were able to make the host list disappear at will.

[Zardoz]

Thanks wga79403 and Ted S.

One of the perplexing aspects of this disappearing hosts problmem is its apparent randomness. But there is nothing random about what is happening at the server end. All it takes is one ICMP port unreachable message for the host to disappear. So the randomness must be associated somehow with the client end. I don't have any definitive explanations for this randomness, but the observations based on the traces suggest a couple of possibilities.

According to our network traces, we are seeing two scenarios at the client end. In one trace, a client who was attempting to connect to FileMaker Server (FMS) closed the UDP port before receiving any UDP packets from the FMS. The client sent two UDP packets from this UDP port to port 5003 on the FMS, but after sending these two UDP packets, the FileMaker Pro (FMP) application on the client did not listen on the UDP port long enough to receive any UDP responses from the FMS. In the other four traces, FMP listened on the client's UDP port long enough to receive the first UDP response from FMS, but did not listen long enough to receive the second UDP response.

We use opener files for most, if not all, of our databases. In the first case, my guess is that the client doubleclicked an opener file and FMP launched long enough to send the two UDP packets to the FMS but then something happened to cause FMP to quit suddenly--either the client's computer shutdown or, more likely, FMP encountered a problem in Windows XP and had to quit. I'm not sure what the nature of the problem would have been, but every so often when I'm working in FMP 3.0v6 in XP, I suddenly see an XP error message window popping up on the screen stating that FileMaker Pro has encountered a problem and had to quit. Sometimes it's when I'm printing and sometimes it's when I'm reordering scripts in ScriptMaker. There seem to be some basic incompatibilities between FMP 3.0v6 and XP that surface once in a while. Anyway, this particular client has successfully connected many other times without bringing the host down.

In the other four traces, my best guess, as unsatisfying as it may seem, is that this is simply a function of random network delays. The network traces reveal that the normal exchange between client and server goes like this (this is what is seen at the server end, but remember that the client appears to send both UDP packets at the outset):

1. incoming UDP packet from client to server

2. outgoing UDP packet from server to client

3. incoming UDP packet from client to server

4. outgoing UDP packet from server to client

5. incoming TCP packet from client to server (client initiates TCP connection)

6. outgoing TCP acknowledgement packet from server to client

This is followed by exchange of many TCP packets until client initiates shutdown of TCP connection.

At the moment of the breakdown, this is what is typically seen at the server:

1. incoming UDP packet from client to server

2. outgoing UDP packet from server to client

3. incoming UDP packet from client to server

4. outgoing UDP packet from server to client

5. incoming TCP packet from client to server (client initiates TCP connection)

6. outgoing TCP acknowledgement packet from server to client

7. incoming ICMP port unreachable packet from client to server (always related to second UDP packet from server to client)

This is followed by exchange of many TCP packets until client initiates shutdown of TCP connection. However, once the FMS receives the ICMP message, no UDP responses are seen from the server to any subsequent client who attempts to connect. Subsequent connection attempts look like this:

1. incoming UDP packet from client to server

2. incoming UDP packet from client to server

Normally, the FMP client waits long enough to receive the second UDP packet from the server before closing the UDP port and initiating a TCP connection. However, it appears as if the FMP client can initiate a TCP connection based on receipt of just one UDP packet from the server. Perhaps FMP is programmed to wait only a certain reasonable length of time for the second UDP packet to arrive from the server. If the first UDP packet from the server is received successfully, but there is congestion or some other problem on the network that delays the arrival of the second UDP packet beyond this reasonable time, FMP still unbinds from and closes the UDP port and initiates a TCP connection. When the second UDP packet arrives, the client UDP port is no longer open and thus an ICMP port unreachable message is sent back to the server.

Again, the clients associated with these four traces were different clients in each case, and all four of these clients have connected successfully on many other occasions without bringing the host down.

The second packets sent each way appear to be redundant--they are identical to the first packet. UDP is considered a fast but unreliable protocol, so maybe the second identical packet is programmed in for redundancy purposes in case the first packet does not make it.

In all five network traces, the ICMP messages were generated by remote clients--clients who are in regional locations in other cities/towns/provinces as opposed to in our head office building. In the first trace, where two ICMP messages were sent, the client did not appear in the FMSERVLOG.txt file because no TCP connection was established. This would be an example of a client who is unable to connect and who takes the host down with them and leaves no trace in the logfile. In the other four traces, the clients who generated the ICMP messages do appear in the FMSERVLOG.txt file--these are the last clients who are shown in the log file as establishing a connection to a database. All subsequent entries in the log file are typically people closing connections to databases, although it is possible to see clients who have already established a TCP connection establishing connections with other databases on the server.

So the logfile can be useful, but since it is possible that a client can take the host down without appearing in the logfile, you can never be certain from examining the logfile alone whether the last person to establish a connection to FMS is the one who generated the ICMP message; the ICMP message could be the result of the next "invisbile" person who attempted to connect but who experienced a problem with FMP before receiving either of the two UDP responses from FMS.

We have six servers running FMS 3.0v4 on W2K3 and we have only seen the disappearing hosts problem on two of these servers--on the two servers that host databases that are used by regional clients. I don't know much about networks, but it seems reasonable to expect that these random network delays might be more likely to occur in those parts of the network where the distance is greatest.

Our data is not conclusive on this, however. Sometimes the logfile shows that the last person who established a connection is a head office client. This happens maybe 20% of the time. The other 80% of the time it is a regional client who is the last person to connect according to the logfile. Without a network trace, it is impossible to know whether the head office client is the one who generated the ICMP message or whether there was some other invisible client who brought the host down a short time later. Maybe it is possible that random network delays could be affecting packet delivery speeds within the head office as well.

Does anyone have any thoughts on what might cause random delays in packet delivery? It would be interesting if someone could confirm how FMP is programmed to act once a UDP response is received from FMS--does it wait a predetermined amount of time for the second UDP response from FMS before closing the UDP port?

Next time, I'll try to put a concise explanation together that summarizes what appears to be happening that causes the host to disappear. There is one possible remedial step to address this problem--blocking or filtering ICMP port unreachable messages from reaching the server running FMS--that I'll discuss next time as well.

[Zardoz]

Before summarizing, there are a few corrections to make. I'm not satisfied with the explanation so far of what is happening at the client end, so I went back to the five traces to see if there was anything else there that might confirm whether a delay is occurring and that might possibly explain the source of the delay. I also performed a few traces on my own client machine to get a better picture of what is seen at the client end during a connection attempt. Although nothing conclusive turned up, some more interesting pieces of the puzzle came to light that suggest some fine-tuning is needed to the theory about what is happening at the client end. In addition, some errors surfaced regarding information that I previously reported.

Specifically, I looked at the following packet intervals in a normal state and when the break occurs:

1. The interval between the arrival of the 1st and 2nd UDP packets from the client.

2. The interval between the departure of the 1st and 2nd UDP response packets from FMS to the client.

In a normal state (i.e., when no ICMP message is generated), the interval between the two UDP packets from the client ["client interval"] is consistently between 0.990403-1.014624 seconds. In other words, they are seen to arrive at the FMS about 1 second apart. So it looks like FMP (at least version 3.0v6) is programmed to send the two UDP packets 1 second apart. The interval between the two outgoing UDP response packets from FMS to the client ["FMS interval"] closely mimics the interval of the incoming packets--they are seen leaving the FMS about 1 second apart. The interval between receipt of a UDP packet and the response to the packet ["FMS response time"] is consistently between 0.000049-0.000068 seconds--FMS seems to respond very quickly to each UDP packet received.

I compared these intervals with those seen at the moment of the break in the four traces in which one ICMP message was generated. In all four traces, the FMS response time was within the range above, and the FMS interval closely mimicked the client interval. In two of the traces, the client intervals at the moment of the break were within the range seen in a normal state--i.e., very close to 1 second. However, in the other two traces, the client intervals were 1.921551 seconds and 2.734555 seconds, meaning that the interval between receipt of the first and second UDP packets from the clients were 0.92 and 1.73 seconds greater than normal, respectively.

The traces on my client machine actually revealed that the two UDP packets are not sent right off the bat--the traces show a response being received from FMS before the second UDP packet is sent to the FMS. However, I believe it is still true that the sending of second UDP packet by the client is not contingent upon receiving a response from FMS, as, in a broken state, two UDP packets are seen arriving at the FMS from each client without any UDP response from FMS. It looks like FMP is programmed to send two UDP packets 1 second apart regardless of whether a response is received to the first one.

The trace on my machine showed intervals of 0.000499 seconds and 0.000410 seconds between transmission of my first UDP packet and receipt of the FMS response and between transmission of my second UDP packet and receipt of the FMS response. Both response times are well under 1 second.

I also captured a trace on my machine in the broken state (i.e., after FMS stopped responding to UDP packets). This trace confirmed that there is no response coming in from FMS--all that is seen are the two outgoing UDP packets from my machine to FMS, and they were sent 1 second apart.

I also tried a trace from my machine using the File/Open/Hosts/SpecifyHost route as opposed to using an opener file. I was trying to get some insights into how or when the transition from UDP to TCP occurs, thinking that this might somehow be part of the problem. So I captured a trace up to the point of seeing the list of databases in the hosts window, thinking that I should not see any TCP packets in the trace because I did not doubleclick a database in the list to open it. However, the trace showed lots of TCP packets between my computer and FMS. An examination of the data portion of the TCP packets sent from FMS to me revealed the names of the all of the hosted databases. So it looks like the list of databases actually comes across in the TCP packets, not in the UDP packets as previously stated. In rereading some WinSock information, it looks like the UDP Recvfrom() call simply returns the address from which the response packet was received--in other words, it is used to confirm or discover that the FMS is listening, but does not capture the list of available databases.

So, putting all of this new information together, I've got a slightly revised theory as to what is happening at the client end to cause the generation of the ICMP port unreachable message, which I'll hopefully get to later tonight or tomorrow.

[Zardoz]

WinSock sources suggest that the typical sequence of library calls for a UDP session is as follows (parameters can be specified in parentheses for each call):

socket()

bind()

sendto() and/or recvfrom()

close()

The socket is created, bound to a port and then UDP datagrams are sent to and/or received from a port on another host (e.g., a server) until the socket is closed. An application may call sendto and recvfrom as many times as desired. There are many other library calls available, but these are probably the ones that we're most interested in.

Based on observations from the network traces, it looks like the sequence of calls used by FMP 3.0v6 is something like this:

socket()

bind()

sendto()

recvfrom()

sendto()

recvfrom()

close()

If at least one of the recvfrom calls returns valid data (i.e., the IP address of the FMS), then, prior to or after the UDP socket is closed, a transition is made to TCP, whereby a new socket call to open a TCP session is made. The TCP socket uses a different port on the client machine. This seems to be the next sequentially available port number on the client machine, such that if the UDP session used port 1535, then the TCP socket would bind to port 1536 or the next available port number. If neither of the UDP recvfrom calls returns valid data, then the connection attempt times out, the socket closes and nothing displays in the hosts window.

This timeout seems to occur after approximately 3 seconds. When I go File/Open/Hosts/SpecifyHosts and type in an invalid FMS IP address, FMP appears to try for about 3 seconds to establish a connection before returning a blank hosts window. Three seconds also appears to be about the length of time it takes for a successful connection to be made and a list of available databases to display.

This is all speculative, but I'm guessing that the first recvfrom call waits for about 1 second to receive a response from FMS. Regardless of whether a response is received, control in the FMP UDP program passes to the second sendto call after 1 second, and FMP sends a second UDP packet. Control then passes to the second recvfrom call, which waits for a specified time interval (likely also 1 second). At the end of this interval, if a response to at least one of the recvfrom calls has been received, then a TCP socket is created and a TCP packet is sent to FMS (this packet is seen arriving at the FMS at almost exactly 1 second after the second UDP packet), whereas if no response has been received to either recvfrom call, then the connection attempt times out and the UDP socket closes.

Based on these speculations, there are three possible scenarios that could result in generating an ICMP port unreachable message: (1) the first recvfrom call times out before receiving a response from FMS; (2) the second recvfrom call times out before receiving a response from FMS; or (3) both recvfrom calls timeout before receiving a response from FMS. If one of the recvfrom calls times out before receiving a response, then one of the incoming UDP response packets from FMS will have nowhere to go and will generate an ICMP port unreachable message. If both of the recvfrom calls time out before receiving a response, then both of the incoming UDP response packets from FMS will have nowhere to go and will generate ICMP port unreachable messages.

In the five traces, we saw one occurrence of both UDP response packets generating the ICMP message and four occurrences of the second UDP response packet generating the ICMP message. But we do not have any occurrences of an ICMP message being generated solely by the first UDP reponse packet. I suspect this is because the first UDP response packet has two shots at being received successfully by the client. If it is delayed and misses the first recvfrom call, then it still has a shot at making it in time for the second recvfrom call. If it misses the second recvfrom call, then the second UDP response packet is also going to miss both calls, and two ICMP messages will be generated. If the first response packet makes the second recvfrom call, then the second UDP response packet will have nowhere to go and will generate an ICMP message.

We see clear evidence of a delay happening in two of the four network traces in which one ICMP message is generated. In one trace, the second UDP packet from the client arrived at the FMS 1.92 seconds after the first packet; it was delayed by almost a full second. In the other trace, the second UDP packet from the client arrived at the FMS 2.73 seconds after the first packet, a delay of 1.73 seconds. Without seeing a corresponding trace on the client machines, it is not possible to know for sure which recvfrom call was filled by the first UDP response packet. But given the significant delays in receiving the second client packets at the FMS, there is a good chance that the second FMS response packets did not arrive before the timeout period expired for the second recvfrom call.

Although the second client packet in the other two traces arrived at the FMS within the normal interval (i.e., approximately one second after the first client packet), it is possible that the first UDP packet from the client was delayed and/or that one or both of the FMS response packets were delayed in reaching the client. Without seeing client traces, it is impossible to know for certain. In both of these traces, the ICMP message is seen arriving back at the FMS between 1.19 and 1.28 seconds after the second FMS response packet was sent. These seem like rather lengthy round trips, given that in the traces on my client machine, the interval between sending my UDP packet to the FMS and receiving a response was in the 0.000410 to 0.000499 seconds range. The relatively lengthy interval between transmission of the second response packet and receipt of the ICMP packet could indicate some general slowness on the network at this time.

Regarding the fifth network trace in which two ICMP messages were generated, the second client packet did arrive at the FMS within the normal 1 second interval after the first packet. However, the two ICMP packets arrived 2.98 and 2.48 seconds, respectively, after FMS had sent the corresponding response packets. This indicates that there might have been some significant network slowness impacting delivery of the two FMS response packets. In this trace, it is possible that the FMP did not crash at all on the client machine and that it was actually a case of the two response packets not reaching the client before both recvfrom calls timed out. This client is seen on the trace attempting another connection 128 seconds later.

None of this is conclusive, but, in the context of the UDP implementation used by FMP, the random network delay theory seems to offer a few plausible scenarios under which ICMP port unreachable messages can be generated. Summary to follow.

[Zardoz]

To summarize, disappearance of the FileMaker Server (FMS) from the hosts window is associated with the receipt by post-NT servers of an Internet Control Message Protocol (ICMP) port unreachable message. The ICMP message is generated by a client during the UDP discovery phase of an attempt at connecting to a FMS. A randomly occurring network delay causes the delivery of one or both UDP response packets from the FMS to the client to fail to reach the FileMaker Pro (FMP) client within the timeout period associated with FMP's WinSock recvfrom calls. By the time the delayed packet reaches FMP, the UDP socket has closed and an ICMP message is returned to FMS. The ICMP message is translated by the WinSock layer in post-NT servers into a WinSock error message (error message # 10054--WSAECONNRESET) which is then passed on to the UDP runtime portion of FMS. FMS is not programmed to expect an error message in this context and therefore cannot clear this message. During this time, the FMS service is still running, port 5003 is still open on the server and TCP sockets continue to function normally. However, FMS cannot respond to any more UDP packets from clients until this message is cleared, and the only way to clear the message is to restart the FileMaker Server service. All it takes is one ICMP message to produce this result, and all it takes to generate one ICMP message is for one UDP response packet from the FMS to fail to reach the FMP client within the required timeout interval associated with the WinSock recvfrom call. Useful references include Microsoft KnowledgeBase articles 245442 and 263823.

As far as possible solutions go, one strategy would be to block these ICMP messages from reaching the server running FMS. I believe ICMP messages can be blocked at routers and/or firewalls. There may be some reluctance to do this, however (I don't think my network and server contacts are too keen on trying this), as ICMP messages provide tremendous network troubleshooting information. If there is a way to block a specific type of ICMP message (e.g., type 3 code 3) that is related to a packet sent from a specific port (e.g. 5003), then this might be more acceptable, but I don't know if this level of specificity is possible.

Another option might be to block UDP altogether since the problem appears to be associated with the UDP discovery phase. FileMaker TechInfo article 106727 mentions the alternative of selectively blocking UDP packet routing for port 5003 but allowing TCP. I don't really understand this alternative, but I would be interested to know if anyone has tried it and whether, without UDP, discovery still works.

Because the problem seems to hit so randomly, I don't imagine that there are any settings that can be tweaked on network devices (routers, switches, etc.) that would make this random network problem go away. But if anyone has any suggestions in this area, please respond.

Other options might include reverting back to running FMS on Windows NT Server or, of course, upgrading to more recent versions of the FileMaker client and server products. But these are not always feasible options for everybody. Regarding more recent versions of FileMaker products, I notice a lot of people reporting this problem with Server 5 or 5.5. What Windows Server operating system are these FMS server products certified to run under? And has anyone seen or heard of this problem occurring with FMS 7?

It appears that remediation attempts that focus on variables within specific databases (e.g., data, scripts, layouts, etc.) or on variables within the Windows server boxes do not work. The problem seems to be strictly a random network occurrence that exploits an incompatibility between post-NT Windows server OSes and certain versions of FMS.

If anyone has other suggestions for dealing with this problem, please post them here.

That's it. Sorry about all of the long-winded posts, but I've attempted to provide all of the relevant supporting data and background information so that the reasoning behind my conclusions and assumptions is somewhat clear. Hopefully I've got at least some of it right.

Incidentally, there is a freely available network tracer and analyzer at www.ethereal.com that you can download and install on client and/or server machines and gather some network traces. It would be interesting to know whether anyone else who is experiencing this problem is seeing ICMP port unreachable messages in the traces, or whether there are other failure mechanisms at play.

[Zardoz]

It looks like the option mentioned above of blocking UDP as described in TechInfo article 106727 might provide a viable solution for those of you using FMP 5.x or 6. TechInfo Article 108618 seems to suggest that the TCP/IP network plugin used by these versions of FMP differs from that used in earlier versions in one significant respect: although they continue to use UDP, successful receipt of UDP packets from FMS is not required in order for the list of hosted databases to be displayed. I don't fully understand everything that 108618 is saying about this, but it certainly sounds like something worth trying.

Even though successful receipt of UDP packets may not be required in FMP 5.x or 6, I suspect that an ICMP port unreachable message will be sent back to FMS for each UDP packet that is not successfully received by the FMP client. So to take advantage of this difference in the network plugin, it will be necessary to block UDP packets altogether from reaching FMS as described in 106727.

If I understand this correctly, for those of us using FMP versions earlier than 5.x, it sounds like this is not a viable option for us, as the displaying of the list of hosted databases seems to depend upon successful UDP receipt by the FMP client. With these earlier FMP versions, blocking UDP packets from reaching the FMS would result in clients never seeing the list of hosted databases.

[Zardoz]

Just a quick update. We are in the very early stages of testing FMS 7 Advanced on W2K3. For fun, I ran some network traces on my client machine (using FM 7 Developer on XP SP1) to capture traffic to and from FMS 7 Advanced in two scenarios: (1) connecting to a database via the TCP network; and (2) connecting to a web-published database (Instant Web Publishing) via Internet Explorer 6.0 (SP1). In both cases, the first packets seen are TCP, not UDP. After the initial exchange of TCP packets in the TCP network scenario, the session continues with a mix of TCP and General Inter-ORB Protocol (GIOP) packets. In the web-publishing scenario, the session continues with TCP and HTTP packets. There are no UDP packets exchanged at any time in either scenario.

UDP appears to be out of the equation entirely in 7, which suggests that the disappearing host problem may not occur in 7. We haven't done any testing over our WAN yet with our regional clients, but I'll try to provide updates as they become available.

[Rundvelt]

Would clearing the application log help with the errors?

(something I heard)

[Zardoz]

Thanks for the suggestion, Rundvelt. I'm not familiar with what types of problems would be solved by clearing the application log, but I would be interested in any further details that you can provide.

Regarding the application log, when the problem occurs with FMS 3.0v4, there is no message of any kind (error or otherwise) generated in any of the server event logs, including the application log. I think this is because FMS is still running (as evidenced by the continued communication with established TCP connections) and the problem that has occurred in the WinSock UDP layer is not the type of problem that would generate a recordable error in an event log.

When the WSAECONNRESET error message is received, my guess is that the UDP portion of FMS enters a suspended or paused state rather than entering a state that would generate an error event--i.e., it is still running but cannot respond further until some action is taken. This would be something like running a FM script in your database that then pauses indefinitely waiting for user input or generates an error message that must be acknowledged before the script can continue. Neither of these types of occurrences are fatal to the FM application and would not be captured as errors in something like an event log, as FM is still running but it cannot do anything until the required user intervention/response occurs. Unfortunately, in the case of the WSAECONNRESET error message, there is no possible user intervention that will address this; the intervention can only come from within the UDP application itself, and it appears that FMS 3.0v4 has not been programmed to deal with this situation.

Incidentally, when FMS 3.0v4 is stopped and restarted, these stop and start events are captured in the server application log.

[Ted S]

Just a final note to close out this old thread.

I've had about 60+ users live on version 7 for about 4 months now and have not had even 1 incident. I believe that this problem was solved in version 7 probably because v7 doesn't utilize UDP (network) protocol unlike earlier versions. Zardoz confirmed this with his testing.

There is a lot, I mean A LOT of information in this thread. Zardoz deserves credit for finding and explaining the root cause.

To future readers of this thread, don't be frightened-off by the volume of data. If you are interested in reading about the deeply technical aspect then read the lengthy posts; if you are interested in a practical solution, just follow the shorter posts and you will find the answer there.

[gdurniak]

I got hit, but for a different reason

I have FM Server 5 on Windows 2000. I just added a second FM Server 5 on Windows XP. Now, one or the other server drops off the "hosts" list (but not both)

There appears to be something in Windows XP SP2 that interferes with the hosts broadcast, in a "mixed" network (XP, 2000, OS 9, OS X).

In my case, removing the second FM Server on XP fixes the problem

[xtrim]

Hi,

can this issue be related to my problem with FMPU on windows 2003?

Is it because windows 2003??

[Ted S]

xtrim,

I quickly read your posts and it doesn't look like the same problem. You are running web based and I was 100% fat client. I also did not lose the FM Service, just the listing of hosted files. In fact there is some evidence that the files continued to be hosted but since nobody could see them we were under the impression that they were no longer being hosted.

I'm sorry but I can't shed any more light on your problem that the others already have.

Good Luck.

[xtrim]

Well, thank you for the quick answer...

[Zardoz]

Further to what Ted said earlier, we've been hosting an IWP solution using FMS 7 Advanced for nearly two months now with no problems. The solution is used by 500+ clients, most of whom are regional/remote.

[gdurniak]

I have FM Server 5 on Windows 2000. I just added a second FM Server 5 on Windows XP. Now, one or the other server drops off the "hosts" list (but not both)

I believe it is related to the UDP problem described in this thread

Is anyone successfully running 2 or more FileMaker Servers 5 or 5.5, on the same network, on Windows 2000 or higher, where users can "see" both servers?

If so, please post your configuration. Thanks!

[olias88]

Zardoz, you are the man!

As per my post on this topic two years ago, we would still intermittently experience this problem. As of early this summer, we had four FMP 5.5 Servers on a mix of W2K and W2k3 OS's hosting around 300 files. We finally gave up trying to get any info about this problem from paid Filemaker support. They would just continue with their "no known issues" mantra.

But Zardoz, you have nailed the problem completely. Thanks for you dilegence and willingness to share your knowledge. Your detective work and explanations have finally let me understand what was going on.

Since early this summer we have migrated to Filemaker 7 and have not had a single instance of hosts disappearing. Now we have a boatload of FMP7 problems, but that's another topic entirely...