Can not disconnect duplicate / ghost clients

[bungi]

I am running FileMaker Server Advanced version 11.0.1.95 on a Windows Server 2008 R2 (64-bit). All the clients are on Windows using either Pro 11.0v1, Pro11.0v2, Pro Advanced 11.0v1, or Pro Advanced 11.0v2.

I have connected users with 0 files open. I also have some users that are listed multiple times, one with files open, and the rest with 0 files open. I've tried disconnecting them through the Admin Console and through the command line, but neither worked. I have the 'set maximum idle time allowed for FileMaker Clients' option set in FileMaker Server as well as the 'Disconnect user from FileMaker Server when idle' option in the client's Privilege Set, but that too does not appear to working correctly. The only way that I can get rid of the duplicate users is to restart the server.

The users with the duplicate connections have said that they close the database normally, and that their workstations haven't crashed.

I also have "authentication failed" errors in the log viewer. I'm not sure that's the cause of the problem because some users that have failed authentication can disconnect from the server just fine and reconnect without having a duplicate connection, but I will investigate it further.

Most of the users open the database through a launcher shortcut, but again, only some of the users are having the duplicate connection problem.

Some users have multiple older versions of FileMaker installed on their workstation, like versions 5 & 8, but they only use the current version (11) to open the database. That wouldn't contribute to this problem, would it?

Any help would be greatly appreciated.

-Eric

[Steven H. Blackwell]

To start with, please update Server to 11.0v2 and clients to 11.0v2 or 11.0v3.

Steven

[Wim Decorte]

What Steven said about the versions. Especially about the server version. Upgrade to FMS11v2 ASAP, it's a major stability upgrade that's been out for almost a year now.

The only way that I can get rid of the duplicate users is to restart the server.

The users with the duplicate connections have said that they close the database normally, and that their workstations haven't crashed.

What distress does these ghost connections cause? Seems to me like a server restart is more disruptive to more users than some isolated ghost users reported in the admin console.

Do the ghost users report that they cannot again until the ghost connection is gone?

I also have "authentication failed" errors in the log viewer. I'm not sure that's the cause of the problem because some users that have failed authentication can disconnect from the server just fine and reconnect without having a duplicate connection, but I will investigate it further.

Most of the users open the database through a launcher shortcut, but again, only some of the users are having the duplicate connection problem.

Not related.

The launcher file no doubht is set to open with the default "admin/blank" credentials and those don't match the account in the target file. But FM does try to pull open the target file with those "admin/blank" credentials and that fails and cause the authentication error in the event log.

[bungi]

Thanks for the replies. I will try updating the Server to the latest version.

The ghost connections don't really cause any distress other than taking up resources on the server. I only found out about the ghost connections when we hit our connection limit (I know my license doesn't restrict the number of simultaneous connections, but I wanted to set one anyways). Users were saying they were getting an error message when trying to open the database(s). I temporarily resolved that problem by bumping up the connection limit and then restarting the server.

Other than that, ghost users do not report any issues. They just show up multiple times (2+) in the Connected Clients tab in the Admin Console. Their first entry in the list will show 0 Open DBs, and the next one will show the actual number of Open DBs (ranging from 1-5).

The number of these ghost connections seem to grow by maybe 10-15 each day. Hopefully updating to the latest Server version will fix this problem.

[Wim Decorte]

a typical scenario would be that a ghost connection is because a client machine crashed or the network connection to it was interrupted and FMS wasn't able to communicate with that client for a "graceful" disconnect. Typically though those ghosts would disappear after a while as FMS figures out those clients are not there anymore. Are you saying they don't go away by the end of the day?

If they don't: try the command line command:

fmsadmin list clients -s

and see if those ghost clients show up there? If they don't it's your admin console acting weird.

What you can try is this command line command:

fmsadmin restart adminserver

which restarts the part of FMS that is responsible for the console without restarting the database engine and kicking off clients.

[bungi]

The ghost connections do not go away by the end of the day. I have about 100 connections now with 0 DBs open, and some of them have been connected since right after the last server restart, which was about 2 weeks ago.

The ghost connections do show up on the command line. I tried restarting the admin console anyways, but that did not do anything.

[jfortes]

Hi there, same problem here...

If I try to disconnect a user who is performing an heavy search, someone who had his computer crashed or someone had to forcequit FMP, his connection keeps alive and slowing down the server as it is shown in the statistics tab as that user was doing so when disconnected. I believe this only happens when the user is stressing the server.

Every time this happens, I have to disconnect every user out, tell them not to enter again, make a backup, restart the server and restore from backup as I cannot properly stop files.

Sometimes I've waited for days to see if those ghost clients would disappear with out result. I've also tried to stop the database server normally without result. Most of the times, if I try not to restore from a backup to restore the service quickly, some files start verification when opening...

After this episodes when the system is not in use, I verify files and recover them to check for corruption without problems

I'm using FMS Adv 11v2 and FMP 11 v2 and v3 clients, IWP users and odbc connections.

I have the same scenario since I was using FMS Adv 10, the only difference was that with FMS 10, those ghost clients did not show up in the admin console although I'm pretty sure they were also there.

Any suggestion?

Thank you in advance...

[bungi]

Updating to version 11.0.2.217 did not resolve the ghost connection issue. However, I did confirm one potential cause of the ghost connections. We require our users to change their passwords every X amount of days. If the user's password is expired, it will register in the logs as an "authentication failed" error, and when the user changes their password, FileMaker Server will create another connection for that user, while still keeping the previous connection.

I hope FileMaker can fix this bug in the next version update.

[Wim Decorte]

Can you narrow it down to users who have just changed their pw?

What kind of plugins do you use in your solution?

Have you considered using External Authentication? From the # of users you seem to have that would be real benefit.

[jfortes]

Updating to version 11.0.2.217 did not resolve the ghost connection issue. However, I did confirm one potential cause of the ghost connections. We require our users to change their passwords every X amount of days. If the user's password is expired, it will register in the logs as an "authentication failed" error, and when the user changes their password, FileMaker Server will create another connection for that user, while still keeping the previous connection.

I hope FileMaker can fix this bug in the next version update.

So, It was not a problem with users authentication if you still have those ghost users. I also have authentication failed errors in my logs, but my ghost clients do not depend on having those auth errors as some users become ghosts without authentication errors.

In my case, I see a connection between users stressing the server and cutting down their connection or being disconnected from the admin console. ¿Have you experienced the same?

[Wim Decorte]

In my case, I see a connection between users stressing the server and cutting down their connection or being disconnected from the admin console. ¿Have you experienced the same?

What's the link between the admin console and the users though? The admin console has been a sore point ever since the current java version was introduced. You shouldn't keep it open longer than you need it to work with so normally you wouldn't see any disconnect issues there.

I haven't seen any evidence of user load and disconnects. Disconnects typically are isolated to the network components (bad NIC, bad cabling, bad switches or ports on them, QoS settings,...).

Do some performance monitoring on your sever, about a week's worth to verify which of the typical bottlenecks are causing your problems: memory, cpu, disk, network.

Authentication errors happen frequently because the file is still set up to log in with a certain account or the launcher file is but the hosted files use different accounts or EA. In this scenario, every single connect to a hosted file will produce an authentication error in the logs.

[PacoLozano]

Hello.

I'm not fluent in Spanish and English. I am translating with google translator. I apologize for that.

I suffer from the problem described here. In my case the application synchronize data with an external MySQL table. Sometimes the source External SQL no longer available. This causes an error and I think leaves blocked the script from running the operation. This forces the user to close your FileMaker Pro inappropriately. Such this may be the cause of the appearance of ghost clients.

Plugins used:

MBS 2.1

[jfortes]

What's the link between the admin console and the users though? The admin console has been a sore point ever since the current java version was introduced. You shouldn't keep it open longer than you need it to work with so normally you wouldn't see any disconnect issues there.

I haven't seen any evidence of user load and disconnects. Disconnects typically are isolated to the network components (bad NIC, bad cabling, bad switches or ports on them, QoS settings,...).

Do some performance monitoring on your sever, about a week's worth to verify which of the typical bottlenecks are causing your problems: memory, cpu, disk, network.

Authentication errors happen frequently because the file is still set up to log in with a certain account or the launcher file is but the hosted files use different accounts or EA. In this scenario, every single connect to a hosted file will produce an authentication error in the logs.

Sorry I think I did not explain my self properly. If I discconect a user who is stressing the server with a long search for example, then that user becomes a ghost and his connection will never dissapear.Therefore, I see a connection between a user stressing the server and disconnecting him using the admin console and so, I guess a user stressing the server and force clossing his connection will cause the same. In both cases, the server will keep stressed.

I haven´t seen any evidence of user load and disconnects either, in fact I see almost the opposite, there is no or few chances of getting rid of users stressing the servers, as when I try to, they keep alive forever. damm ghosts!

I did performance monitoring on my server (xserve with 8 cores, 12Gb of unused RAM, SSD HD to store production files and 1Gb network ) and in my application. As a result, I changed some processes and scripts. Right now, my performance issues are caused only by advanced users who can perform non standard searches in non standard layouts with calculated fields or related fields. In the past, with previous versions of FMS I had no problem in disconnecting them, but now I can´t, and the only wayout I see is not allowing them to access those layouts what will considerably reduce their autonomy.

Regarding authentications errors, I do not see the connection between them and ghost users but I have them also. Do you believe this has something to do with ghosts?

Thanks for your interest.

Hello.

I'm not fluent in Spanish and English. I am translating with google translator. I apologize for that.

I suffer from the problem described here. In my case the application synchronize data with an external MySQL table. Sometimes the source External SQL no longer available. This causes an error and I think leaves blocked the script from running the operation. This forces the user to close your FileMaker Pro inappropriately. Such this may be the cause of the appearance of ghost clients.

Plugins used:

MBS 2.1

From my experience, 99% sure that a client force closing his connection will end up with a ghost client.

Vamos, que fijo que es eso.

[Vaughan]

If I discconect a user who is stressing the server with a long search for example, then that user becomes a ghost and his connection will never dissapear.

Why are you disconnecting live users?

[PGageler]

I sympathise with all those having this problem, as I have experience it on and off for months. I am surprised however that most responses have been to criticise rather than to offer any solution.

I experienced an issue with Ghost users just today. One user contacted me saying that a simple search resulted in his FileMaker client freezing. For whatever reason, users then started appearing multiple times in the console - apparently databases they were already logged into began freezing and/or quitting them out. Each time they re-logged in (only to kicked off again) FMS Console recorded them again. AT this point the Console become unresponsive - it is impossible to disconnect users and some databases will not respond to the command to close. The FileMaker Service will then refuse to stop, leaving a hardware restart as the only option.

FileMaker Server 11 (I am running FMSA 11.0v3 on a Windows 2003 Server) is flakey to the point of major frustration and embarrassment. I just wish FileMaker were not so distracted with Bento, and get back to fixing these debilitating problems...especially when FM11 has been out for something like 14 months now.

[Bob476]

I'm suffering from this same issue, and while it's not usually a primary user issue, it really kills your day when it's a developer who has to force-close for whatever reason, and then remains as ghost client in role of modifying the database. So even though you can connect again, you cannot manage the database at all from anywhere until you get rid of the ghosts.

And the only way it seems at this point to get rid of ghosts is to restarted the darn server service and force everyone out. So development has to take a powder for the day, if you want to keep it up for your clients, and then wait for a window to restart.

[Wim Decorte]

One user contacted me saying that a simple search resulted in his FileMaker client freezing. For whatever reason, users then started appearing multiple times in the console - apparently databases they were already logged into began freezing and/or quitting them out. Each time they re-logged in (only to kicked off again) FMS Console recorded them again. AT this point the Console become unresponsive - it is impossible to disconnect users and some databases will not respond to the command to close. The FileMaker Service will then refuse to stop, leaving a hardware restart as the only option.

Hmmm... this sounds very much like something we had been experiencing.

- is the data pasted in or imported in? We've seen this happen with pasted data in records that came from lotus notes and had trailing invisible characters at the end. It totally threw off FMP and caused FMS to sorta crash

- the FMS crash sometimes manifested itself by a *.DMP file in the FMS logs folder. Do you see any of those?

We solved the issue by cleaning the data and using a custom menu for the paste function that would clean the data as it came in. We also upped the processor count on the machine so that it wouldn balk at complex finds by multiple users.

[PGageler]

Hmmm... this sounds very much like something we had been experiencing.

- is the data pasted in or imported in? We've seen this happen with pasted data in records that came from lotus notes and had trailing invisible characters at the end. It totally threw off FMP and caused FMS to sorta crash

- the FMS crash sometimes manifested itself by a *.DMP file in the FMS logs folder. Do you see any of those?

We solved the issue by cleaning the data and using a custom menu for the paste function that would clean the data as it came in. We also upped the processor count on the machine so that it wouldn balk at complex finds by multiple users.

Thanks for the advice Wim. No, there was no *.DMP file in the logs folder. I am surprised that FileMaker have not provided a filed option to strip formatting and so avoid the issue altogether.

[Vaughan]

I am surprised that FileMaker have not provided a filed option to strip formatting and so avoid the issue altogether.

First get the Trim4() custom function from Brian Dunning's web site. Then set up an auto-enter calculation on the fields you wan to keep clean:

Trim4( TextFormatRemove( self ) )

Set the auto-enter to replace the data.

[PGageler]

Thanks Vaughan - I am familiar with the function to remove formatting, and apply it to fields that I know people are likely to paste data into from another source (like a web page). But to have to do this on every field (given people can really paste into any field form any source) is an incredibly cumbersome process. There really should be simple option on a field (or even the whole file) to allow or strip formatting on text.

[Vaughan]

To some extent it's not FMP's problem if the imported data has bad characters in it: arguably it's also not for FMP to determine which characters are bad or good (think of the way that Excel makes assumptions about data and changes it without prompting). Many people use applications like Text Wrangler or BBedit to clean up text files before import.

[Steven H. Blackwell]

(given people can really paste into any field form any source)

You can, of course, simply disable the ability to paste into fields.

Steven

[Wim Decorte]

You can, of course, simply disable the ability to paste into fields.

Steven

Absolutely! And if you allow it, use a custom menu for the paste action that will strip out all the unwanted characters out of there.

[PGageler]

Absolutely! And if you allow it, use a custom menu for the paste action that will strip out all the unwanted characters out of there.

Thanks - that's a great suggestion, and easily implemented.

[Joost Miltenburg]

Having the same issues, here.

MacOS X Server 10.6.8

FMServer 11v4

FM9v3 clients.

So you you think it is related to illegal data / characters? Stripping for that is always a good idea, but really FM(I) should not allow them being stored in the first place.

Had a user just now how was gone already, but still it showed up and the pc was off. However, I see now that the user has been removed from the server page. Still, it more than 20 mins or so...

I think it also may have to do with database corruption, that keeps users from disconnecting.

Anyway something

[Chris Papas]

I have the same issue on and off for 8 months now. I don't think it's bad data because it went away for at least 4 months and has now started again.

Anyway, why would bad data stop the server from disconnecting a client? The way it should work is

- The server tells the client to disconnect

- No response from the client because they are not there

- Wait X minutes, no response so drop the connection

Data and client should be irrelevant.

We don't suffer performance issues it's just that it eventually runs out of licences. You cannot disconnect clients or close files. So what i do is up the licence count so more people can log in and late in the evening when I know no-one is working I back the files up, stop the FM service and restart with the backup files.

Server version 11.0.3.309 on Windows Server 2003 SP2

[sirglandon]

Anyway, why would bad data stop the server from disconnecting a client? The way it should work is

- The server tells the client to disconnect

- No response from the client because they are not there

- Wait X minutes, no response so drop the connection

Data and client should be irrelevant.

I agree completely. However, it seems to me that this occurs when the users get disconnected or they force quit FileMaker while in the midst of a long script or find, or while waiting on an external data source. The server doesn't seem able to determine that the client is no longer connected when the client has not completed certain tasks it initiated.

Another odd but related fact: If I close an unrelated database in Admin Console, the server will flush out all the disconnected clients from the problem database. Weird! But this only works a few times, then it's back to the usual fix.

[gary clark]

Wow, this is what is happening for  me using the latest filemaker server advanced 12.

Can someone tell me how they implemented a custom menu with a function for pasting trimmed text? I am not sure how i would clean the text before pasting it. Or would you clean it as you pasted.

Thanks.

[gary clark]

When setting up a custom paste menu, would you say it is important to paste without type to deal with this kind of problem?