mac | pc Test for open files on FM Server

[Doug Gardner]

I'm looking for the best way to test that files are open and "normal" on a number of FM Server 13 Windows deployments that a client has scattered all over the place. Note that this is testing for a positive result, not looking for a negative result (like, that a file isn't open because of problem). 

 

There are a couple of ways that come to mind: Write an OS-level script to get the filenames and status from fmsadmin at the command line, send them to a text file and deal with that appropriately. Another way to do it is have an FM script do something like check that each file is open on the host and record the results. Either way seems like more work than it should be. Is there a better way? A simple way? All help is appreciated. 

 

Doug

 

[Wim Decorte]

There is probably nothing more simple than using OS-level scripts and the fmsadmin command line. The challenge really is collecting the data in a central place especially if the servers are in different networks...

 

Not sure that I follow the distinction between a positive and negative result, can you expand on that?

[Doug Gardner]

Hey Wim, thanks for responding. 

 

Yes, I agree that the challenge is collecting the data. Ideally, the data would somehow be gathered at a central location and a routine would check each set of files against the canonical list for each location, then the results would be summarized for each server, possibly in an email. So, once or twice a day the admin would get a simple report. One thing that I just remembered, though, is that I have seen cases where a FM Server reports that a file is open, it's status is "normal", and the file isn't really available on the network. I can't remember where I've seen this, or how long ago, and I have no idea whether it's a reasonable concern. And I might be overly cautious because these aren't super-high value, "mission critical", people-might-not-get-their-medication type systems that I'm concerned with in this case.  

 

Positive results: a check show that these files are open and responding properly. Negative results: a check shows that these files are not open or not responding. FM Server can alert you to a problem, but it won't send you an alert that everything is ok. The distinction wouldn't matter if we could trust that all problems always get reported, but as a matter of logic, that's not possible because one problem could be that the reporting isn't working. 

 

On another, somewhat related note, it's a bummer that the DatabaseNames function hasn't been updated for operating on FM Server. It just returns the name of the file in which the script runs, not all the files that are running on the server. There might be a security reason for that, but I'm guessing it's just that it's an old function that hasn't been revisited in light of more recent changes.

[Wim Decorte]

 

On another, somewhat related note, it's a bummer that the DatabaseNames function hasn't been updated for operating on FM Server. It just returns the name of the file in which the script runs, not all the files that are running on the server. There might be a security reason for that, but I'm guessing it's just that it's an old function that hasn't been revisited in light of more recent changes.

 

No, it's a security concern and it is in line with the sub-admin groups that you can configure on FMS: on a multi-tenancy server (think cloud host) you do not want someone to see all the file names of the files on that server.  It's an obvious attack vector.

 

If you want a positive result on the monitoring, then the monitoring can not be done from inside FM.  If FMS is down, it can not run any schedules (FM scripts or OS scripts), so the monitoring needs to be done from outside FMS completely so that it can report if FMS is down or if the file is down.

 

Sounds like a good topic for an SOFA meeting.

Jonathan Mickleson did a good session on this at the 2013 devcon.  There are various approaches depending on the OS of the servers involved.

[Claus Lavendt]

Hi Doug

Probably not an option, but if the files has an account with XML privileges turned on you could create a monitoring solution, that performs a query on each file. If you get returned a result, then the file is open and accessibly. I would conclude that if an xml query work, then the file would also be available through a FileMaker client. Of course network issue would be a factor as well, but if all files fails on the same server, it would indicate that either the entire server has a problem or a network problem.

Such approach would of course require that each file gets an account for this and wpe is turned on for each server.

But it would be easy to create e.g. a FileMaker solution, that performs these queries and then you have a centralized place to watch status.

 

Just an idea, if it could be an option.....

[Doug Gardner]

No, it's a security concern and it is in line with the sub-admin groups that you can configure on FMS: on a multi-tenancy server (think cloud host) you do not want someone to see all the file names of the files on that server.  It's an obvious attack vector.

 

I was thinking that the function could be rewritten so that when it's run from FMS, it observes the purview of the administrator group from which the scheduled script runs. That could be a non-trivial rewrite, though, because functions are designed to operate within the client and might not be able to reach into the server environment without doing significant work. 

 

Regarding the initial post...

 

I agree that monitoring cannot be done with FMS if you want a positive result—something outside the target FMS must actively perform the checking, at least ultimately. Here's what I think I'm going to do, given that there is no web access to the server machines, but there is external FM access. All criticism is greatly appreciated. 

 

______________________________________

On the FM Server Deployments

 

Create a script (called "Heartbeat") with just one script step: Exit Script [ Result: Get ( FileName )], paste it into each file on the server (about 10 files). 

 

In one of the files (that already happens to have file references to the other files), create a script "Heartbeat_Poll_Internal" that runs Heartbeat on each of the files and records the results in a log, timestamped, etc.

 

______________________________________

On an FM Server Set to Poll the Others

 

Run a script, "Heartbeat_Poll_External", that collects the most recent log entry in each of the FM Server Deployments. 

 

The following results lead to flags for investigation:

A machine is down, (so no report).

The main file is down (the one that runs Heartbeat_Poll_Internal).

Report returned showing that one or more files are down.

 

This data can be displayed in a dashboard, and the results (positive and negative) can be emailed to the admin. 

 

_________

Pros

 

Very quick and easy to deploy, with minimal database footprint.

Uses existing software, configuration, and setup.

 

_________

Cons

 

The FM Server that is setup to poll the others could stop responding, and it's up to the admin to notice that a status report hasn't been received (there are ways to shift the locus of this type of problem, but I don't think there's a way to solve it—it leads to an infinite regress).

 

It's possible for a file to be accessible to FM Server that is not accessible to others on the network—this does not address that type of problem. 

[Doug Gardner]

Hi Doug

Probably not an option, but if the files has an account with XML privileges turned on you could create a monitoring solution, that performs a query on each file.

 

 

Hi Claus,

That's very close to the solution I've arrived at, though without the XML part (because it's not an option). If you've got a minute, I'd appreciate any critical comment on the basic method outlined in the previous post. Thanks!

[Claus Lavendt]

Hi Doug

The reason I suggested the xml path, is that it will normally timeout rather quickly, just leaving the test as no connection.

When you use a FM file, that should run a script in all files on server, you will get lost connection as you need to setup link to each file in External Data Sources. When you run a script, or even opens the script, your FM client will try to open the remote database. However, if that is not available, you will be promted to select another route to that file. If not, you will need to close the DB file on your "Monitoring client" and reopen.

Let's say you are monitoring 5 servers and the first one is down. Then your script will never get to testing the 4 others.

 

Another thing, you should take into account, is that when you call a script on a hosted file, you will probably also call the startup script, so you might need to put in an escape there as well.

 

If xml access is not an option, you can of course use a tool, such as Simon, to monitor if there is access to the server, through port 5003. This gives you information about, whether the server is running and can be accessed from where your Simon is sitting. I use this in general to monitor all my clients servers. Normally, a file is not closed, unless I do it. Of course problems can occur and close a file.

 

I would not go with the FM monitor solution, if I needed to rely on native FileMaker connection. Exactly because any disruption between the monitoring client and a server, will stop the rest of the test, due to the hard link to the hosted DB. And then you would need a monitoring solution, to monitor, if your Monitor solution has stopped......

 

I belive you could use a fmsadmin LIST FILES command on the server to get a list of all hosted files, including files managed by group administrators. (Wim could correct me there...)

I think I would write an AppleScript or VB script to through that command and post the result, either to a CWP site, that can insert the result into your monitoring solution, or by sending an email. Setup a cron script or similar on windows, to run e.g. each 5 min. or what you need. In your monitoring solution, check for intervals and if you are missing a report from a server, make an alert, so you could check the reason.

 

Just my thoughts on this. Maybe others have better ideas...

[Wim Decorte]

I belive you could use a fmsadmin LIST FILES command on the server to get a list of all hosted files, including files managed by group administrators. (Wim could correct me there...)

I think I would write an AppleScript or VB script to through that command and post the result, either to a CWP site, that can insert the result into your monitoring solution, or by sending an email. Setup a cron script or similar on windows, to run e.g. each 5 min. or what you need. In your monitoring solution, check for intervals and if you are missing a report from a server, make an alert, so you could check the reason.

 

 

Yes, I've used that in the past.  Problem with it is that you need to embed admin console credentials in the command and it is not trivial to get that done in a secure manner.

[Claus Lavendt]

Yes, I've used that in the past.  Problem with it is that you need to embed admin console credentials in the command and it is not trivial to get that done in a secure manner.

That's what I thought....

Think I would go and try hard to get xml enabled as it, to me, seems to be the best and easiest way to perform monitoring of DB files, without too many security concerns...

[dansmith65]

... If not, you will need to close the DB file on your "Monitoring client" and reopen.

 

 

Doug said he was going to be using an FM Server to monitor the other servers, so I don't think this would be an issue.

 

Doug, I actually think your method sounds pretty good. If it was me, I think I would be happy with FileMaker Server telling me that the file is open, but you already said you didn't want that, so I won't go into it.

 

There is a way to verify that the monitoring server is running as expected:

Have it run a script sequence which runs an OS level script after the FileMaker "Heartbeat_Poll_External" script you mentioned. The OS level script would need to send data (it could be a boolean value) to a web-based monitoring/alerting system like http://aws.amazon.com/cloudwatch/. I'm sure there are other services that would work, but I just used cloudwatch recently to do something similar, which is why I'm mentioning it. Cloudwatch is able to send you alerts both if the value is False (perhaps when the FileMaker script returns an error), or when no data is received at all. This second alert is the important one; if your FileMaker Server schedule is not running for any reason, then you'll get an alert because it's not able to send data to Cloudwatch.

[Doug Gardner]

Hi Doug

The reason I suggested the xml path, is that it will normally timeout rather quickly, just leaving the test as no connection.

When you use a FM file, that should run a script in all files on server, you will get lost connection as you need to setup link to each file in External Data Sources. When you run a script, or even opens the script, your FM client will try to open the remote database. However, if that is not available, you will be promted to select another route to that file. If not, you will need to close the DB file on your "Monitoring client" and reopen.

Let's say you are monitoring 5 servers and the first one is down. Then your script will never get to testing the 4 others.

 

Hi Claus,

 

I appreciate the response. Actually, the behaviour is a little bit different than you describe. There is no prompt to select another route to the missing file under a specific set of circumstances.

 

For the internal check (that is, the script running server-side, checking to see which of its files are open), as long as you're running the script with Set Error Capture [On], there is no pause and no dialogue when a file is not open or not found. The attempt to run the script in a missing file just returns an error 100.

 

For the external check (that is, the script running on the FM Server that is polling the main file on each of the other FM Servers), again as long as you're running the script with Set Error Capture [On], the script runs all the way through and either an error 100 or 802 is returned. (On FM Server, the errors are suppressed anyway, but this works without interruption on FM Pro, too.)

 

I understand that this might be unexpected behaviour. There are all kinds of ways you can end up with a dialogue asking you to locate a missing file. I should have mentioned that in this case the files in question do not have any tables represented on the relationship graph, so there is never a case where they are required to display data.

 

You brought up a great point, and made me consider how FM Server running the script would handle a case where a file was not available. How would that affect subsequent attempts? Would the file need to be closed and re-opened (because the FileMaker Way is to try once, then stop trying until the file is closed and re-opened, as is the case with FM Pro)? I've done some preliminary testing, running the script on FM Server on files hosted elsewhere, then bringing a file down, running the script (and being notified the file is down), then bringing the file back up and running the script (and being notified the file is available). So far, it's looking good. 

 

Thanks again for all the help.

[Doug Gardner]

There is a way to verify that the monitoring server is running as expected:

Have it run a script sequence which runs an OS level script after the FileMaker "Heartbeat_Poll_External" script you mentioned. The OS level script would need to send data (it could be a boolean value) to a web-based monitoring/alerting system like http://aws.amazon.com/cloudwatch/.

 

Thanks for the ideas, Dan. If I hit a snag, I might end up going the route of an OS-level script, but I'm trying to avoid it because processing the results requires bringing in other technologies and external connections, and I think its best to have the smallest possible set of dependencies—the simpler it is, the easier it is to troubleshoot, and the less it relies on being really clever. 

[dansmith65]

The purpose of the OS script I mentioned isn't to process your results, it's only to notify you if your central monitoring server stops running.

[Claus Lavendt]

 

Hi Claus,

 

I appreciate the response. Actually, the behaviour is a little bit different than you describe. There is no prompt to select another route to the missing file under a specific set of circumstances.

 

For the internal check (that is, the script running server-side, checking to see which of its files are open), as long as you're running the script with Set Error Capture [On], there is no pause and no dialogue when a file is not open or not found. The attempt to run the script in a missing file just returns an error 100.

 

 

Yeah, I really didn't read your first post again about running this off a server. In that case you are right about the behavior.

 

I think it would be ok to just test each file once in the script. At least in my experience, I have found that if FM is trying to run a script or open a file which fails after timeout, that file is really not available. So in the same sequence, it would not be needed to test several times.

If you run the server script every 2 min. and the same file fails in 2 runs, then you could say the file is down and let your monitoring process give you an alert.

You could say, that if the server is down or the connection to the server is down, then it could take some time, if you run a test on all 125 hosted files on that server. In this case, it could be helpful to ping the server first, in order to see if the server is actually running and available. However, I am not sure you can do this, without using OS script.

 

Anyway, good luck, hope my thoughts did not send you way off...