Downloading / monitoring logs

[Peter Wagemans]

Because I write a lot of server side scripts and often using custom WPE urls, one of the most annoying things in FileMaker Server 17 is that it is now impossible to monitor the logs from the admin console. I always have to download them. I found an alternative way to monitor them, by using FTP. With some nifty perl scripts, I can now even tail the Event.log and Access.log.

It's a whole other story for the other logs, like the wpe.log, the Stats.log... It is just impossible to download those over FTP. Earlier I already noticed they don't play nice with FSEvents on the Mac either.

Has anyone here created a good solution to monitor those files, given that:

• I have only access to the FileMaker Server using FTP (Implicit SSL) and the admin console, no SSH, RDP or other management tools.
• I do have the possibility to install plug-ins on the server.

Maybe some of you created some server side script that is able to monitor those files. Or you have some other solution.

[HOnza]

If you can install plug-ins on the server then you can either use a file manipulation plug-in, such as SimpleFile, or a plug-in that can run shell commands, such as Toolbox, along with some useful command line tools. On a Mac I found a combination of tail, grep, and expand the most powerful.

[Peter Wagemans]

Thanks for the reply Honza. Pity you are not really offering a tested solution, but you did manage to squeeze in two links to your site. So this at least this works for you 😊 that makes one of us.

It is kinda creepy to hear that the only way this is going to work, is not supposed to work. Server side scripts should be sandboxed and they clearly are not.

[OlgerDiekstra]

If server side scripts were sandboxed, that would severely limit what you could do. Plugins like BE for instance can access network drives from the server (which I use extensively) and that wouldn't work any more. In fact, I would have to resort to less secure methods. For instance, I allow users to upload pdf documents to the server which are then stored on a NAS. Only the server needs access to those documents. If scripts were sandboxed, I have a challenge with remote users that are not on the same network as they would need to access the NAS remotely in order to drop files there.

And that's just one example. There's no doubt a whole range of functionality that will stop working if scripts (on the server) were sandboxed.

[Peter Wagemans]

True, there would be a whole bunch of nice things you cannot do anymore (the way you do them). But there would also a number of very awful things an attacker wouldn't be able to do either.

You could do what you mention in another way. I is possible to share the Documents folder, even over a firewall. You can also install other software on the server that synchronises files to the sandboxed folder.

If you have full access to the FileMaker Server machine, there are other ways to do things. Maybe not always, but it might surprise you how many things can be done.

On the other hand, consider this situation:

A developer is working on a solution, which is hosted on a FileMaker Server. This is a contractor, and is not supposed to have access to the operating system in any way, this should only be possible by the IT people of the company. The current situation allows the developer to use the server side plug-ins ( that he requires for the development ) to gain access to parts of the OS that should be restricted to him. This is no imaginary situation. I have multiple setups like that - and I am the contractor...:-)

At least, lowering security to allow for the current situation should be an option, off by default. If you really know what the consequences are, it is then your call of judgement to allow that.

[OlgerDiekstra]

On a cloud based system (ie AWS) I'd agree. There's usually very little point to having OS access, as there's not much to do anyway (ie, connecting to internal systems would only be possible via a VPN or such and incur a lot of latency which makes access to local system from the local client more beneficial).

However, on a local server, access to the OS has many benefits, and sandboxing would definitely get in the way more than not. One way local IT can protect themselves is by virtualizing the FM server and only using a dedicated FM server that the dev can tinker on.

IT also has the option of restricting the FMS user to what they can do and access. The FM user doesn't have to be an admin user, so it's up to IT to set things up to their liking.

[Peter Wagemans]

There is something I would like to add to your remark.

A standard installation of FileMaker Server on Windows sets the service to be executed by the system account, which is the default for every service on Windows. When you set the user to another account though, the FileMaker installer refuses to install if that user does not have administrative privileges.

You need that alternative account, if you want to have network access, since the system account does not have network access, and you cannot grant this either. So I presume you are currently running the service under an alternative user account, in order to access your NAS.

I wonder WHY FIleMaker enforces the service to run under a administrative account. It does not do that on macOS.

7 hours ago, OlgerDiekstra said:

IT also has the option of restricting the FMS user to what they can do and access. The FM user doesn't have to be an admin user, so it's up to IT to set things up to their liking.

So... I had to explain this first, before reacting to you last paragraph. It is indeed possible to restrict the FMS user, and things seems to run fine, after you remove the fmserver account from the admin group. But I'm not sure this will cause misbehaviour somewhere later down the line. On Windows. Because the installer does not like it. I would like to pursue this further and ask the question to FileMaker Inc.

If this would run without problems, this would be already a great first step in the sandboxing process, and the fmserver account used during server side OS calls would not be able to attack the rest of the OS in such a direct way, while still being able to access networked drives.  It would still be able though to read and even write every file in the FileMaker Server folder. We recently had a 45 minute "hacking" session during .fmp in Berlin, where we quickly found out which file to modify in order to get full admin console access. This is very ugly. IT could restrict access to some files to read-only, though, but that would require a lot of knowledge about the inner workings of FileMaker Server, and I do not think they would be able to plug every security hole.

7 hours ago, OlgerDiekstra said:

sandboxing would definitely get in the way more than not. 

Yes. But that is also the purpose of sandboxing.

Quote

One way local IT can protect themselves is by virtualizing the FM server and only using a dedicated FM server that the dev can tinker on.

But not always. And I think virtualising does not add any benefit, except for what e.g. ESX offers to isolate the VM into a DMZ, and have snapshots in case things go south. It's also a matter of responsabilities. I do not want to have full access to a FileMaker Server, even when the IT people who are managing that server are ... my collegues. They do not have the full access password to the development, and I do not have full access to the FileMaker Server. And this is the way it should be. It would be a great setup for a (limited access) development server, but not for a production one. But then again, you can run a VM dev server on your own machine as well.

I think it also depends on who your customer, and what the entire setup is.  As I mentioned before, it should be an option, and whoever is responsible should be able to choose to do things the hard - and secure way, or not. If you are the big boss over everything, including development and IT, that is a big difference with a corporate setup.

Coming back to my original question: I would really like to monitor those extra logs. Indeed, the only way to do so, is to make use of the unsecure setup FileMaker Server has to today, and use a server side plug-in to manipulate those logs so I can start monitoring them. I presume FileMaker is well aware of these issues, so I think it's not worth it to invest time into some monitoring tool, that would probably cease to work when they close things up. I hope that log monitoring makes it way back again into the admin console.

 

[OlgerDiekstra]

11 hours ago, Peter Wagemans said:

There is something I would like to add to your remark.

A standard installation of FileMaker Server on Windows sets the service to be executed by the system account, which is the default for every service on Windows. When you set the user to another account though, the FileMaker installer refuses to install if that user does not have administrative privileges.

You need that alternative account, if you want to have network access, since the system account does not have network access, and you cannot grant this either. So I presume you are currently running the service under an alternative user account, in order to access your NAS.

Local System account most definitely has network access. My FMS runs with the local system account. Whenever I need access to the NAS though, I check whether drives are mapped or whether the local system account is logged on to the NAS. I use BE to check if I have access to the NAS. Then I use BE to log on to the NAS if I have to, invoking the 'Net Use' command. Once logged in, that session remains active until the server is rebooted.

I have the option to either map a drive (via BE or using a batchfile that's invoked via BE, I also run maintenance scripts on the NAS triggered by FM, so that I can pass parameters back and forth), or just use a UNC.

11 hours ago, Peter Wagemans said:

I wonder WHY FIleMaker enforces the service to run under a administrative account. It does not do that on macOS.

MacOS and Windows are very different beasts. OSX was originally based on BSD and still carries a lot of similarities to Linux/Unix platforms. Windows bears no resembles to any of those. Security in nix based platforms was part of the original design, Windows security is much more an after thought than by design, the first implementations of UAC we're horrible, and even now it's still not as good as on nix platforms. The UAC concept is good, but the implementation on Linux based platforms (I don't use unix platforms so can't comment much on current OS's) is still way better.

11 hours ago, Peter Wagemans said:

But not always. And I think virtualising does not add any benefit, except for what e.g. ESX offers to isolate the VM into a DMZ, and have snapshots in case things go south. It's also a matter of responsabilities. I do not want to have full access to a FileMaker Server, even when the IT people who are managing that server are ... my collegues. They do not have the full access password to the development, and I do not have full access to the FileMaker Server. And this is the way it should be. It would be a great setup for a (limited access) development server, but not for a production one. But then again, you can run a VM dev server on your own machine as well.

I think it also depends on who your customer, and what the entire setup is.  As I mentioned before, it should be an option, and whoever is responsible should be able to choose to do things the hard - and secure way, or not. If you are the big boss over everything, including development and IT, that is a big difference with a corporate setup.

Coming back to my original question: I would really like to monitor those extra logs. Indeed, the only way to do so, is to make use of the unsecure setup FileMaker Server has to today, and use a server side plug-in to manipulate those logs so I can start monitoring them. I presume FileMaker is well aware of these issues, so I think it's not worth it to invest time into some monitoring tool, that would probably cease to work when they close things up. I hope that log monitoring makes it way back again into the admin console.

Virtualization of systems is not just useful in DMZ scenarios. With todays hardware and virtualization technologies, it's trivial to run a pair of virtual hosts (VMWare, or other technologies, I use ESX and VirtualBox extensively), run a pair of DC's on both, then have your FMS running on either and other virtual systems as needed on just two physical servers. If one dies, non-essential systems can be paused, and all critical systems fail over to the remaining host. Users need know nothing.

That gives IT the time to repair or replace the failing host. Hardware upgrades are much easier as there's no downtime required. Just add another host, migrate VMs across and done.

Resource utilization is much better on virtualised systems, you need less cooling, power, space. There's a lot of benefits to virtualizing internal systems. And you can easily isolate a server and limit access to other systems. Just because the FMS is running as a local system account, doesn't mean it has unlimited access to other systems. Far from it.

11 hours ago, Peter Wagemans said:

We recently had a 45 minute "hacking" session during .fmp in Berlin, where we quickly found out which file to modify in order to get full admin console access. This is very ugly. IT could restrict access to some files to read-only, though, but that would require a lot of knowledge about the inner workings of FileMaker Server, and I do not think they would be able to plug every security hole.

I don't see how that's significant. If you have physical access to a system its usually game over anyway. Can you see the EAR password once you have full admin access? If the database is not encrypted at rest, you don't need full admin access, just copy the database. If it does use EAR, and you can't get the password through the admin console, what's the point? The DB is still encrypted. Even if the DB doesn't use EAR, you would still need passwords to get to the data. Granted, without EAR you might be able to discover those passwords in the DB, if you know where to look, though I don't think FileMaker makes it easy for you.

Just because you can compromise a system means nothing if you can't do something with it.

As a developer you need to ensure the server is setup in such a way to ensure its secure and as tamper proof as can be. That goes for the server setup as well as the database. IT needs to ensure that the developer can do everything he/she needs to on the FM server. They can restrict (using all sorts of tools) what you can and can't do, even as an admin. You don't necessarily need to log on to the FMS as admin to access the console, you can use a restricted user or access the console remotely.

IT can use tools such as Cisco Observable Networks to keep an eye on what's happening on the network, the right network kit (ie Cisco Meraki) will also make their lives a lot easier, and with the right AV protection on desktops and servers and system management tools (ie SCCM or Meraki System Manager), they can keep on top of things very well.

I am both a developer and IT Manager. While sandboxing would certainly be good to have as an option, I don't think it's a high priority. This might change if FM servers are compromised more frequently, but I haven't heard of any instance where that was the case. And if servers aren't compromised then security is good enough. Or maybe there's just not a high interest in the hacking community to compromise FileMaker systems.

 

 

[Peter Wagemans]

 

6 hours ago, OlgerDiekstra said:

I don't see how that's significant. If you have physical access to a system its usually game over anyway.

The hack could be done with a server side plug-in. I don't want to put words in your mouth, but you seems to be pointing out that when someone malicious has access to such plug-ins running on a server it's "game over". Which is exactly my point.

6 hours ago, OlgerDiekstra said:

While sandboxing would certainly be good to have as an option, I don't think it's a high priority

Let's agree to disagree on the priority. If you don't need the option, it's logical that it has no priority for you.

6 hours ago, OlgerDiekstra said:

This might change if FM servers are compromised more frequently, but I haven't heard of any instance where that was the case. And if servers aren't compromised then security is good enough. Or maybe there's just not a high interest in the hacking community to compromise FileMaker systems.

Now you have. Must we conclude now that if servers are comprised, securety is not good enough? How frequent do you have in mind?

I know an effort has been done since 2010 to increase security. So this is not completely fair of me, I admit. But do we have to have a few sucessfully compromised servers before the obvious security holes are plugged?

Your last sentence is very correct. If the hacking community would have an interest, we would be screwed already.

[OlgerDiekstra]

2 hours ago, Peter Wagemans said:

The hack could be done with a server side plug-in. I don't want to put words in your mouth, but you seems to be pointing out that when someone malicious has access to such plug-ins running on a server it's "game over". Which is exactly my point.

If someone is capable of uploading a plugin, then yes, it could be 'game over'. But how would someone upload a plugin? For clarities sake, when I talk about hacking, I assume this hacker has no normal access to systems, and tries to gain access through vulnerabilities or (common) misconfigurations.

For someone to install a plugin, they would either need remote access to the server through RDP or similar (IT responsibility to restrict that), drop it in the correct folder through a share on the C: drive (again IT responsibility to ensure the C: drive doesn't get shared more than it should (I wouldn't even enable the admin share, but that breaks some stuff), or through the admin console.

In all cases strong passwords and (network) security will mitigate those options. Monitoring network will enhance visibility of what goes on. The only other option would be physical access, which can easily be prevented with a locked room.

Granted, a lot of this is moot when the offender is a (disgruntled) employee with admin access. But even that can be mitigated.

Good security (especially nowadays) is mandatory anyway. My network is guarded by Meraki kit (router, switches, and AP's). They all talk to one another. They have a great dashboard that gives me plenty of power to ensure my network stays save. Meraki security devices have Snort capabilities that investigate traffic back and forth in realtime, it scans for virusses/malware in traffic that passes through its ports.

In addition, I have Observable Networks installed, that silently monitors my network and has now built a very accurate baseline of network traffic. I've had a few alerts of someone poking at my network. The thing with Observable Networks is that it alerts you almost immediately and empowers you to take action before they even get a chance to get a foot in the door.

Noone in my network is an administrator on their machine. I won't allow it. AV is up to date. Backups are made nightly of every important file, multiple times, 'offline' as well (offline being in the cloud). All fully automated. Regularly checked of course.

That mitigates the chances of something happening on my network. And if something does happen, at least I've got a decent chance of recovering unscathed.

2 hours ago, Peter Wagemans said:

Let's agree to disagree on the priority. If you don't need the option, it's logical that it has no priority for you.

It's not that I don't need the option, it's that I don't really see the need for it. And maybe FileMaker is of the same opinion where there are so many ways to protect your systems, that sandboxing doesn't really solve anything. But I am not opposed to the idea.

2 hours ago, Peter Wagemans said:

Now you have. Must we conclude now that if servers are comprised, securety is not good enough? How frequent do you have in mind?

Ow, that's not a real hack. Someone stuffed up maybe, but reading through the thread someone just panicked, called it a hack and then started looking at what happened. Maybe it was a hack, who knows, the OP never returned to finish the thread.

2 hours ago, Peter Wagemans said:

I know an effort has been done since 2010 to increase security. So this is not completely fair of me, I admit. But do we have to have a few sucessfully compromised servers before the obvious security holes are plugged?

Looking at MS, Apple, name any vendor, I'd say yes, that's how things generally go. And not just tech companies. Takata has been in the news of late, making dodgy airbags. Apparently, profit was worth a few lives. If companies can get away with it, some will certainly try. That doesn't mean I approve of it, far from it, but unfortunately that is the world we live in.

I think FileMaker can be regarded as a good company/citizen, although I have no proof to show for that. But even then, they have to look at desired features and determine which will have the biggest impact, not just for their profit margins, but also for their developers and customers.

By all means, suggest it as a feature. Who knows, they may not have considered it.