FileMaker Server and Veeam Backup ( Virtual Snapshots )

[Josh Ormond]

I am working on adjusting configuration of a server and setting up my backup strategy ( or better, retooling it ).

 

We currently have a couple of files on a Virtual Server ( Windows Server 2012 ). FileMaker Server 13v5.

 

The IT department is using Veeam Backup & Replication to back up the whole server image. Has anyone had any experience with problems with the files where snapshots are being taken to backup the image of the server in a virtual environment? I'm particularly concerned about data corruption...not so much about connection issues. since they run everything after hours.

 

Could you give me examples of the problems you have had? Did you find a safe way around snapshot causing corruption?

[Wim Decorte]

Unless FMS has been put in a paused state at the time of the snapshot, the snapshot backups will not yield valid FM files.  When you restore a file like like that and open it, it will state "improperly closed".  Best case the file will need to pass a consistency check, worst case the file will be reported by FM as damaged and will not open.

 

Volume and machine snapshots are a good way of quickly restoring a machine with its software and settings, but they are no good for backing up live FM files.

 

If you can be sure of the timing for the snapshots then you can use the FMS command line to pause the files and resume them after the snapshot.  This is old-school so all user interaction to FMS will be suspended until the snapshot is done.  Depending on how long that takes, users will see the coffee cup cursor icon.

 

I recently did a proof-of-concept along those lines using Windows VSS snapshots on the volume that had their FM files, triggering the snapshot from the command line after putting FMS in pause and resuming afterwards.  For a 300GB volume, the whole process took 4-5 seconds.  That may be acceptable or not acceptable depending on the business needs.

[Richard Fincher]

Interesting answer. We don't presently run FMS on a Virtual Machine at the moment, but I have thought about doing so in the future.

[Wim Decorte]

The issue is not limited to VMs.  Any backup software that uses the Windows VSS mechanism will have the same issue.

[Josh Ormond]

I'm not using the snapshots as my main backup. I was mainly concerned about corrupting the live files themselves.  The backups will be pulled/copied into an network drive and backup run there to a tape drive.  

 

I will look at pausing the files while that happens. I may not have another choice if the snapshot causes corruption.

[Dave Allen]

I have been backing up our 2012 r2 server using veeam to do incremental backups daily not pausing filemaker files and not creating snapshot been doing the for about year now with no problems and use to do same with server 2008 with filemaker server on it we use backup removable drive which I have take home and restored to my home server to work on had no corruption

[Josh Ormond]

My company is specifically using snapshot. I just needed to make sure it wouldn't corrupt the files.

[Wim Decorte]

 had no corruption

 

If you read my post, I am NOT saying that you WILL have corruption.  But that you RISK corruption.  It's a proven unreliable backup method for the live files.  The fact that it has not resulted in corruption in individual cases does not make it a safe mechanism.

[Steven H. Blackwell]

I have been backing up our 2012 r2 server using veeam to do incremental backups daily not pausing filemaker files and not creating snapshot been doing the for about year now with no problems and use to do same with server 2008 with filemaker server on it we use backup removable drive which I have take home and restored to my home server to work on had no corruption

Don't mistake absence of evidence for evidence of absence of a high level of risk.  Wim is 100% correct in issuing a strong cautionary note about snapshot type activities.  I would go even further to say that snapshots should occur only when the databases files are closed.

 

Steven

[IdealData]

I had a long battle with  my support company who provide multiple VMs and they insisted on snapshots to support their support contract.

 

I have machines that run 24 hours and they were getting disconnected because the network card was being shutdown during the snapshot resulting in broken connections.

 

Eventually we abandoned continuos snapshots and backups, except for the occasion when I need to refresh the snapshot after software updates.

[CaroJo]

Thanks for this thread - I have a similar requirement from a client who wants to take full snapshot nightly as part of their backup strategy (FM12 server running on Windows Server 2008 R2 VM).

I heed the words of caution re need to close all databases - but should the server be stopped as well?

 

[Josh Ormond]

Ideally, if you can do it yes. You should be able have them include something to gracefully shut everything down.

My understanding of the snapshots, it pauses the server, takes the image of the drive, and then resumes the drive. I'm in a position I can get the files paused. Which is what Wim mentioned above.

I'm not sure if the extra effort to shutdown the database server ( FMS engine ) is worth it or not. I'll leave that answer to someone more knowledgeable about that than myself.

[Wim Decorte]

but should the server be stopped as well?

 

​It does not need to be.  Pausing the files is enough.  But if you can stop the files, then that's better.

Either way you will want to make sure that whatever mechanism you use, you do some checking to verify that the files were actually paused / closed before proceeding.

[kmtenor]

Josh: 

We are also using Veeam, and are in need of putting a proper solution in place for suspending our databases.  Can you confirm with your IT department whether they simply put the "FMSADMIN PAUSE" command into the Veeam backup Pre-Freeze script for your server, or if they wrote something more sophisticated?  Obviously, they would have to also put an "FMSADMIN RESUME"  into the Post-thaw script, as well. 

I believe the Veeam setting for "Require successful script execution" should be sufficient to determine that the files are actually paused, but I would take other suggestions on that front, as well.

Any information your IT department is willing to share would be helpful.  The struggle we are having is to determine the exact right commands to put in place, and how to implement them properly so that we don't cause more problems than we solve.  Once we have the databases pausing for backup properly, we're hoping some of the issues we're having with our database server (14.0.2 on 2008 R2/VMWare) will be resolved.

Thanks.

-Kevin

 

[Wim Decorte]

The more checking you can do the better.  You can use your favorite OS-level scripting language to execute the fmsadmin command and check for the result.

 Once we have the databases pausing for backup properly, we're hoping some of the issues we're having with our database server (14.0.2 on 2008 R2/VMWare) will be resolved.

What are the issues?

 

 

[Josh Ormond]

Basically Kevin.

• fmsadmin pause
• give enough time for the files to pause
• fmsadmin list files -s
• verify the status of all files ( 'Paused' )
• once all databases are paused, wait until the snapshot is complete.
• fmsadmin resume
• fmsadmin list files -s
• verify the status of all files ( 'Normal' )

Edited September 14, 2015 by Josh Ormond

[kmtenor]

Thanks for that info, Josh.  That's a great insight into the right process.

So, do your guys run the snapshot backup manually every time, or do they have the procedure you outlined above automated somehow?  Clearly, we can issue the pause and restart commands automatically - it's double-checking that everything actually paused that is proving tough to automate.

Thanks.

-Kevin

 

 

[kmtenor]

What are the issues?

We are currently tracking two issues we believe may be caused by our lack of following best-practices where snapshot backups are concerned:

1. We have a database that has phantom records. These records are messed up in such a way that we are facing a complete database rebuild in order to resolve them.  We are holding off on doing that, however, until the backup best-practices are in place.  This issue was discussed in greater depth on another forum.

2. We are seeing an issue with clients not being kept up to date when records change at the server.  Rebooting the server seems to solve the issue, which (in our opinion) rules out issues with the network.  While there may be underlying issues with the server itself (bad install, OS-level problems, etc), our current course of action is to clean up the backup procedure before taking measures such as reinstalling and migrating to Server 2012, etc.  We currently have a case open with FileMaker concerning this issue.  Cleaning up the backup process is our next troubleshooting step with them.

So, yes, we agree with you that running VSS backups without properly stopping FM Server (which, it cannot be stated enough, is NOT VSS-aware) is dangerous.

-Kevin

[Josh Ormond]

I believe they set up a VBScript to run those commands. I had given them a basic outline and they set it up. The snapshot runs nightly at 8pm. The script to pause the files is done at 7:45pm. If the databases do not pause for some reason, or the script times out, I get an email. I'm not sure how it works in detail, I delegated that part if it, and just confirmed it was working after they set it up.

What else are you running on that server? Anti-virus, other backup software? Is it a file server?

[kmtenor]

Thanks, Josh.

On the theory that you get nothing when you ask for nothing... is there any chance your server folks would share a generalized version of their VBScript?  If not, I understand.

Our server is just for FM, no AV, nothing else running on it.  We do run Symantec NetBackup in the environment (on a different server), which DOES do file-level backup.  Even though we're planning to remove Symantec as a backup solution, we might switch back to that temporarily to backup the filemaker-produced backup directory nightly while we figure out how to automate and properly test the veeam/fmsadmin process ourselves.

 

Thanks again.

-Kevin

[Josh Ormond]

They won't let me share the file as it is now. However, I can look and see if I have the basic outline. I have a feeling it was a sample file someone else sent me.

 

Is NetBackup touching the live FM files? If yes, that is likely your source of corruption.

[kmtenor]

Understandable, Josh.  I hope you agree it was at least worth asking.  Thanks for looking to see if you have a sample.  I will do some more research, as well.  We have 70 databases on this server, so checking that they all have stopped could be an interesting bit of fun.  Yes, we have discussed splitting this server into two (or more) within the environment.  We have a site license and plenty of VM Head-room.

No, NetBackup was never touching the served databases themselves.  It appears as though at one point we were backing up one "RC_Data_FMS" directory that was in the live folder, but there's no need to do that any more.  NetBackup backups haven't run in several months, as we've been relying on Veeam even without the fmsadmin pause script (yes, I know, bad on us), as we know the backups directory will always be intact, even in a snapshot.

-Kevin

[Josh Ormond]

No worries at all. If I can find the file, it would have been compliments of Wim's help that I have it in the first place.

Checking for the paused files is just a matter of making sure all of the files are 'paused'. Sometimes the easiest way to do that is to just look for a "normal" status. Then you know something is open.

It's been quite a while. So the chances are I don't have the original file anymore...but I will definitely look for it.

[Wim Decorte]

Here's a general VBscript that uses the fmsadmin command line.  Its purpose is to do a graceful shutdown of FMS with lots of checking along the way (but no error reporting of any kind - that needs to be added based on your requirements)

 

' Author: Wim Decorte
' Version: 2.0
' Description: Uses the FileMaker Server 10 command line to disconnect
'               all users And close all hosted files
' 
' This is a basic example.  This script is not meant as a finished product,
' its only purpose is as a learning & demo tool.
'
' This script does not have full Error handling.
' For instance, it will break if there are spaces in the FM file names.
' The script also does not handle infinite loops in disconnecting clients
' or closing files.
'
' This script is provided as is, without any implied warranty or support.

Const WshFinished = 1
q = Chr(34)  ' the " character, needed to wrap around paths with spaces


'--------------------------------------------------------------------------------------------
' Change these variables to match your setup

theAdminUser = ""
theAdminPW = ""
pathToSAtool = "C:\Program Files\FileMaker\FileMaker Server\Database Server\fmsadmin.exe"

'--------------------------------------------------------------------------------------------


SAT = "cmd /c " & q & pathToSAtool & q & " " ' watch the trailing space

callFMS = SAT
If Len(theAdminUser) > 0 Then
	callFMS = callFMS & " -u " & theAdminUser
End If
If Len(theAdminPW) > 0 Then
	callFMS = callFMS & " -p " & theAdminPW
End If

listClients = callFMS & " list clients"
disconnectClients = callFMS & " disconnect client -y"
listfiles = callFMS & " list files -s"
closeFiles = callFMS & " close file "
stopServer = callFMS & " stop server -y -t 15"

' hook into the Windows shell
Set sh = WScript.CreateObject("wscript.shell")

' get a list of all clients and force kick them off
clientIDs = getCurrentClients()
clientCount = UBound(clientIDs)

' loop through the clients and kick them off
If clientCount > 0 Then
		fullCommand = disconnectClients
		Set oExec = sh.Exec(fullCommand)
		' give FMS some time and then requery the list of clients
		Do Until oExec.Status = WshFinished
			WScript.Sleep 50
		Loop
		Do Until clientCount = 0
			WScript.Sleep 1000
			Debug.WriteLine "Waiting for clients to disconnect..."
			clientIDs = getCurrentClients()
			clientCount = UBound(clientIDs)
		Loop
End If

' get list of files and close them
fileIDs = getCurrentFiles()
fileCount = UBound(fileIDs)

' loop through the files and close them
If fileCount > 0 Then
	Do Until fileCount = 0
		fullCommand = closeFiles & fileIDs(0) & " -y"
		Set oExec = sh.Exec(fullCommand)
		' give FMS some time and then requery the list of files
		Do Until oExec.Status = WshFinished
			WScript.Sleep 50
		Loop
		fileIDs = getCurrentFiles()
		fileCount = UBound(fileIDs)

	Loop
End If

' all clients and files stopped
' shut down the database sever (does not stop the FMS service!)
fullCommand = stopServer
Set oExec = sh.Exec(fullCommand)
Do Until oExec.Status = WshFinished
	WScript.Sleep 50
Loop

' done, exit the script
Set sh = Nothing
WScript.Quit
' ------------------------------------------------------------------------------

Function getCurrentClients()

	tempCount = 0
	Dim tempArray()
	Set oExec = sh.Exec(listClients)
	
	' in case there are no clients...
	If oexec.StdOut.AtEndOfStream Then Redim temparray(0)
	
	' read the output of the command
	Do While Not oExec.StdOut.AtEndOfStream
		strText = oExec.StdOut.ReadLine()
		strText = Replace(strtext, vbTab, "")
		Do Until InStr(strtext, "  ") = 0
			strText = Replace (strtext, "  ", " ")
		Loop
		If InStr(strText, "Client ID User Name Computer Name Ext Privilege") > 0 OR _
			InStr(strText, "ommiORB") > 0 OR _
			InStr(strText, "IP Address Is invalid Or inaccessible") > 0 Then
			' do nothing
			Redim temparray(0)
		Else
			tempClient = Split(strtext, " ")
			tempCount = tempCount + 1
			Redim Preserve tempArray(tempCount)
			tempArray(tempCount-1) = tempClient(0)
		End If
	Loop
	
	getCurrentClients = tempArray

End Function

Function getCurrentFiles()

	tempCount = 0
	Dim tempArray()
	Set oExec = sh.Exec(listfiles)
	
	' in case there are no files...
	If oexec.StdOut.AtEndOfStream Then Redim temparray(0)
	
	' read the output of the command
	Do While Not oExec.StdOut.AtEndOfStream
		strText = oExec.StdOut.ReadLine()
		strText = Replace(strtext, vbTab, "")
		Do Until InStr(strtext, "  ") = 0
			strText = Replace (strtext, "  ", " ")
		Loop
		If InStr(strText, "ID File Clients Size Status Enabled Extended Privileges") > 0 OR _
			InStr(strText, "ommiORB") > 0 OR _
			InStr(strText, "IP Address Is invalid Or inaccessible") > 0 OR _
			Left(strtext, 2) = "ID" Then
			' do nothing
			Redim temparray(0)
		Else
			tempFile = Split(strtext, " ")
			status = LCase(tempFile(4))
			If status = "normal" Then
				tempCount = tempCount + 1
				Redim Preserve tempArray(tempCount)
				tempArray(tempCount - 1) = tempFile(1) & ".fp7"
			End If
		End If
	Loop
	
	getCurrentFiles = tempArray

End Function

 

Edited September 14, 2015 by Wim Decorte

[kmtenor]

THANK YOU, WIM!

That was exactly the "hint" we needed.  

Thanks again.

-Kevin