SafetyNet GIANT log files

[merlyn383]

I just migrated to a new server (as part of the upgrade to  FMS 17) and as such had to reinstall SafetyNet. This morning the FMS process terminated abnormally because I ran out of free space. In diagnosing that I found that SafetyNet created 46GB of log files in the 6 days that it's been installed. Each day is progressively larger than the previous: 1.1GB, 4.5GB, 6.8GB, 9.1GB, 11.3GB, and 13.6GB.

I'm sure that some of this is likely because I was accidentally including 13.6GB in external container storage files in the backups (can't wait to see that AWS bill) but that log progression still seems crazy.

Is this a bug or just a side effect of my mistake?

Edited June 28, 2018 by merlyn383

[ryan360Works]

Hi merlyn383,

That large amount if files could definitely cause a lot more logging that normal. The backups run every 5 minutes so if SafetyNet has to parse through all of those files, I could see the logging being quite verbose. I assume you have already deleted the log and removed the external container storage from the backup. Do the logs seem to be growing that much now?

[merlyn383]

Hi Ryan,

I haven't had a chance to get SafetyNet to run yet. When the disk filled up the FMS DB process terminated in some weird way that hosed the FMS installation. So I reinstalled that, reinstalled SafetyNet and MirrorSync, and am currently getting errors from SafetyNet about database lock acquisition failures

Offsite backup failed while getting list of remote files: java.sql.SQLException: Database lock acquisition failure: lockFile: org.hsqldb.persist.LockFile@1a9bc3ad[file =C:\Program Files\360Works\SafetyNet2_data\SyncData\safetynet_hsql.lck, exists=true, locked=false, valid=false, ] method: checkHeartbeat read: 2018-06-28 20:04:49 heartbeat - read: -6781 ms.

Back to the logs, are those informative logs or necessary for operation? If informative, any chance for an option to reduce verbosity?

[ryan360Works]

There is a logging.properties file that controls the logging level but there are 4 different components that each have their own logging level and without seeing which component(s) is the reason for the excessive logging, I can't tell you which one(s) to turn down. Of course you could always turn off all logging but I would not recommend that because if you run into any issues, we wouldn't be able to trouble shoot without logs. Once SafetyNet is back up and running I would be curious to see if the issue happens again and if so take a look at the logs. If you can grab them before they get overly huge, they should compress to a much smaller size.

As for the file locking stuff, I suspect there might be some residual effects from the HD filling up during a backup that has cause some issues in the SafetyNet sync data. Delete the .lck file that is referenced in the file path in the error message and see if that clears the issue.