'scp' fails within shell called from FMSA

[Savery]

I've written a little shell script which essentially takes a backup of a FM database (from passed arguments). The last step is to scp the ZIPd file back to our remote server.

Testing the script from the command line works a treat. We get a lovely fresh copy of the database archived and sent to our remote server using scp.

However, if I use 360Works SM to call this shell script with the same parameters it fails when attempting the scp command. Note: I am calling the FileMaker script, which uses SM to call the shell script, as a scheduled task in FileMaker Server Advanced 10. There is a very good reason for this, and this means that, unfortunately, this cannot be changed.

I'm 99% certain that this is not a SSH problem as I've used keygen to share the keys and this is proven to work time and again from the command line.

Have I hit some weird limit of SM?

I'm happy to post snippets of the code that I'm using, if you feel it'll help you diagnose the fault.

TIA, Steve

[fseipel]

Can you set ssh LogLevel to verbose and redirect standard output AND standard error to a file under OSX?

i.e.

-o LogLevel verbose

and

>& error_log or for multiple commands, may wish to append >>& error_log

I've had similar issues in Windows, but not with SCP; a batch files run when double-clicked, but fails when run on a schedule by Windoze. Typically this problem has related to upgrades from XP to Vista wherein mapped drives don't exist in the emulated DOS session and so full UNC paths must be specified. Or it's caused by Windoze running these scripts as a different user with different permissions. I've fixed these and other problems by logging standard output and error from all commands to a file; this method should allow diagnosis on OSX also, by reviewing the resulting files.

If you're just running a *single* command in SM, the standard output should be returned to FM; does your SM script also return standard error?

i.e. you may want to concactenate process.getInputStream().getText() AND process.getErrorStream().getText() and return THAT result to Filemaker rather than only standard output. This may reveal the error without having to import logfiles. The demo that comes with SM for shell script, only returns standard output and not standard error, this is why I bring this up. Since many errors go only to standard error, this is a critical distinction.

[Savery]

If you're just running a *single* command in SM, the standard output should be returned to FM; does your SM script also return standard error?

i.e. you may want to concactenate process.getInputStream().getText() AND process.getErrorStream().getText() and return THAT result to Filemaker rather than only standard output. This may reveal the error without having to import logfiles. The demo that comes with SM for shell script, only returns standard output and not standard error, this is why I bring this up. Since many errors go only to standard error, this is a critical distinction.

All I was getting back was 'ERROR', which, while true, wasn't particularly informative. :

I have add the -v arg to my scp line in the shell script and will report back as soon as I can.

[Jesse Barnum]

We recently had a bug report that is probably relevant in this case - the do shell script in ScriptMaster is overly simplistic for handling spaces in shell commands. It does not correctly handle quotes or backslashed escape sequences, so they get treated as multiple arguments.

The solution is to tokenize the shell command string ourselves instead of just passing it unmodified to the Runtime class, because the Java Runtime class is where the bug lies (it just breaks everything out by spaces), and then pass in a command array instead of a full string.

I have this on my todo list for ScriptMaster, but can't promise any delivery date at the moment. If you're handy with Java/Groovy, you can take a look at the Java StringTokenizer class and modify the do shell script module yourself.

[Savery]

We recently had a bug report that is probably relevant in this case - the do shell script in ScriptMaster is overly simplistic for handling spaces in shell commands. It does not correctly handle quotes or backslashed escape sequences, so they get treated as multiple arguments.

I 'made' a new instance of the 'do shell' function so that I could pass parameters separately:

RegisterGroovy( "ExecuteBackup( FMAdmin ; FMPass ; FMDatabase ; DGUser ; DGServer ; LocalVol )" ; "String[] cmd = ["./dg_backup.sh", FMAdmin, FMPass, FMDatabase, DGUser, DGServer, LocalVol ];¶

Process process = Runtime.getRuntime().exec( cmd );¶

String processResult = process.getInputStream().getText();¶

process.waitFor();¶

exitValue = process.exitValue();¶

return processResult;" )

Does this look like it could cause a problem? The shell script called uses double-quotes around all variables.

[Jesse Barnum]

I think that looks OK. Remember one more thing, which is that shell scripts called from Java do not have the environmental variables that would normally be set in your .profile file. This is not usually a problem as long as you fully qualify your command names (ie. /bin/ls as opposed to ls).

[Savery]

I think that looks OK. Remember one more thing, which is that shell scripts called from Java do not have the environmental variables that would normally be set in your .profile file. This is not usually a problem as long as you fully qualify your command names (ie. /bin/ls as opposed to ls).

Right, so the 'scp' command in my shell script would need to be a fully-qualified path. I'll give that a try, although I should add that other commands before 'scp' all run fine without full-paths.

The shell itself starts with the #!/bin/bash which I assumed, rightly or wrongly, would give the system enough to know where everything lived.

One more thing that I've not been sure of: who is running the commands? Would it be logged-in user of the server, or fmserver user? I can't remember now whether I've allowed for either or both of these when modifying the sudoers list as I've been going round in circles lately.

Thanks for your input - very very welcomed! :

[Jesse Barnum]

I think that the command will run as the 'fmserver' user if you're running it from a scheduled shell script. However, the best way to find out is to call '/usr/bin/whoami' from the script itself, which will return the username.

[Savery]

The scripts are indeed being run as 'fmserver'. Now I just need to determine whether this user can scp. Could be that this is the problem. All other tasks are fmsadmin commands, bar one (which is independent of user).

[Jesse Barnum]

By the way, the problem of parsing white spaces in shell commands is fixed in ScriptMaster 4. I would recommend switching to that if you have not done so already.