Windows Server Bash / FTP - or Automated FTP sync option?

[Ocean West]

I am reworking my process of using FMP server side script.

1. Run Server Scheduled Script every 45 minutes
2. Log in to FTP get directory listing (plugin / curl)
3. Create Record for each file in listing
4. Fetch file via (plugin / curl) to text field
5. Write file to server's Document folder
6. Move file on FTP to sub directory (compete)
7. Import files from document folder in to table

It works but think kludgy and FMS probably requires more overhead to deal with the FTP plugin.

Is there a way to do the following

1. Run a server sequence script every 45 minutes
2. Run a bash script on server login and fetch new files from ftp (need help with this)
3. Save files to FMS Documents/incoming
4. Run script that gets incoming files
5. Run script that imports files
6. Run script that moves files to complete directory
7. Run bash script with file listing as parameter (need help with this)
8. Bash script logs in to FTP moves files to complete folder on FTP OR DELETES them (need help with this)

I am hoping this would be much more efficient for FMP to only have to do file imports vs having to also FTP. 

Thanks SD

[Wim Decorte]

Why not use the "insert from URL" with the FTP cURL options?  No plugins needed.

[Ocean West]

The problem  with using insert from url I would still have to capture the file as text or binary then write it to the os just to perform an import. 

And then at some point move or delete the remote files  

My thought was in FMP only have to get directory contents and then import and move. 

 

I also couldn’t find examples of the curl syntax to move or remove files 

[Wim Decorte]

There's going to be a fair number of moving parts, whichever way you do it, by sticking with "insert from URL" you keep the moving parts in FM's space with fewer outside dependencies.  And better error trapping and handling.

 

[Ocean West]

So I was able to make connection get file lists and loop thru and download files, with curl.

Been googling all morning can't find the correct settings for the following...

• seems like every call i have to pass user name & password - (plugins seem to have a starting / ending routine)
• is there a disconnect command
• can't figure out how to move / rename / delete remote file
• with curl there is no direct way to output file to desktop still need a field or variable to hold the raw data.

 

[OlgerDiekstra]

Depending where the ftp server is located maybe forgo ftp and mount the remote system to a folder (using SMB or NFS or whatever is available, and more secure) and use BaseElements to manipulate the filesystem? You could permanently mount it or mount on demand in the script (also using BE).

Much simpler than messing with ftp. That's so last century.

Edited June 24, 2019 by OlgerDiekstra

[Wim Decorte]

Or even write your own little mini web service (microservice) to deploy on that remote machine.  We do that a lot in the case where files get produced from one system and need to be imported into FMS.

Real-world example:

- "other system" drops thousands of XML files to a server through an MQ messaging system

- small custom Windows service watches the folder and every time a file gets added, parses it and adds it to a small Sqlite db (all very lightweight but it can handle thousands of files being dropped in the watched folder at once).  This service could have just turned around and push the data to FMS through the Data API, XML API, PHP API or ODBC/JDBC but we wanted the freedom to de-couple both mechanisms (receiving the files, and adding the data to FMS), so we added:

- small custom web service (ASP.NET in this case but could be any technology) basically provides a GET endpoint that will read from the Sqlite database and send all the data in JSON format to the caller and then delete it from the Sqlite db (so that there is no persistent storage in  sqlite)

- over on the FM side there is a schedule that calls that web service every 5 minutes.

This whole setup was built to replace a standard FM import-and-process routine that was too slow to keep up.  It's blazingly fast.

Because we chose to add that Sqlite db in the middle, we are free to shut down FMS for as long as we need to, without interrupting the incoming flow of data from the other system.