Jump to content
Ocean West

Windows Server Bash / FTP - or Automated FTP sync option?

Recommended Posts

I am reworking my process of using FMP server side script.

  1. Run Server Scheduled Script every 45 minutes
  2. Log in to FTP get directory listing (plugin / curl)
  3. Create Record for each file in listing
  4. Fetch file via (plugin / curl) to text field
  5. Write file to server's Document folder
  6. Move file on FTP to sub directory (compete)
  7. Import files from document folder in to table

It works but think kludgy and FMS probably requires more overhead to deal with the FTP plugin.

Is there a way to do the following

  1. Run a server sequence script every 45 minutes
  2. Run a bash script on server login and fetch new files from ftp (need help with this)
  3. Save files to FMS Documents/incoming
  4. Run script that gets incoming files
  5. Run script that imports files
  6. Run script that moves files to complete directory
  7. Run bash script with file listing as parameter (need help with this)
  8. Bash script logs in to FTP moves files to complete folder on FTP OR DELETES them (need help with this)

I am hoping this would be much more efficient for FMP to only have to do file imports vs having to also FTP. 

Thanks SD

Share this post


Link to post
Share on other sites

Why not use the "insert from URL" with the FTP cURL options?  No plugins needed.

Share this post


Link to post
Share on other sites

The problem  with using insert from url I would still have to capture the file as text or binary then write it to the os just to perform an import. 

And then at some point move or delete the remote files  

My thought was in FMP only have to get directory contents and then import and move. 

 

I also couldn’t find examples of the curl syntax to move or remove files 

Share this post


Link to post
Share on other sites

There's going to be a fair number of moving parts, whichever way you do it, by sticking with "insert from URL" you keep the moving parts in FM's space with fewer outside dependencies.  And better error trapping and handling.

 

Share this post


Link to post
Share on other sites

So I was able to make connection get file lists and loop thru and download files, with curl.

Been googling all morning can't find the correct settings for the following...

  • seems like every call i have to pass user name & password - (plugins seem to have a starting / ending routine)
  • is there a disconnect command
  • can't figure out how to move / rename / delete remote file
  • with curl there is no direct way to output file to desktop still need a field or variable to hold the raw data.

 

Share this post


Link to post
Share on other sites
Posted (edited)

Depending where the ftp server is located maybe forgo ftp and mount the remote system to a folder (using SMB or NFS or whatever is available, and more secure) and use BaseElements to manipulate the filesystem? You could permanently mount it or mount on demand in the script (also using BE).

Much simpler than messing with ftp. That's so last century.

Edited by OlgerDiekstra

Share this post


Link to post
Share on other sites

Or even write your own little mini web service (microservice) to deploy on that remote machine.  We do that a lot in the case where files get produced from one system and need to be imported into FMS.

Real-world example:

- "other system" drops thousands of XML files to a server through an MQ messaging system

- small custom Windows service watches the folder and every time a file gets added, parses it and adds it to a small Sqlite db (all very lightweight but it can handle thousands of files being dropped in the watched folder at once).  This service could have just turned around and push the data to FMS through the Data API, XML API, PHP API or ODBC/JDBC but we wanted the freedom to de-couple both mechanisms (receiving the files, and adding the data to FMS), so we added:

- small custom web service (ASP.NET in this case but could be any technology) basically provides a GET endpoint that will read from the Sqlite database and send all the data in JSON format to the caller and then delete it from the Sqlite db (so that there is no persistent storage in  sqlite)

- over on the FM side there is a schedule that calls that web service every 5 minutes.

This whole setup was built to replace a standard FM import-and-process routine that was too slow to keep up.  It's blazingly fast.

Because we chose to add that Sqlite db in the middle, we are free to shut down FMS for as long as we need to, without interrupting the incoming flow of data from the other system.

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


  • Similar Content

    • By Cable
      Is anybody else seeing an issue with exporting to .xlsx in FMP 18?
      I have a scripted find/export that has never been a problem before. But, ever since I updated to FMP 18 I sometimes get errors with the Excel spreadsheet that is created.
      As you can see in the attached, if I make my changes and hit save it says I have to Save a Copy (if I use the same name it saves over the top of the open file just fine) because the original spreadsheet was an "older version of Excel".

       
      If I open it and try to save immediately, I get an error saying a "new version [is] available".

       
      I am using FileMaker Pro 18 on a Windows 10 machine and have Office 365 Premium.
    • By Buckie
      Hi folks. I'm thinking of upgrading an FMS server, a physical machine that I own and administer, so I wanted some input on whether FM Server is going to benefit from more cores, as for example found in AMD's Ryzen CPUs or whether I should instead opt for less cores but with more performance per core (Intel). Does it utilize threads extensively when many clients are issuing requests, and if yes, has it been like that since, say, version 12 or is it just a recent development?
      And also, is this behavior similar in both macOS and Windows or are they different?
    • By Tondats
      Hello FM Experts!
      Does FileMaker can create a Queuing System?
      Here's an example of a Queuing System: 
      Do you think I can create a Queuing System using FileMaker?
    • By Jarvis
      I currently spend about $60 per month to a company that hosts my FMP 14 databases on a shared server.
      How complicated would this be to manage on my own?   I am not interested in hosting sites for anybody else, just those for my own company.
      What kind of dedicated computer gear would I need to make this happen?  Would I need to have a dedicated IP address?
      Can this be done by a rube like myself or would it be better to continue paying someone else for this work?
    • By Jarvis
      How complicated is it to migrate solutions written under FM Pro 14 to FMP 18?
      Will I need to rebuild them from 14 to 15 then 15 to 16 etc?
×
×
  • Create New...

Important Information

By using this site, you agree to our Terms of Use.