External container storage file path

[eyeguy]

I am trying to set up a container field in a hosted solution running under filemaker server.

The container field will hold primarily pdf's and I want the file to be stored on the server and streamed to the user.

This file is accessed on a local network but also remotely.

I am not sure how to set up the filepath.

Firstly I do not seem to be able to do it unless the file is local on my machine i.e not across the network. Is this correct?

Secondly I do not know what the syntax should be. The File will live in a folder called "Letters which is on a drive called "Documents" and the server is called "mecserver".

Remotely I can mount the drive by using the Connect to server on OSX and typing in afp://xxx.xxx.xxx.xxx (IP address)

Does this mean that the file path is afp://xxx.xxx.xxx.xxx/Volumes/Documents/Letters?

Joseph

[Wim Decorte]

Once the file is on the server, the file path will always be a subfolder of the folder where the FM file itself is. You can not choose a folder that is somewhere else on the server or somewhere else on the network.

[eyeguy]

Acyually my question was purely about the correct syntax for the filepath

[Wim Decorte]

But it is a moot question. Streaming remote container data only works if the file is hosted through FMS, and if the file is on FMS you can not have the container data anywhere else but in a subfolder of where the fmp12 file itself is. That's automatic, you can not change it.

So even if you figured out the correct syntax and set it in the local file, as soon as you upload the file to FMS, that path will be disgarded.

[eswanborg]

WOW - I just discovered this as well. What a huge hole to what could be a great new feature. This really makes External Containers relatively useless to our company. We're using plug-ins already to store containers on our fileserver where we have much better back-up and control - not to mention leaving the FMServer to do only what it's supposed to - server databases.

If we had to build out our FMS machines to not only server databases but also store all the container data we'd be sunk.

Really a shame that something that works really well locally, is completely and by design, broken when hosted by FMS.

Bummer.

[Wim Decorte]

Yes and No. I agree that when you look at it strictly from a "document management" perspective it falls very short. Not even close to what you can do with SuperContainer. I would have to blame their marketing efforts here though because it was never really touted (but also never debunked) as a document management feature.

there are some tremendous benefits to it, especially with the changed backup behavior of hard-linking to unchanged files. So if your remote container data does not change between backup schedules FMS does not spend time making new copies of it, just adds a new hard link to the hard disk space it occupies. That's a huge performance benefit.

But for now you have treat the external container data files as if they were still inside FM.

[Reed]

So this backup behavior could be a huge lifesaver for a 800 GB document database I just created via ODBC import. It has 10's of 1000's of containers which upon import are stored directly in the file, so I have an 800 GB fmp12 file. Are there pitfalls to the transfer process I should be aware of when transferring such a large amount of data to external storage?

[Klypto]

I wanted to reply to this thread since it was the last one I happened upon in my search for better document management and manipulation with Filemaker Server (on windows) and External Container fields. It is also the only thread that I was able to find that directly deals with the issue of external container storage directory on the server. It was discouraging to find that you can't change the default directory of the storage folder when hosting the solution on a server.



However, I did manage to get it to work through the use of directory hard links on the server (or in the OP's case of a networked dive, symbolic links) which is a file/folder path concept that is similar to table occurrences in a database.

In my case using windows server, I wanted the server to use print (email through google cloud print), email PDF's without having to purchase 3rd party plugins, and still have the files be accessible from a public shared read-only network folder on the server. Since the server can only effectively work within the documents and temporary paths, the file storage directory under the Databases path would have been useless.

I got fed up with the silly notion that the files can only be stored in that one location and be inaccessible to the server (and potentially others), so I copied a backup of the contents of the folder in Filemaker Server/Data/Databases/RC_Data_FMS and deleted the RC_Data_FMS folder.

I then used the command line utility "mklink" to make a hard link called RC_Data_FMS to the Filemaker Server/Data/Documents folder:

mklink /H "C:Program FilesFileMakerFileMaker ServerDataDatabasesRC_Data_FMS" "C:Program FilesFileMakerFilemaker ServerDataDocuments"

 

And then pasted back the files into the RC Folder.

So if you were to navigate through the RC_Data_FMS folder, the path would still look like //Filemaker Server/Data/Databases/RC_Data_FMS/[Database Name]/[Table]/[Field], but you are actually in //Filemaker Server/Data/Documents/[Database Name]/[Table]/[Field]...

Or are you?

Applications can't tell the difference and neither could you if you weren't looking directly at the icon of the linking reference folder in it's directory.


Filemaker Server can then import files and add email attachments from it's own hosted file's container items through the Documents folder. You can then use the same technique for a folder in a network drive to reference this Documents folder or have a subdirectory within the Documents folder that references a location on the networked drive. This allows people to access these same files without having to store duplicates.


For mac, these articles may help:

http://30stechie.wordpress.com/2012/02/11/redirecting-folders-in-os-x-to-a-network-drive/

http://gigaom.com/apple/how-to-create-and-use-symlinks-on-a-mac/



The security implications of this is dependent on your implementation of path referencing and the file/folder permissions you set on the server OS.

Also, please be sure to avoid creating recursive directory loops, like opening Documents folder has a sub directory that leads to the Desktop which has a sub directory leads to the Documents folder etc. The system won't stop you from doing it, but it's just totally illogical and will probably screw up search indexing as well as confuse applications and users.



It seems to work fine for me so far, hopefully there aren't any snags that I am not aware of.

[jfletch]

Klypto, what is the latest on this technique? Have you found it to be reliable?

 

I have a need to do this and can't find anything conclusive on it. Some say to use symlinks and you (and others) indicate that FMS might prefer hard links. 

 

-jonathan fletcher

[Klypto]

Sorry looking at the post now I see that I confused the terminology.   Hard links would be used for files only where as Symbolic links and Directory Junctions can be used on file directories.  

 

What I used on our server was a directory junction because we may in the future choose to allow read-only network access to invoices and document pdfs that are stored by filemaker.  Symbolic links are interpreted client side so they do not work for remote directories.

 

The command in the example should be:

 

mklink /J "C:Program FilesFileMakerFileMaker ServerDataDatabasesRC_Data_FMS" "C:Program FilesFileMakerFilemaker ServerDataDocuments"

 

 

 

 

Expanding on the original idea on that post, our system now automatically sends out dropship orders for things we don't have locally via email to other third party companies & storage with an attachment of the invoice PDF for them to put in the shipment for the customer.  It's been running by itself for about 3-4 months now without issues.

 

Do note that you may need some sort of system to keep track of versions of the file.  Changing a file in filemaker or adding one using the same name results in a number version being appended to the end of the file name as a new file on the server instead of the file being replaced.  

 

INVOICE_1570.pdf with a newly added line item will now have both the original INVOICE_1570.pdf and the changed one INVOICE_1570_1.pdf on the server.  If you want to reference the most current one in server-side scripts you need to know the version number.

 

 

Added note: I have not yet tested editing files on the server filesystem that are stored in the remote container, but I get the feeling that it would be a bad idea.  

[jon.klassen]

Klypto, you are my hero. I have been trying for too long to figure a work around for sending emails with attached PDFs through the a server side script. Your solution is working great so far

[Wim Decorte]

And it's still a bad idea.  RC data folders and files should be treated like other live FM files.  Anything you do to expose them so that other processes touch them increases the risk of the data getting corrupted).

[rkass068]

So is this same issue with filemaker server 13? I have a database hosted on FMS13 and Because my server is only about 70gb I would like the containers to store the files on a network drive. My server is connected to the network drive constantly. Can I change the filepath so that the server can store containers to the external network drive or will it default back to the subfolder on the server drive where the DB is hosted?

[Wim Decorte]

Nope.  That would be a very bad idea.  Just get a larger hard drive for your server. Or a secondary drive.

 

In FMS13 you can pick another folder for your RC data, it does no longer have to be the default subfolder of where the file is.  But it still has to be an internal drive.

 

For the obvious stability/fragility and speed reasons.  Disk I/O is on of the biggest performance bottlenecks for FMS and by using an ethernet connection (if you could) then you would throttle the i/o throughput to very slow speeds compared to what a computer can do internally.

[Matt Stec]

Why is it then that Filemaker website says: "When externally stored container data is hosted on FileMaker Server, the data can be located on a different volume from the databases, including on a remote volume. For more information, see FileMaker Server Help."

[Bailey Kessing]

Hey Wim,

Can I ask a follow up question? I seem to not be understand something. We are running a windows server with FM14 (all up to date). The databases are stored in the "usual" place (filewin:/e:/Program Files/FileMaker/FileMaker Server/Data/Databases/) but I would like the container data of one field to be stored on another local drive. I could move the database file with that container to the new location and "Enable additional database folder 1" with the new location (filewin:/f:/Databases2/) but can I just set the container to store that container data in this location? I thought with the new versions you could but maybe I am wrong…or am not getting the settings correctly set in all the various locations! 

[Wim Decorte]

On 2/1/2016 at 5:04 PM, Matt Stec said:

Why is it then that Filemaker website says: "When externally stored container data is hosted on FileMaker Server, the data can be located on a different volume from the databases, including on a remote volume. For more information, see FileMaker Server Help."

That new feature is in 14; this thread started back in 2012 when you could not have the RC data separate from the main FM file.

Also; a lot of this thread has to do with not exposing the stored RC data from *outside* FM; that's still a very bad idea.

57 minutes ago, Bailey Kessing said:

but can I just set the container to store that container data in this location? 

No, it's not an option you set at the container level, it is an FMS config that works on a number of choices:

1) first you have to enable a non-default folder for your FM files

2) then you can specify where the RC data for the FM files in that non-default folder will be stored.  And that can be on another drive on that machine

The choice applies to all files you put in the non-default FM folder.  Not for some containers in some files.

 

 

[Bailey Kessing]

Ah, Thanks Win, that is what I was finding but assumed/hoped that I was doing something wrong and that I could designate the path for the container individually. Thanks again. 

 

However to explore this a bit more…once I set up the non-default location for the databases…the RC data can then be anywhere I set in the server settings for the container data?

Edited April 5, 2016 by Bailey Kessing
completing

[Wim Decorte]

57 minutes ago, Bailey Kessing said:

However to explore this a bit more…once I set up the non-default location for the databases…the RC data can then be anywhere I set in the server settings for the container data?

That is correct; as long as it is on the same server.  You can not specify a networked folder.

[Frupper]

Hello Wim,

please is any way to use external container storage for 90000 .jpg files? If FMS transfer 90 thousand files into one folder, it is little bit crazy to have 90000 files in one folder, isn't it?

Because in this time we have all files as reference in container field (source files are on fileserver divided by thousands) and we can't use web direct for publish our database on web because the .jpg preview is not visible in browser.

Many thanks for your answer and sorry for my english.

(we have FileMaker Server 15.0.2.13 on Mac OS X platform)

Edited February 9, 2017 by Frupper

[Wim Decorte]

Using RC does not automatically mean that all files end up in one folder.  You can specify a folder structure in the container field setup.  Most commonly you would use the record ID or some other record piece of data as the subfolder name.

[Frupper]

Thank you very much Wim.

Works fine :-)