Insert From URL may be broken when moving to 13

[Brooks]

I make heavy use of the insert from URL script step along with the RestFM server for posting to another file and running sync scripts. Something has changed with the way 13 handles insert from URL along with the GetAsURLEncoded. When using strings that work fine in 12 will simply not work in 13. I have called FM tech and they are looking into it. By simply removing the GetAsURLEncoded function the strings will work fine in 13. Yes, I know that could cause problems, but at least it works as a temporary solution for anyone else who finds this to be broken in their solutions.

[Wim Decorte]

It's my understanding that the "insert from URL" always automatically does a URL encoding.  (Unlike the web viewer where you can turn that automatic behavior off).

 

So I'm actually very surprised that it used to work in 12 by adding the GetAsURLencoded.

[Brooks]

I haven't heard that before Wim. That would explain why it works fine without the encode step though. I picked up that behavior from using Goya's RESTfm server. Nick uses the encode step before all of the insert steps when posting to the RESTfm server in all his stock scripts. Do you now of anywhere that is documented? I have been in touch with fm tech support and according to them the new behavior is a bug.

[Wim Decorte]

Nick at Goya has a good write-up of the automatic encoding behavior with the "insert from URL" script step:

 

http://www.goya.com.au/blog/insert-url-and-encoding

 

The article is now slightly dated because "insert from URL" does support HTTP POST, but the encoding behavior has not changed.

So despite what FI support says, it is not a bug.

[Brooks]

Thanks Wim. I hadn’t read that. I check ed in with Nick and this was his response below.

 

The result will be that RESTfm 3.0 will be required to sync or send data from FMP 13. There is a change to the way the InsertFromURL step encodes the data it’s sending, which means the data received is different at the server end when sent from 12 vs 13. 

So in v3, we’ve got a whole new data method, and to that we’ve added a new fix for the v13 encoding. The way it’s all architected, it’s going to be difficult to back port this into RESTfm 2.

[DanShockley]

I would definitely classify automatic encoding with no option to turn it off as a bug. Here's an example of where the automatic encoding behavior makes it impossible to submit a search string to a PHP page that includes a leading equals sign to do a search similar to that done in FileMaker (exact word match): 

 

If you are crafting a URL that has key-value pairs and the value for one of them needs to be "=Bob" (without the quotes) for example, you cannot submit that. So, your URL would look like this: 

http://server.example.com/search.php?FirstName=%3DBob

That would result in the PHP doing a search for "=Bob" (without the quotes). However, the Insert From URL script step automatically encodes the % in %3D, which means that THIS is what is sent: 

http://server.example.com/search.php?FirstName=%253DBob

That would result in the PHP doing a search for "%3DBob" (without the quotes) and FAILING to find anything.  

So, you might instead say to yourself "Oh, maybe I shouldn't encode it myself, since Insert From URL does that for me," and use this URL: 

http://server.example.com/search.php?FirstName==Bob

If you do that, Insert From URL sees nothing to encode. OK. Fine. Then, most reasonable web services that are tokenizing that URL would see the second equals sign as indicating the END of the value, so the PHP would get NULL as the value for FirstName and the "=Bob" part would be discarded as superfluous. The search would thus be empty, either returning an error or every record (not just Bob's), depending on your search PHP code. 

So, the automatic encoding of Insert From URL is a serious liability in that it makes it impossible to submit certain (not rare!) data using it. I really hope FileMaker decides to fix this. The easiest way to fix it would be to add an option to Insert From URL to turn off automatic encoding so that developers can handle the encoding themselves in these kinds of situations where automatic encoding is guaranteed to cause failure. That would also mean we'd avoid any incompatibility with existing scripts. 

[Wim Decorte]

 

So, the automatic encoding of Insert From URL is a serious liability in that it makes it impossible to submit certain (not rare!) data using it. 

 

Fully agreed.  It makes using Insert From URL with the new HTTP POST very difficult when the url contains characters that are valid but that the script step decides to URLencode.