Jump to content
TJ53

Virtual list using JSON

Recommended Posts

TJ53    3

On a separate thread I asked Bruce Robertson if the new FM16 JSON functions could be used together with the virtual list technique he created. I haven’t experimented with it but I wonder if JSON adapt well to this great technique. Any thoughts? thanks!

Share this post


Link to post
Share on other sites
Wim Decorte    446

In theory: yes.  The question will come down to whether the JSON building and parsing functions are faster or slower than the regular text functions I guess.

  • Like 1

Share this post


Link to post
Share on other sites
jbante    140

Technically, it could be done. As Wim mentioned, performance is a major constraint. The JSON functions are slow compared to the built-in functions for return-delimited lists, and very slow compared to repeating variables.

  • Like 1

Share this post


Link to post
Share on other sites
TJ53    3

Thank you for the responses, I hadn't thought about the speed factor, it makes a lot of sense.

So ... are the JSON functions slow for building or parsing? I haven't tested them with big chunks of text. I see that in the recent FM16 update the speed of the "sort values" and "unique values" functions have been improved. Is the speed of the JSON functions an issue in a similar way? I haven't read anything about it ... any interesting links about this? thanks!

Share this post


Link to post
Share on other sites
jbante    140

The JSON functions in FileMaker 16 have constraints the SortValues and UniqueValues do not.

The JSON parsing functions check that the input they're trying to parse is in fact completely valid JSON. FileMaker has to do this to be able to return an error result when appropriate. You can't know if the outer-most structure in a JSON string is correctly terminated without starting at the beginning, checking every inner structure, and getting all the way to the end.  Due to how the JSON format works, this requires stepping through the entire string for every single operation. (This could be resolved by FileMaker using a more efficient internal data structure for JSON in place of the raw string, but performance tests from a few folks suggest that FileMaker isn't doing this.) In comparison, the GetValue function only has to parse as many newline characters as it takes to get to the value you want, and it can ignore everything in a string after that point. With repeating variables, there doesn't need to be any parsing at all (depending on how you do it) — you can just use the whole content of the variable (repetition).

The JSON writing functions have a similar constraint. They validate any JSON you had to start with, and then the JSON writing functions are somewhat fastidious in organizing the contents of their JSON outputs.

You can make parsing nested structures in JSON less slow by parsing out inner structures, then getting the details from the inner structures, rather than pulling from the outer structure for every inner detail. This reduces how much total work FileMaker spends validating the JSON for each read.

When speed is more important than simple code, do this:

Let ( [
    _dictionary0 = JSONGetElement ( $json ; 0 ) ;
    _0a = JSONGetElement ( _dictionary0 ; "a" ) ;
    _0b = JSONGetElement ( _dictionary0 ; "b" ) ;
    
    _dictionary1 = JSONGetElement ( $json ; 1 ) ;
    _1a = JSONGetElement ( _dictionary1 ; "a" ) ;
    _1b = JSONGetElement ( _dictionary1 ; "b" )
] ;
    ...
)

Not this:

Let ( [
    _0a = JSONGetElement ( $json ; "[0].a" ) ;
    _0b = JSONGetElement ( $json ; "[0].b" ) ;
    _1a = JSONGetElement ( $json ; "[1].a" ) ;
    _1b = JSONGetElement ( $json ; "[1].b" )
] ;
    ...
)

And for writing JSON, add as much in each JSONSetElement call as you can, rather than using many separate calls. Do this:

JSONSetElement ( "[]" ;
    [ "[0].a" ; 1 ; JSONNumber ] ;
    [ "[0].b" ; 2 ; JSONNumber ] ;
    [ "[1].a" ; 3 ; JSONNumber ] ;
    [ "[1].b" ; 4 ; JSONNumber ]
)

Not this:

Let ( [
    _json = JSONSetElement ( "[]" ; "[0].a" ; 1 ; JSONNumber ) ;
    _json = JSONSetElement ( _json ; "[0].b" ; 2 ; JSONNumber ) ;
    _json = JSONSetElement ( _json ; "[1].a" ; 3 ; JSONNumber ) ;
    _json = JSONSetElement ( _json ; "[1].b" ; 4 ; JSONNumber )
] ;
    ...
)

While these patterns make FileMaker's JSON functions less slow, it isn't enough to make them competitive with return-delimited lists or repeating variables for speed.

  • Like 1

Share this post


Link to post
Share on other sites
TJ53    3

That's some great information to take into account, thank you!!

Share this post


Link to post
Share on other sites
john renfrew    29

Jeremy, have you tested against the 16.0.2 update?? `does it improve speed??

Share this post


Link to post
Share on other sites
Mike Duncan    13

A couple things that come to mind if you go down this path...there is a limit to how many variables you can set with a Let statement, 1000. And if you use the "JSONFormatElements" function, it may re-sort your data unexpectedly. I would consider JSON for passing variables, possibly, over using for virtual lists.

Mike

  • Like 1

Share this post


Link to post
Share on other sites
jbante    140
On 7/15/2017 at 11:39 PM, john renfrew said:

Jeremy, have you tested against the 16.0.2 update?? `does it improve speed??

I have now. The 16.0.2 update does not change the performance characteristics of the JSON functions relative to return-delimited lists or repeating variables.

  • Like 1

Share this post


Link to post
Share on other sites
john renfrew    29

Thanks Jeremy..

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now


  • Who Viewed the Topic

  • Similar Content

    • By crayfish
      I've been using Daniel Smith's JSON scripts from (http://www.modularfilemaker.org/module/json/) to parse JSON.
      It's worked quite reliably (thanks Daniel!), but I'm wondering if anyone has compared its parsing speed to MBS's JSON plugin or the new native Filemaker 16 JSON functions? I'm still on 15. I'm working with some larger JSON files that are taking a long time to parse and hoping for a faster solution.
      Any data on which option parses JSON fastest? Any disadvantages/advantages to each option?
      Thanks!
    • By 357manch
      With the introduction of cURL options in FM16's Insert from URL script step I've been investigating the possibility of being able to save a copy of my FM solution directly to my Dropbox folder using the https://content.dropboxapi.com/2/files/upload element from Dropbox V2 API.
       
      I've successfully created the access token to pass through as part of this step and the Insert from URL script step successfully creates the stated folder and file in my Dropbox.  However, the file size is always 0 bytes.  I've tried 2 different methods
       
      METHOD 1
      #Export a compacted copy of this file to Temporary Folder
      Set Variable [ $file_path ; Get ( TemporaryPath ) & "Solution.fmp12" ]
      Save a Copy as [ "$file_path" ; compacted ]
       
      #Set my cURL options to variable
      Set Variable [ $curl_header ;
      "curl -X POST https://content.dropboxapi.com/2/files/upload \\" & ¶ &
      "--header \"Authorization: Bearer " & PREFERENCES::Dropbox_AccessToken & "\"  \\" & ¶ &
      "--header \"Dropbox-API-Arg: {\\\"path\\\": \\\"/DB2FM/Solution.fmp12\\\"}\" \\" & ¶ &
      "--header \"Content-Type: application/octet-stream\" \\" & ¶ &
      "--data-binary @" & $file_path ]
       
      #Dropbox URL
      Set Variable [ $dropbox_url ; "https://content.dropboxapi.com/2/files/upload" ]
       
      #Make Request - Set to variable adn then format to JSON field
      Insert from URL [ Verify SSL Certificates ; With dialog: Off ; $json ; $dropbox_url ; cURL options: $curl_header ]
      Set Field [ PREFERENCES:Dropbox_JSON ; JSONFormatElements ( $json ) ]
       
       
       
      METHOD 2
      #Export a compacted copy of this file to Temporary Folder and then import to a container field
      Set Variable [ $file_path ; Get ( TemporaryPath ) & "Solution.fmp12" ]  //note1
      Save a Copy as [ "$file_path" ; compacted ]  //note1
      Insert File [ PREFERENCES::Container ; "$file_path" ]  //note1
      #Set variable following instructions on FileMaker website which states:
      #  -  Direct file access is replaced with a FileMaker variable prefixed by the @ character.
      #  -  To access a file, you can set the variable to a container field, which contains the file, then use that variable as the parameter of the cURL option.
      Set Variable [ $container_path ; PREFERENCES::Container ]
       
      #Set my cURL options to variable
      Set Variable [ $curl_header ; 
      "curl -X POST https://content.dropboxapi.com/2/files/upload \\" & ¶ &
      "--header \"Authorization: Bearer " & PREFERENCES::Dropbox_AccessToken & "\"  \\" & ¶ &
      "--header \"Dropbox-API-Arg: {\\\"path\\\": \\\"/DB2FM/Solution.fmp12\\\"}\" \\" & ¶ &
      "--header \"Content-Type: application/octet-stream\" \\" & ¶ &
      "--data-binary @" & $container_path ]
       
      #Dropbox URL
      Set Variable [ $dropbox_url ; "https://content.dropboxapi.com/2/files/upload" ]
       
      #Make Request - Set to variable adn then format to JSON field
      Insert from URL [ Verify SSL Certificates ; With dialog: Off ; $json ; $dropbox_url ; cURL options: $curl_header ]
      Set Field [ PREFERENCES:Dropbox_JSON ; JSONFormatElements ( $json ) ]
       
       
       
      JSON Response For Both Methods
       
      I get the following JSON Response for both methods
      {
          "client_modified" : "2017-05-16T11:22:47Z",
          "content_hash" : "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
          "id" : "id:heJWQ1SWwJsAAAAAAcMWNw",
          "name" : "Solution.fmp12",
          "path_display" : "/DB2FM/Solution.fmp12",
          "path_lower" : "/db2fm/solution.fmp12",
          "rev" : "3a0412f00f58892",
          "server_modified" : "2017-05-16T11:22:48Z",
          "size" : 0
      }
       
      Any ideas as to why the file is created but with no content.  I've also tried this with a number of different file types (.png, .txt, .pdf) by manually inserting a file into the container and greying out the 3 steps marked //note1 above but the result is always the same.
       
    • By jbante
      FileMaker 16 introduced a collection of built-in functions for manipulating data serialized as JSON. This makes it easier for FileMaker applications to interact with many web services. This will also make JSON the de facto standard format for scripts within FileMaker to pass parameters and results to each other, improving code sharing within the FileMaker community.
      JSON does not have a broad palette of scalar data types to choose from: text, number, boolean, and null. Even with those, FileMaker's JSONGetElement function always returns a text result, even when the serialized JSON value is a number or boolean. So I made a handful of custom functions and scripts for sending and receiving typed data with JSON.
      The module is hosted on GitHub, or you can download it directly.
    • By bcooney
      I have a popover portal that displays a virtual list. My client would like to have two or more windows open at the same time. (ugh, i know).
      Currently, if the user has two windows each with a popover portal displayed, they are parsing the same $$vlist, and so one popover's portal is showing the wrong data.
      My options seem to be:
      find a way to make the $$vlist window specific find a way to close the first popover if the user makes a different window active (both popovers do not need to be visible at the same time) hide one popover by storing a key in the virtual list that compares the record to this key in the $$vlist, and if the key isn't there, the portal hides. (don't like this UX). Thoughts? Is there a switched window trigger??
    • By AlanTrewartha


      View File fmresultsetjson.xsl
      XSLT to convert Filemaker's REST web-service XML response into clean JSON
      https://twitter.com/alantrewartha
      2017-03-06
      Adapted from the original Six Fried Rice XSLT here http://sixfriedrice.com/wp/products/filemaker-to-json-converter/
      Changed:
      * single quotes to JSON-standard double-quotes
      * removed "fieldDefinitions" block (personal preference)
      * stopped every field value being presented as an [ array ] even when not a "repeating field"
      * added whitespace escaping code
      The white-space escaping code was entirely cribbed from
      https://github.com/doekman/xml2json-xslt/blob/master/xml2json.xsl
      Submitter AlanTrewartha Submitted 03/06/2017 Category Solutions FM Version FM Version: 0  
×

Important Information

By using this site, you agree to our Terms of Use.