Jump to content

jbante

Members
  • Content count

    603
  • Joined

  • Last visited

  • Days Won

    32

jbante last won the day on July 16

jbante had the most liked content!

Community Reputation

140 Excellent

About jbante

  • Rank
    member

Profile Information

  • Gender
    Not Telling
  • Location
    California

FileMaker Experience

  • Skill Level
    Expert
  • FM Application
    16 Advanced

Platform Environment

  • OS Platform

FileMaker Partner

  • Certification
    7
    9
    10
    11
    12
    13
    14
    15
  1. Virtual list using JSON

    I have now. The 16.0.2 update does not change the performance characteristics of the JSON functions relative to return-delimited lists or repeating variables.
  2. Virtual list using JSON

    The JSON functions in FileMaker 16 have constraints the SortValues and UniqueValues do not. The JSON parsing functions check that the input they're trying to parse is in fact completely valid JSON. FileMaker has to do this to be able to return an error result when appropriate. You can't know if the outer-most structure in a JSON string is correctly terminated without starting at the beginning, checking every inner structure, and getting all the way to the end. Due to how the JSON format works, this requires stepping through the entire string for every single operation. (This could be resolved by FileMaker using a more efficient internal data structure for JSON in place of the raw string, but performance tests from a few folks suggest that FileMaker isn't doing this.) In comparison, the GetValue function only has to parse as many newline characters as it takes to get to the value you want, and it can ignore everything in a string after that point. With repeating variables, there doesn't need to be any parsing at all (depending on how you do it) — you can just use the whole content of the variable (repetition). The JSON writing functions have a similar constraint. They validate any JSON you had to start with, and then the JSON writing functions are somewhat fastidious in organizing the contents of their JSON outputs. You can make parsing nested structures in JSON less slow by parsing out inner structures, then getting the details from the inner structures, rather than pulling from the outer structure for every inner detail. This reduces how much total work FileMaker spends validating the JSON for each read. When speed is more important than simple code, do this: Let ( [ _dictionary0 = JSONGetElement ( $json ; 0 ) ; _0a = JSONGetElement ( _dictionary0 ; "a" ) ; _0b = JSONGetElement ( _dictionary0 ; "b" ) ; _dictionary1 = JSONGetElement ( $json ; 1 ) ; _1a = JSONGetElement ( _dictionary1 ; "a" ) ; _1b = JSONGetElement ( _dictionary1 ; "b" ) ] ; ... ) Not this: Let ( [ _0a = JSONGetElement ( $json ; "[0].a" ) ; _0b = JSONGetElement ( $json ; "[0].b" ) ; _1a = JSONGetElement ( $json ; "[1].a" ) ; _1b = JSONGetElement ( $json ; "[1].b" ) ] ; ... ) And for writing JSON, add as much in each JSONSetElement call as you can, rather than using many separate calls. Do this: JSONSetElement ( "[]" ; [ "[0].a" ; 1 ; JSONNumber ] ; [ "[0].b" ; 2 ; JSONNumber ] ; [ "[1].a" ; 3 ; JSONNumber ] ; [ "[1].b" ; 4 ; JSONNumber ] ) Not this: Let ( [ _json = JSONSetElement ( "[]" ; "[0].a" ; 1 ; JSONNumber ) ; _json = JSONSetElement ( _json ; "[0].b" ; 2 ; JSONNumber ) ; _json = JSONSetElement ( _json ; "[1].a" ; 3 ; JSONNumber ) ; _json = JSONSetElement ( _json ; "[1].b" ; 4 ; JSONNumber ) ] ; ... ) While these patterns make FileMaker's JSON functions less slow, it isn't enough to make them competitive with return-delimited lists or repeating variables for speed.
  3. Virtual list using JSON

    Technically, it could be done. As Wim mentioned, performance is a major constraint. The JSON functions are slow compared to the built-in functions for return-delimited lists, and very slow compared to repeating variables.
  4. "The San Diego Workforce Partnership (SDWP) seeks a contractor to support our team of FileMaker developers in the development, maintenance, and improvement of SDWP’s custom software suite." Quotes are due by 16 June 2017 at 5 p.m. PDT. http://workforce.org/rfps/2017/06/06/rfq-filemaker-pro-consulting-services
  5. FM 16 Variable Repetition limitation change?

    There are two types of solutions here: solutions with no risk of collision, and solutions where the risk of a hash collision can be resolved so that it has no negative consequences.
  6. FM 16 Variable Repetition limitation change?

    The conversation did start with windows, but you're right. I did get fixated on one example. Field names are another example where the capacities of the 16- or 24-bit approaches are still perfectly acceptable. Several high profile "big data" database-like tools are perfectly comfortable using mere 64-bit IDs for globally unique values, and they only went with 64 because it fit conveniently with an Int64. I'm perfectly comfortable following their lead with a 56-bit hash method because it fits conveniently in 10^17. But heaven forbid we should settle for such a small ID space that it can only uniquely identify every square meter on the combined surfaces of the first 5 planets in the solar system. My next CRM is really going to need to keep 6 planets in repeating variables at any given time. I'll have to upgrade my computer to 19.6 petabytes of RAM just to keep track of the memory addresses for all the repetitions. You're right that a serial number with a 3 letter prefix is guaranteed to cause a hash collision after 10 records with Mod ( Code ( Table::id ) ; 1e17 ), which is why everyone has been recommending against that and discussing alternatives for several days. Even when you do get a hash collision, so what? We can just do the same thing as any of several perfectly workable solutions in widespread use by other hash tables. Hash table collisions are a solved problem. But heaven forbid we should have to be confronted with yet another problem the internet is happy to give us a feast of solutions to, apparently.
  7. FM 16 Variable Repetition limitation change?

    For small collections of data, yes, JSON is an obvious convenient solution. For large arrays, repeating variables have a large performance advantage. I haven't seen many FileMaker apps doing anything where the performance of repeating variables would override the convenience of JSON (but those are certainly the most fun!). No solution can have enough windows to track information on for the performance difference between JSON and repeating variables to be a significant bottleneck, because the fact that there are so many windows would always be the bigger problem.
  8. FM 16 Variable Repetition limitation change?

    The work behind the cryptographic hash algorithms (plus MD5) built-in to FileMaker didn't really have hash tables in mind, but their properties are still useful. One of the information-theoretic design goals of cryptographic hashes (or at least a necessary consequence of those goals) is that any subset of the bits should be an equally good hash as any other equal-sized subset. This is fundamental to what makes a hash useful for any cryptographic or error detection purpose. If working with only a "buzz-sawed" fraction of the full cryptographic hash doesn't preserve the uniform probabilistic behavior, it was a bad hash to start with. It's because of the work making the hashes good that I'm comfortable with the risks. 10^17 / 16^32 is a pretty small number. That doesn't make 10^17 (or 2^24) a small number.
  9. FM 16 Variable Repetition limitation change?

    But the number of simultaneous windows during a single session is exactly what's relevant to the quality of a hash based on window names. A collision across different sessions is inconsequential here. With 24 bits, you need to be working with 184 window names simultaneously for the probability of a collision to reach 0.001. 20 is a gratuitously large number of windows for a single file to have open; the probability of a collision is about 1 in 100,000 in that case. I'm comfortable with that. Given that window names tend to be systematically calculated, the practical probability may be substantially lower after developer testing. Since developers have the option to check if a hash bucket is taken and try a slightly different window name instead, I don't think it's a big deal. If you really want to go overboard and use the full ~56.5 bits we get in 10^17, the probably of a collision among 20 windows is 1.9e-15. You can have 447,214 windows before the probability of a collision gets to one in a million. 20 users with 20 windows (with new names) each every day will take more than 3 years to find a collision. Hash tables are a solved problem, and the solution is not to have 2^128 buckets. The solution is to work with however many buckets you have.
  10. FM 16 Variable Repetition limitation change?

    So what's the threshold of acceptability for this application? "Acceptability" is relative to the problem being solved, not relative to the other possible solutions (some of which, as you pointed out, aren't even possible solutions anymore). What's acceptable is what fits in a variable repetition number, which apparently is 10^17. I think even less than that could be acceptable. We're talking about what amounts to a hash table of simultaneous open windows during a single session here, not a primary key that needs to be unique across the entire universe for a few decades. 10^17 buckets is a gratuitously large hash table for any application, and even just 16M is plenty of headroom for window management. I really think 24 bits is fine.
  11. FM 16 Variable Repetition limitation change?

    A cryptographic (or any other) hash is "playing the roulette", too. 10^17 is still ~56.5 bits of information, which is more than most database's primary keys ever need, more information (entropy) than most passwords represent, and surely overkill a-plenty for humble window names. You mentioned a challenge with the full content of any of the built-in hash functions being larger than this 56.5 bits, so I threw out the option of a different hash that might be easier to fit in the constraints of the situation. Granted, Mod ( Code ( Get ( WindowName ) ) ; 10^17 ) turns out not to capture anything close to that 56.5 bits in practice... Interesting! That especially makes sense if your window names tend to start with the same 4 characters, which I imagine is pretty common. The Code function puts the first characters in a string as the least significant digits in a number, and Mod is then taking those same least significant digits. Left, on the other hand, is taking the most significant digits, corresponding to the end of the string which is more likely to be different. I hadn't thought of that! Left ( ... ; 17 ) is still only working with the information from a fixed range of 4 characters, which seems like a likely liability. My guess is that window names share their last 4 characters less often than their first 4 characters, but that it still happens frequently enough that we might be uncomfortable promoting this as a habit anyone could use any time. To get a full 56 bits of information out of a full window name, the most efficient way to go may well be to use one of the built-in hash functions like comment suggested (md5 is built-in before v16 and doesn't require a second function to convert it to hex!), truncate to the first (or last, it doesn't matter) 14 hex characters, and convert the hex to decimal: NumberFromHex ( Left ( GetContainerAttribute ( Get ( WindowName ) ; "md5" ) ; 14 ) ) Alternately, if you really don't like custom functions, are happy to require FileMaker 16, and expect to have significantly fewer than 16,777,216 windows open at once, you could use the new CryptDigest function, convert it to base 64 (giving you 6 bits of information per character of the base 64 representation), and do Mod ( Code ( ... ) ; 10^17 ) on that to get 24 bits of meaningful information reflecting the full window name: Mod ( Code ( Left ( Base64Encode ( CryptDigest ( Get ( WindowName ) ; "md5" ) ) ; 4 ) ) ; 10^17 ) You could also replace the Mod ( ... ; 10^17 ) in this case with Right ( ... ; 17 ), but I do think you'd be better served with Right than Left in this case. Right ( ... ; 17 ) will retain 24 bits of information value, whereas Left ( ... ; 17 ) will lose some. The last character (in left-most position after Code) may have a 3-digit instead of a 2-digit code, but the right-most 2 of those 3 digits don't overlap with the codes of any other characters in base 64. With Left ( ... ; 17 ), you risk losing a more informative digit from the code of the first character.
  12. FM 16 Variable Repetition limitation change?

    See if Mod ( Code ( Get ( WindowName ) ) ; 10^17 ) is unique enough.
  13. FileMaker 16 introduced a collection of built-in functions for manipulating data serialized as JSON. This makes it easier for FileMaker applications to interact with many web services. This will also make JSON the de facto standard format for scripts within FileMaker to pass parameters and results to each other, improving code sharing within the FileMaker community. JSON does not have a broad palette of scalar data types to choose from: text, number, boolean, and null. Even with those, FileMaker's JSONGetElement function always returns a text result, even when the serialized JSON value is a number or boolean. So I made a handful of custom functions and scripts for sending and receiving typed data with JSON. The module is hosted on GitHub, or you can download it directly.
  14. You definitely can make encapsulated or generalized logic in FileMaker. You can't do it the same way you would in Java or Haskell, but Java and Haskell can't encapsulate or generalize logic the same way the other does, either. FileMaker's abilities seem about on par with C in this respect. Every useful programming environment has its building blocks, such as in a standard library; and the contents of that standard library dictate much of how programs written for that environment operate, both internally and with the outside world. The operating environment or standard library is often a more significant contribution to the value of a programming language than the logical structures it uses. Java is popular because of the portable virtual machine, despite being awful as a language. JavaScript is another example. C(++) is popular because its operating environment is close to the silicon, despite that making it more difficult to encapsulate certain highly consequential concerns. FileMaker scripting starting with options available in the user interface is just one reasonable choice of building blocks to start with, and a pretty inspired choice, designed to minimize the learning curve for "citizen developers". In a Darwinian sense, FileMaker's choice is successful. (The script engine also includes many steps that have no counterpart in the user interface.) Every Turing machine has its tape, and failing to embrace the nature of that tape will doom a developer to never using that machine well.
  15. That's not a meaningful distinction. "Slow compared to other programming environments" doesn't make FileMaker slow in any absolute sense. "Slow compared to user needs for a particular application" is much more meaningful, and once an application is fast enough to exceed user needs, pursuing additional speed may be wasted effort. FileMaker passes this threshold for many applications, including applications working with millions of records. Working with large data sets efficiently with FileMaker may take additional work, which makes it no different from any other programming environment, and achieving the best possible performance from FileMaker often takes less effort than achieving the best possible performance from other tools. At least FileMaker doesn't claim that processing speed is an absolute top priority, then make design decisions contradicting that every step of the way — just look at what the HPC community has to say about Hadoop.
×

Important Information

By using this site, you agree to our Terms of Use.