Jump to content

jbante

Members
  • Content count

    603
  • Joined

  • Last visited

  • Days Won

    32

Everything posted by jbante

  1. Virtual list using JSON

    I have now. The 16.0.2 update does not change the performance characteristics of the JSON functions relative to return-delimited lists or repeating variables.
  2. Virtual list using JSON

    The JSON functions in FileMaker 16 have constraints the SortValues and UniqueValues do not. The JSON parsing functions check that the input they're trying to parse is in fact completely valid JSON. FileMaker has to do this to be able to return an error result when appropriate. You can't know if the outer-most structure in a JSON string is correctly terminated without starting at the beginning, checking every inner structure, and getting all the way to the end. Due to how the JSON format works, this requires stepping through the entire string for every single operation. (This could be resolved by FileMaker using a more efficient internal data structure for JSON in place of the raw string, but performance tests from a few folks suggest that FileMaker isn't doing this.) In comparison, the GetValue function only has to parse as many newline characters as it takes to get to the value you want, and it can ignore everything in a string after that point. With repeating variables, there doesn't need to be any parsing at all (depending on how you do it) — you can just use the whole content of the variable (repetition). The JSON writing functions have a similar constraint. They validate any JSON you had to start with, and then the JSON writing functions are somewhat fastidious in organizing the contents of their JSON outputs. You can make parsing nested structures in JSON less slow by parsing out inner structures, then getting the details from the inner structures, rather than pulling from the outer structure for every inner detail. This reduces how much total work FileMaker spends validating the JSON for each read. When speed is more important than simple code, do this: Let ( [ _dictionary0 = JSONGetElement ( $json ; 0 ) ; _0a = JSONGetElement ( _dictionary0 ; "a" ) ; _0b = JSONGetElement ( _dictionary0 ; "b" ) ; _dictionary1 = JSONGetElement ( $json ; 1 ) ; _1a = JSONGetElement ( _dictionary1 ; "a" ) ; _1b = JSONGetElement ( _dictionary1 ; "b" ) ] ; ... ) Not this: Let ( [ _0a = JSONGetElement ( $json ; "[0].a" ) ; _0b = JSONGetElement ( $json ; "[0].b" ) ; _1a = JSONGetElement ( $json ; "[1].a" ) ; _1b = JSONGetElement ( $json ; "[1].b" ) ] ; ... ) And for writing JSON, add as much in each JSONSetElement call as you can, rather than using many separate calls. Do this: JSONSetElement ( "[]" ; [ "[0].a" ; 1 ; JSONNumber ] ; [ "[0].b" ; 2 ; JSONNumber ] ; [ "[1].a" ; 3 ; JSONNumber ] ; [ "[1].b" ; 4 ; JSONNumber ] ) Not this: Let ( [ _json = JSONSetElement ( "[]" ; "[0].a" ; 1 ; JSONNumber ) ; _json = JSONSetElement ( _json ; "[0].b" ; 2 ; JSONNumber ) ; _json = JSONSetElement ( _json ; "[1].a" ; 3 ; JSONNumber ) ; _json = JSONSetElement ( _json ; "[1].b" ; 4 ; JSONNumber ) ] ; ... ) While these patterns make FileMaker's JSON functions less slow, it isn't enough to make them competitive with return-delimited lists or repeating variables for speed.
  3. Virtual list using JSON

    Technically, it could be done. As Wim mentioned, performance is a major constraint. The JSON functions are slow compared to the built-in functions for return-delimited lists, and very slow compared to repeating variables.
  4. "The San Diego Workforce Partnership (SDWP) seeks a contractor to support our team of FileMaker developers in the development, maintenance, and improvement of SDWP’s custom software suite." Quotes are due by 16 June 2017 at 5 p.m. PDT. http://workforce.org/rfps/2017/06/06/rfq-filemaker-pro-consulting-services
  5. FM 16 Variable Repetition limitation change?

    There are two types of solutions here: solutions with no risk of collision, and solutions where the risk of a hash collision can be resolved so that it has no negative consequences.
  6. FM 16 Variable Repetition limitation change?

    The conversation did start with windows, but you're right. I did get fixated on one example. Field names are another example where the capacities of the 16- or 24-bit approaches are still perfectly acceptable. Several high profile "big data" database-like tools are perfectly comfortable using mere 64-bit IDs for globally unique values, and they only went with 64 because it fit conveniently with an Int64. I'm perfectly comfortable following their lead with a 56-bit hash method because it fits conveniently in 10^17. But heaven forbid we should settle for such a small ID space that it can only uniquely identify every square meter on the combined surfaces of the first 5 planets in the solar system. My next CRM is really going to need to keep 6 planets in repeating variables at any given time. I'll have to upgrade my computer to 19.6 petabytes of RAM just to keep track of the memory addresses for all the repetitions. You're right that a serial number with a 3 letter prefix is guaranteed to cause a hash collision after 10 records with Mod ( Code ( Table::id ) ; 1e17 ), which is why everyone has been recommending against that and discussing alternatives for several days. Even when you do get a hash collision, so what? We can just do the same thing as any of several perfectly workable solutions in widespread use by other hash tables. Hash table collisions are a solved problem. But heaven forbid we should have to be confronted with yet another problem the internet is happy to give us a feast of solutions to, apparently.
  7. FM 16 Variable Repetition limitation change?

    For small collections of data, yes, JSON is an obvious convenient solution. For large arrays, repeating variables have a large performance advantage. I haven't seen many FileMaker apps doing anything where the performance of repeating variables would override the convenience of JSON (but those are certainly the most fun!). No solution can have enough windows to track information on for the performance difference between JSON and repeating variables to be a significant bottleneck, because the fact that there are so many windows would always be the bigger problem.
  8. FM 16 Variable Repetition limitation change?

    The work behind the cryptographic hash algorithms (plus MD5) built-in to FileMaker didn't really have hash tables in mind, but their properties are still useful. One of the information-theoretic design goals of cryptographic hashes (or at least a necessary consequence of those goals) is that any subset of the bits should be an equally good hash as any other equal-sized subset. This is fundamental to what makes a hash useful for any cryptographic or error detection purpose. If working with only a "buzz-sawed" fraction of the full cryptographic hash doesn't preserve the uniform probabilistic behavior, it was a bad hash to start with. It's because of the work making the hashes good that I'm comfortable with the risks. 10^17 / 16^32 is a pretty small number. That doesn't make 10^17 (or 2^24) a small number.
  9. FM 16 Variable Repetition limitation change?

    But the number of simultaneous windows during a single session is exactly what's relevant to the quality of a hash based on window names. A collision across different sessions is inconsequential here. With 24 bits, you need to be working with 184 window names simultaneously for the probability of a collision to reach 0.001. 20 is a gratuitously large number of windows for a single file to have open; the probability of a collision is about 1 in 100,000 in that case. I'm comfortable with that. Given that window names tend to be systematically calculated, the practical probability may be substantially lower after developer testing. Since developers have the option to check if a hash bucket is taken and try a slightly different window name instead, I don't think it's a big deal. If you really want to go overboard and use the full ~56.5 bits we get in 10^17, the probably of a collision among 20 windows is 1.9e-15. You can have 447,214 windows before the probability of a collision gets to one in a million. 20 users with 20 windows (with new names) each every day will take more than 3 years to find a collision. Hash tables are a solved problem, and the solution is not to have 2^128 buckets. The solution is to work with however many buckets you have.
  10. FM 16 Variable Repetition limitation change?

    So what's the threshold of acceptability for this application? "Acceptability" is relative to the problem being solved, not relative to the other possible solutions (some of which, as you pointed out, aren't even possible solutions anymore). What's acceptable is what fits in a variable repetition number, which apparently is 10^17. I think even less than that could be acceptable. We're talking about what amounts to a hash table of simultaneous open windows during a single session here, not a primary key that needs to be unique across the entire universe for a few decades. 10^17 buckets is a gratuitously large hash table for any application, and even just 16M is plenty of headroom for window management. I really think 24 bits is fine.
  11. FM 16 Variable Repetition limitation change?

    A cryptographic (or any other) hash is "playing the roulette", too. 10^17 is still ~56.5 bits of information, which is more than most database's primary keys ever need, more information (entropy) than most passwords represent, and surely overkill a-plenty for humble window names. You mentioned a challenge with the full content of any of the built-in hash functions being larger than this 56.5 bits, so I threw out the option of a different hash that might be easier to fit in the constraints of the situation. Granted, Mod ( Code ( Get ( WindowName ) ) ; 10^17 ) turns out not to capture anything close to that 56.5 bits in practice... Interesting! That especially makes sense if your window names tend to start with the same 4 characters, which I imagine is pretty common. The Code function puts the first characters in a string as the least significant digits in a number, and Mod is then taking those same least significant digits. Left, on the other hand, is taking the most significant digits, corresponding to the end of the string which is more likely to be different. I hadn't thought of that! Left ( ... ; 17 ) is still only working with the information from a fixed range of 4 characters, which seems like a likely liability. My guess is that window names share their last 4 characters less often than their first 4 characters, but that it still happens frequently enough that we might be uncomfortable promoting this as a habit anyone could use any time. To get a full 56 bits of information out of a full window name, the most efficient way to go may well be to use one of the built-in hash functions like comment suggested (md5 is built-in before v16 and doesn't require a second function to convert it to hex!), truncate to the first (or last, it doesn't matter) 14 hex characters, and convert the hex to decimal: NumberFromHex ( Left ( GetContainerAttribute ( Get ( WindowName ) ; "md5" ) ; 14 ) ) Alternately, if you really don't like custom functions, are happy to require FileMaker 16, and expect to have significantly fewer than 16,777,216 windows open at once, you could use the new CryptDigest function, convert it to base 64 (giving you 6 bits of information per character of the base 64 representation), and do Mod ( Code ( ... ) ; 10^17 ) on that to get 24 bits of meaningful information reflecting the full window name: Mod ( Code ( Left ( Base64Encode ( CryptDigest ( Get ( WindowName ) ; "md5" ) ) ; 4 ) ) ; 10^17 ) You could also replace the Mod ( ... ; 10^17 ) in this case with Right ( ... ; 17 ), but I do think you'd be better served with Right than Left in this case. Right ( ... ; 17 ) will retain 24 bits of information value, whereas Left ( ... ; 17 ) will lose some. The last character (in left-most position after Code) may have a 3-digit instead of a 2-digit code, but the right-most 2 of those 3 digits don't overlap with the codes of any other characters in base 64. With Left ( ... ; 17 ), you risk losing a more informative digit from the code of the first character.
  12. FM 16 Variable Repetition limitation change?

    See if Mod ( Code ( Get ( WindowName ) ) ; 10^17 ) is unique enough.
  13. FileMaker 16 introduced a collection of built-in functions for manipulating data serialized as JSON. This makes it easier for FileMaker applications to interact with many web services. This will also make JSON the de facto standard format for scripts within FileMaker to pass parameters and results to each other, improving code sharing within the FileMaker community. JSON does not have a broad palette of scalar data types to choose from: text, number, boolean, and null. Even with those, FileMaker's JSONGetElement function always returns a text result, even when the serialized JSON value is a number or boolean. So I made a handful of custom functions and scripts for sending and receiving typed data with JSON. The module is hosted on GitHub, or you can download it directly.
  14. You definitely can make encapsulated or generalized logic in FileMaker. You can't do it the same way you would in Java or Haskell, but Java and Haskell can't encapsulate or generalize logic the same way the other does, either. FileMaker's abilities seem about on par with C in this respect. Every useful programming environment has its building blocks, such as in a standard library; and the contents of that standard library dictate much of how programs written for that environment operate, both internally and with the outside world. The operating environment or standard library is often a more significant contribution to the value of a programming language than the logical structures it uses. Java is popular because of the portable virtual machine, despite being awful as a language. JavaScript is another example. C(++) is popular because its operating environment is close to the silicon, despite that making it more difficult to encapsulate certain highly consequential concerns. FileMaker scripting starting with options available in the user interface is just one reasonable choice of building blocks to start with, and a pretty inspired choice, designed to minimize the learning curve for "citizen developers". In a Darwinian sense, FileMaker's choice is successful. (The script engine also includes many steps that have no counterpart in the user interface.) Every Turing machine has its tape, and failing to embrace the nature of that tape will doom a developer to never using that machine well.
  15. That's not a meaningful distinction. "Slow compared to other programming environments" doesn't make FileMaker slow in any absolute sense. "Slow compared to user needs for a particular application" is much more meaningful, and once an application is fast enough to exceed user needs, pursuing additional speed may be wasted effort. FileMaker passes this threshold for many applications, including applications working with millions of records. Working with large data sets efficiently with FileMaker may take additional work, which makes it no different from any other programming environment, and achieving the best possible performance from FileMaker often takes less effort than achieving the best possible performance from other tools. At least FileMaker doesn't claim that processing speed is an absolute top priority, then make design decisions contradicting that every step of the way — just look at what the HPC community has to say about Hadoop.
  16. Show me a programming language that doesn't have dozens of the same kind of rant written about it, and I'll show you a programming language no one really uses for professional purposes. One note on execution speed, though, since this was in the context of rendering QR Codes in a web viewer: JavaScript rendering a large QR Code in a web viewer isn't particularly fast, either. If FileMaker scripts aren't good enough in that respect, neither is JavaScript (or at least not JavaScript in a web viewer). A plug-in or a web service (with a low-latency connection) would be a stronger comparison.
  17. I'm not sure what you mean by "true code". Can you elaborate? What makes code "true" that FileMaker scripting does not satisfy, as you understand it? As an abstract-computer-science-minded kind of person, any series of instructions that a computer will execute is "code" enough for me — maybe I'll throw in Turing completeness if I'm being really picky, but FileMaker scripts satisfy that criterion, too.
  18. Nice! Yes, Barcode Creator uses a similar approach: building base 64 that can be converted to container data. The details are more complicated for different image file types and barcode symbologies, but that's the gist of it. I'd suggest you try EPS next if FileMaker's support for EPS in containers weren't deprecated. Other folks have used existing JavaScript libraries to render a QR Code in a web viewer. There are some challenges with that, though: The JavaScript QR Code libraries are mostly designed to draw the QR Code. I haven't seen one that actually makes a separate image file, but I haven't really been looking. In case you opt to display a QR Code in a web viewer, printing web viewers looked pretty awful on Windows last time I checked (on-screen resolution instead of print-quality). This might be worth someone checking again. Images in container fields print much better. Web viewers don't work in server-side scripts, if you ever want to move barcode creation from clients to the server. All the JavaScript libraries I've seen that generate QR Codes don't do it well. I have yet to see a JavaScript QR Code library that both supports all the basic encoding modes and optimizes the size of the encoded data.
  19. TriggersDisable is designed to be called by a script. It won't disable triggers without a script running to help developers avoid shooting themselves in the foot. What you're seeing in the Data Viewer is how it's designed to work.
  20. Others have commented on the usage of the functions, so I won't repeat that. The behavior of the Triggers* functions you describe sounds like a bug, but I can't reproduce it. Can you post the script where you saw that behavior?
  21. Performance question with scripts

    How different is the export process for each layout? In other words, how much of the single-long-script version would live inside the If...Else If...Else branches vs. how much would be the same steps (configured the same way) running outside the If[] block? The more steps in common, the more sense it makes to have the one long script. The more different steps there are, the more it makes sense to have separate smaller scripts. There's also another option for how to break up the logic which might be worth considering: one script for each layout, and each of those scripts calls common sub-scripts for substantial chunks of steps that are exactly the same. I lean towards this approach when there are both large chunks of logic different between contexts, AND large chunks of logic in common.
  22. See https://san-diego-workforce-partnership-1.workable.com/jobs/431628.
  23. Map Directions (fmEasyMap)

    If you're driver is already looking at the information on a smart phone, it's better to hand-off each destination to a navigation app than to generate the directions in FileMaker. This blog post describes how (at least on an iOS device). FileMaker can still maintain the order of destinations, and the Google and MapQuest APIs both include solutions to the traveling salesman problem that you can use in FileMaker to determine what the order of destinations should be.
  24. 0x0 - illegal xml character

    A solution to the original problem! Huzzah! And I really like a more thorough enumeration of the difficulties getting FileMaker to work with the null character. However, null-terminated strings are generally avoided whenever possible these days. Despite the other awkward handling of the null character, FileMaker is probably not using null-terminated strings under the hood, either in the calculation engine or in field storage: Let ( [ _null = Base64Decode ( "AA==" ) ; _string = "a" & _null & "a" ] ; Position ( _string ; _null ; 1 ; 1 ) // = 2 )
  25. Computer Science Degree

    You do not need a computer science degree to be hired as a FileMaker developer. Most FileMaker developers don't have any computer science training, in fact, so having one is a competitive advantage for an entry-level position. Be aware, though, that knowledge and skills unique to computer science (as distinct from general computer programming and IT) are often not what makes the biggest difference between a good and a great FileMaker application. A lot of computer science doesn't go as far as a little computer science combined with a little design.
×

Important Information

By using this site, you agree to our Terms of Use.