FM 16 Variable Repetition limitation change?

[Ron Cates]

I have been using a navigation technique I developed back in FM12 and have used successfully in 12, 14 and 15 that utilizes global variables to track forward and back history. In order to make it multi window friendly I use Code ( Get ( WindowName ) ) as the variable's repetition number.

When I updated to FM16 the navigation script failed. While investigating the failure I found that Code ( Get ( WindowName ) ) as a variable repetition in FM 16 resolves different than it does in FM 15.

I created a test file and a script that set two variables as shown below. The name of the window is codeTest

Here are the results in the Data Viewer of both FM15 and FM16

As we can see the Variable repetition result is different in FM16 for code ( Get ( WindowName ) ) and different than the result in the variable value...

I am on Windows 10

I've tested on both local and hosted files with the same result.

( Unfortunately this was not the actual failure I was looking for. In the script in question the global variables do not get created at all in FM16 which appears to be a different issue. Still investigating.. )

 

Can anyone confirm this? Or has there been any other strange behavior when setting variables reported? Also, now what?

Thanks

Ron

 

codeTest.zip

 

 

Edited June 2, 2017 by Ron Cates

[Ron Cates]

As a follow up. I set a variable $window to Code ( Get ( WindowName ) )  and then entered $window as the variable repetition.

The value of $window shows as 11600115001010008400101001000011100099

but putting $window in the variable repetition shows as 101001000011100099

And as I'm entering this post I just realized that the perameter shows as the last 18 digits of what the value in $window

 

Well, that seems to be it. The Variable parameter seems to be limited to 18 digits and it uses the last 18.

I entered 11600120001010011600032001140010100103001100011100108000320011600115001010008400101001000011100099 as the variable repetition and the result was 101001000011100099 as the repetition

 

Is this expected behavior?

 

[Ron Cates]

So here is the issue simplified which had nothing to do with code ( Get ( windowName ) ) after all.

Variable repetitions in FM16 appear to be limited to 18 digits. If more than 18 digits are entered it will use the last 18 digits of the entered string whereas FM 15 and prior did not have this limitation.

 

This throws a big monkey wrench into my works as I use the technique of "Bucket Variables" as I now recall it being called in several techniques that I've developed and use on a regular basis. Not only do I use code ( Get ( WindowName ) ) but I also use Code ( record::ID ) in many places to set a variable to an individual record.

Ugh! Now to start checking through my solutions to find where this is going to bite me.

I would certainly welcome any suggestions for a work around for unique repetitions since I'm loving the new features of FM 16 but have to adjust all my files to be able to use it...

Edited June 2, 2017 by Ron Cates

[bcooney]

omg! Calling WIM!!!

I see the same issue on Mac OS Sierra ( 10.12.5) FMA 16.0.1.162

[comment]

Hmm. There is a difference in the tech specs regarding the Limit of Repetitions for Variables in an Iterative loop parameter: 

• version 15: 10^400 for the variables if they are defined as numbers
• version 16: 10^17 for the variables if they are defined as numbers

I've never understood what exactly they mean by "in an iterative loop" or "if they are defined as numbers" (is there another limit for variables defined as text?), but it seems to agree with your findings.

 

1 hour ago, Ron Cates said:

I would certainly welcome any suggestions for a work around

I don't know. My first thought was to use the new cryptographic functions to produce a digest of the value, but I believe the smallest digest would be 16^32 (MD5), which is much larger than 10^17. I suspect the only choice is to create some sort of a lookup array.

 

 

 

Edited June 2, 2017 by comment

[jbante]

See if Mod ( Code ( Get ( WindowName ) ) ; 10^17 ) is unique enough.

[bcooney]

On TechNet:

https://community.filemaker.com/thread/173734

Mod ( Code ( Get ( WindowName ) ) ; 10^17 ) didn't seem to work for me.

Left ( Code ( Get ( WindowName ) ) ; 17 ) seems to work ok.

[comment]

11 minutes ago, jbante said:

See if Mod ( Code ( Get ( WindowName ) ) ; 10^17 ) is unique enough.

How exactly does one go about "seeing" such thing? Suppose you're using a UUID (16^32)  as the input; truncate it to 10^17 and you're basically playing the roulette.

 

[Ron Cates]

The post on TechNet says 17 digits but I count 18. Am I miscounting or are they. Lol!

Regardless, it was pointed out on that post that only the set variable script step shows the limitation, meaning that a variable set with a Let() function can set the variable with the repetition as desired.

Let (

  $$LONG_REPETITION[9999999999999999999999999999999999999999999999999] = 10;

  1

)

I think that make the most sense for my fix. That way I only have to adjust everywhere the variables are being set and where they are being referenced can remain the same.

I haven't tested it myself yet, but will tomorrow..

[comment]

3 minutes ago, Ron Cates said:

only the set variable script step shows the limitation, meaning that a variable set with a Let() function can set the variable with the repetition as desired.

Beware. It's very difficult to verify such claim - see the discussion starting here: 
http://fmforums.com/topic/52580-cant-getnthrecord-using-a-variable/?do=findComment&comment=247009

 

[jbante]

7 hours ago, comment said:

How exactly does one go about "seeing" such thing? Suppose you're using a UUID (16^32)  as the input; truncate it to 10^17 and you're basically playing the roulette.

A cryptographic (or any other) hash is "playing the roulette", too. 10^17 is still ~56.5 bits of information, which is more than most database's primary keys ever need, more information (entropy) than most passwords represent, and surely overkill a-plenty for humble window names. You mentioned a challenge with the full content of any of the built-in hash functions being larger than this 56.5 bits, so I threw out the option of a different hash that might be easier to fit in the constraints of the situation. Granted, Mod ( Code ( Get ( WindowName ) ) ; 10^17 ) turns out not to capture anything close to that 56.5 bits in practice...

 

7 hours ago, bcooney said:

Mod ( Code ( Get ( WindowName ) ) ; 10^17 ) didn't seem to work for me.

Left ( Code ( Get ( WindowName ) ) ; 17 ) seems to work ok.

Interesting! That especially makes sense if your window names tend to start with the same 4 characters, which I imagine is pretty common. The Code function puts the first characters in a string as the least significant digits in a number, and Mod is then taking those same least significant digits. Left, on the other hand, is taking the most significant digits, corresponding to the end of the string which is more likely to be different. I hadn't thought of that!

Left ( ... ; 17 ) is still only working with the information from a fixed range of 4 characters, which seems like a likely liability. My guess is that window names share their last 4 characters less often than their first 4 characters, but that it still happens frequently enough that we might be uncomfortable promoting this as a habit anyone could use any time.

To get a full 56 bits of information out of a full window name, the most efficient way to go may well be to use one of the built-in hash functions like comment suggested (md5 is built-in before v16 and doesn't require a second function to convert it to hex!), truncate to the first (or last, it doesn't matter) 14 hex characters, and convert the hex to decimal:

NumberFromHex ( Left ( GetContainerAttribute ( Get ( WindowName ) ; "md5" ) ; 14 ) )

Alternately, if you really don't like custom functions, are happy to require FileMaker 16, and expect to have significantly fewer than 16,777,216 windows open at once, you could use the new CryptDigest function, convert it to base 64 (giving you 6 bits of information per character of the base 64 representation), and do Mod ( Code ( ... ) ; 10^17 ) on that to get 24 bits of meaningful information reflecting the full window name:

Mod ( Code ( Left ( Base64Encode ( CryptDigest ( Get ( WindowName ) ; "md5" ) ) ; 4 ) ) ; 10^17 )

You could also replace the Mod ( ... ; 10^17 ) in this case with Right ( ... ; 17 ), but I do think you'd be better served with Right than Left in this case. Right ( ... ; 17 ) will retain 24 bits of information value, whereas Left ( ... ; 17 ) will lose some. The last character (in left-most position after Code) may have a 3-digit instead of a 2-digit code, but the right-most 2 of those 3 digits don't overlap with the codes of any other characters in base 64. With Left ( ... ; 17 ), you risk losing a more informative digit from the code of the first character.

Edited June 3, 2017 by jbante

[comment]

1 hour ago, jbante said:

A cryptographic (or any other) hash is "playing the roulette", too.

Very true - but the odds are much, much, much better. Astronomically better. So much better that they cross the threshold of acceptability - the same way we accept UUID as being unique, when in fact it's just almost certainly unique. When you reduce the odds by a factor of 3*10^22 (!!!), I am not sure I want to play anymore.

[jbante]

4 hours ago, comment said:

Very true - but the odds are much, much, much better. Astronomically better. So much better that they cross the threshold of acceptability - the same way we accept UUID as being unique, when in fact it's just almost certainly unique. When you reduce the odds by a factor of 3*10^22 (!!!), I am not sure I want to play anymore.

So what's the threshold of acceptability for this application? "Acceptability" is relative to the problem being solved, not relative to the other possible solutions (some of which, as you pointed out, aren't even possible solutions anymore). What's acceptable is what fits in a variable repetition number, which apparently is 10^17. I think even less than that could be acceptable. We're talking about what amounts to a hash table of simultaneous open windows during a single session here, not a primary key that needs to be unique across the entire universe for a few decades. 10^17 buckets is a gratuitously large hash table for any application, and even just 16M is plenty of headroom for window management. I really think 24 bits is fine.

Edited June 3, 2017 by jbante

[comment]

17 minutes ago, jbante said:

So what's the threshold of acceptability for this application?

I don't know. And I would not dwell on how many windows are open during a single session, but rather on the odds of a collision during the entire lifetime of the solution (which could well be much greater than one would estimate intuitively) against the possible damage due to such collision. 

Also, if there is a solution that does not involve any risk of collision, I would prefer it. I do not implement UUIDs when serial numbers will suffice. 

 

Edited June 3, 2017 by comment

[normanicus]

This may not be appropriate by why not use Get ( CurrentTimestamp ) as your repetition number? This, unless you are creating more than a window/second, is guaranteed to be unique, to be compatible with whatever version of Filemaker and has the benefit of being in historical order.

As I said, maybe I'm missing the point.

[comment]

5 minutes ago, normanicus said:

why not use Get ( CurrentTimestamp ) as your repetition number?

I believe the idea here is that each window can retrieve its own repetiton's value by using the Get ( WindowName ) function. 

[jbante]

36 minutes ago, comment said:

I don't know. And I would not dwell on how many windows are open during a single session, but rather on the odds of a collision during the entire lifetime of the solution (which could well be much greater than one would estimate intuitively) against the possible damage due to such collision. 

But the number of simultaneous windows during a single session is exactly what's relevant to the quality of a hash based on window names. A collision across different sessions is inconsequential here.

With 24 bits, you need to be working with 184 window names simultaneously for the probability of a collision to reach 0.001. 20 is a gratuitously large number of windows for a single file to have open; the probability of a collision is about 1 in 100,000 in that case. I'm comfortable with that. Given that window names tend to be systematically calculated, the practical probability may be substantially lower after developer testing. Since developers have the option to check if a hash bucket is taken and try a slightly different window name instead, I don't think it's a big deal.

If you really want to go overboard and use the full ~56.5 bits we get in 10^17, the probably of a collision among 20 windows is 1.9e-15. You can have 447,214 windows before the probability of a collision gets to one in a million. 20 users with 20 windows (with new names) each every day will take more than 3 years to find a collision.

Hash tables are a solved problem, and the solution is not to have 2^128 buckets. The solution is to work with however many buckets you have.

Edited June 3, 2017 by jbante

[comment]

3 minutes ago, jbante said:

But the number of simultaneous windows during a single session is exactly what's relevant to the quality of a hash based on window names.

It's one of the relevant factors, not the only one. The number of sessions during the solution's lifetime is also relevant here, despite the fact that a collision across different sessions is harmless. Your odds of winning a lottery do not depend only on the chances of a single ticket in a single drawing. They increase with the number of drawings for which you bought a ticket.

 

9 minutes ago, jbante said:

Hash tables are a solved problem

Indeed they are. But hacked hash tables, where an established algorithm is buzz-sawed in half, are not. I believe people have worked very hard to develop and test (and sometimes to break) these algorithms. You may be right about your assessment of the risks; I am just not willing to take your word (and little else) for it.

---
P.S. I am using "half" figuratively. 10^17 / 16^32 is not half, or even close to it. It's almost zero.

 

[jbante]

1 hour ago, comment said:

Indeed they are. But hacked hash tables, where an established algorithm is buzz-sawed in half, are not. I believe people have worked very hard to develop and test (and sometimes to break) these algorithms. You may be right about your assessment of the risks; I am just not willing to take your word (and little else) for it.

---
P.S. I am using "half" figuratively. 10^17 / 16^32 is not half, or even close to it. It's almost zero.

The work behind the cryptographic hash algorithms (plus MD5) built-in to FileMaker didn't really have hash tables in mind, but their properties are still useful. One of the information-theoretic design goals of cryptographic hashes (or at least a necessary consequence of those goals) is that any subset of the bits should be an equally good hash as any other equal-sized subset. This is fundamental to what makes a hash useful for any cryptographic or error detection purpose. If working with only a "buzz-sawed" fraction of the full cryptographic hash doesn't preserve the uniform probabilistic behavior, it was a bad hash to start with. It's because of the work making the hashes good that I'm comfortable with the risks.

10^17 / 16^32 is a pretty small number. That doesn't make 10^17 (or 2^24) a small number.

Edited June 3, 2017 by jbante

[Ron Cates]

I created a little custom function some time ago to allow me to set variables on the fly with dynamic variable names and repetitions. So to address my problem I just need to have the set variable script steps that are affected use this function to set the variables using the same repetitions as before.

It simply wraps a Let function with Evaluate.

The custom function is simply

 Evaluate ("Let ( $$" & ~var & "[" & ~rep & "]=" & Quote ( ~value ) & ";\"\" )")

Parameters are ~var (  name of the variable ), ~rep ( repetition ) and ~value

~setVarByName ( ~var ; ~rep ; ~value )

So I can set my set variable script steps to something like this

~setVarByName ( "Back" ; Code ( Get ( WindowName )  ; GetLayoutID )

I've tested this and it works as expected.

[Wim Decorte]

I'm not so worried about things like window names.  The audit log approach I showed at Devcon relies on Code() calls on field names for instance and those will fail now.  Many data collection routines where I use strings like Project Manager names, project names, as buckets for reporting will now fail too.  Unless I take Ron's approach to use Evaluate() calls on Let() to set my variables.

Played around with HexEncoode on Md5 Digests but even then only a portion of the string gets converted to a number and while it reduces to likelihood of generating the same number for similar strings it's still too close to be comfortable.

Obviously with 16 my approach would be to use JSON or Arrays and not repeating vars but this not bode well for the upgrade process of many solutions.

[Josh Ormond]

Is there any sense of a reason for the change? It seems to be a very significant change of capability.

[bcooney]

Great question, Josh. 

[jbante]

On 6/3/2017 at 4:29 PM, Wim Decorte said:

Obviously with 16 my approach would be to use JSON or Arrays and not repeating vars but this not bode well for the upgrade process of many solutions.

For small collections of data, yes, JSON is an obvious convenient solution. For large arrays, repeating variables have a large performance advantage. I haven't seen many FileMaker apps doing anything where the performance of repeating variables would override the convenience of JSON (but those are certainly the most fun!). No solution can have enough windows to track information on for the performance difference between JSON and repeating variables to be a significant bottleneck, because the fact that there are so many windows would always be the bigger problem.

[comment]

@jbante

You seem to be locked to the idea that this technique is used for tracking multiple windows and nothing else. Wim gave an example of tracking field names, and OP mentioned tracking primary keys. I myself have posted a demo that remembers a selected child record for each parent, using the same method. Now, as long as the parent primary key is a plain serial number, you can still have up to 10^17 records in the parent table with no problems. But if your serial numbers have a 3-letter prefix, and you have used Code() to map them to a repetition number, your solution will now fail - without warning - when you exceed 10 (ten!) records in the parent table. 

 

[Wim Decorte]

8 hours ago, jbante said:

For small collections of data, yes, JSON is an obvious convenient solution. For large arrays, repeating variables have a large performance advantage. I haven't seen many FileMaker apps doing anything where the performance of repeating variables would override the convenience of JSON (but those are certainly the most fun!). No solution can have enough windows to track information on for the performance difference between JSON and repeating variables to be a significant bottleneck, because the fact that there are so many windows would always be the bigger problem.

I doubt that I have ever used this technique on more than a few thousand records in the found set.  That's usually the starting point; a found set of data and wanting to cross-tab report on it.

There is one huge advantage to JSON and Arrays as we have them now: portability.  The biggest downside with variable reps is that there is no good way to pass them around between scripts; and I hate to use global variables to have to get around that.  Now we can pass around just one object and it can be complex.  We could that before too; it's just a whole lot easier now.

[jbante]

21 hours ago, comment said:

@jbante

You seem to be locked to the idea that this technique is used for tracking multiple windows and nothing else. Wim gave an example of tracking field names, and OP mentioned tracking primary keys. I myself have posted a demo that remembers a selected child record for each parent, using the same method. Now, as long as the parent primary key is a plain serial number, you can still have up to 10^17 records in the parent table with no problems. But if your serial numbers have a 3-letter prefix, and you have used Code() to map them to a repetition number, your solution will now fail - without warning - when you exceed 10 (ten!) records in the parent table.

The conversation did start with windows, but you're right. I did get fixated on one example. Field names are another example where the capacities of the 16- or 24-bit approaches are still perfectly acceptable. Several high profile "big data" database-like tools are perfectly comfortable using mere 64-bit IDs for globally unique values, and they only went with 64 because it fit conveniently with an Int64. I'm perfectly comfortable following their lead with a 56-bit hash method because it fits conveniently in 10^17. But heaven forbid we should settle for such a small ID space that it can only uniquely identify every square meter on the combined surfaces of the first 5 planets in the solar system. My next CRM is really going to need to keep 6 planets in repeating variables at any given time. I'll have to upgrade my computer to 19.6 petabytes of RAM just to keep track of the memory addresses for all the repetitions.

You're right that a serial number with a 3 letter prefix is guaranteed to cause a hash collision after 10 records with Mod ( Code ( Table::id ) ; 1e17 ), which is why everyone has been recommending against that and discussing alternatives for several days. Even when you do get a hash collision, so what? We can just do the same thing as any of several perfectly workable solutions in widespread use by other hash tables. Hash table collisions are a solved problem. But heaven forbid we should have to be confronted with yet another problem the internet is happy to give us a feast of solutions to, apparently.

Edited June 6, 2017 by jbante

[comment]

Your sarcasm is wasted on me. And (pardon the pun) at this point you're just rehashing what you said before.

There are two types of solutions here: solutions with risk of collision, and solutions with no such risk. As long as a no-risk solution is available, I see no reason to even consider a risky alternative. As I said before, I will not use UUIDs where serial numbers can be used. (I could add that I certainly won't use chopped-off UUIDs, when I have nothing but your personal assurance that "it's all right" - but then I would be repeating what I already said too.)

Of the three examples discussed so far, only the one tracking windows is truly problematic. For tracking records, you can simply use Get (RecordID) directly as the repetition number. For tracking field names, you can lookup their position in FieldNames(). You cannot do the same with WindowNames(), because its result can change during the session - but since the creation of the variable is scripted, you can maintain your own WindowNames array where each tracked window will have a permanent entry.

 

 

Edited June 6, 2017 by comment

[Josh Ormond]

Interesting post about this on the FM Community site. This was apparently not an intentional behavior change. https://community.filemaker.com/message/671487#671487

Quote

All:

Thank you for your comments.

This issue was previously reported to Testing and Development via Fred(CH) forwarding me this information privately. They were able to reproduce the issue and it was not an intentional change in behavior. No further information is currently available, but I will post back here when I can! 

TSPigeon

FileMaker, Inc.

 

[jbante]

9 hours ago, comment said:

There are two types of solutions here: solutions with risk of collision, and solutions with no such risk.

There are two types of solutions here: solutions with no risk of collision, and solutions where the risk of a hash collision can be resolved so that it has no negative consequences.

Edited June 6, 2017 by jbante

[bcooney]

Apparently, the issue is fixed in FileMaker 16.0.2!

[comment]

35 minutes ago, bcooney said:

the issue is fixed in FileMaker 16.0.2

And they also updated the specs!