jbante

Many people use "developer" and "programmer" interchangeably. Many other people distinguish between "developer" and "programmer" different ways. One way to distinguish between them that is somewhat unique to the FileMaker community is that a "programmer" just does programming logic (data schema, scripts, calculations), whereas a "developer" is also skilled at user interface design. The US Bureau of Labor Statistics (BLS) does make a distinction in it's Standard Occupational Classification (SOC): In short, according to the BLS, a "computer programmer" implements a specification written by someone else, whereas a "software developer" is also involved in the discovery and design that would lead to such a specification. Most of the FileMaker community falls solidly in the "developer" category, according to this scheme. (Note that I'm using the 2018 revision to the SOC scheme, which does not make a distinction between "applications" and "systems" developers like the 2010 SOC did. FileMaker developers would definitely be "applications" developers in the older scheme.) Please don't take the BLS definition as an authoritative tie-breaker. Like modern dictionaries, the SOC system is more descriptivist than prescriptivist: it tells you what the BLS tends to observe in the wild; it's not a mandate for what the "correct" difference is.

FileMaker has used fixed-point arithmetic for a long time.

For folks with a background emphasizing data analysis, there are examples of folks pulling FileMaker data into R and Python by hosting a FileMaker database through an ODBC connection. There isn't much training to be had on this except the documentation for FileMaker, R, and Python on how to use ODBC connections. For folks with a background emphasizing software development, there are examples of folks using a plugin to crunch numbers using Java (with the 360Works plugin) or JavaScript (with the BaseElements plugin) libraries. For the price of more labor setting things up, there are also examples of setting up analytical functions in a web micro-service running libraries in some other programming language, and accessing that from FileMaker using the Insert from URL script step. My background emphasizes math, and, for a combination of reasons, I like implementing analytical methods natively in FileMaker without plugins or external web services. Personal health issues have limited how much I could invest in that, but I do try to come up with something when specific requests come up on the forums and I'm having a good day. Unfortunately, very few FileMaker developers are comfortable with the numerical methods involved in any non-trivial technique, so I don't have much help in that regard. (To be fair, most of the R and Python communities aren't comfortable with those methods, either; but they still found a way to support the folks who are.) The training on this amounts to knowing FileMaker scripting and calculations well, and the literature on numerical methods. Folks have been implementing data visualizations more sophisticated than the built-in charts for many years with web viewers. We haven't made as much progress on this as I'd like. FileMaker as a tool in general is typically favored for applications where the app owners are more interested in minimizing the work it takes to build their minimum viable product than investing the effort it takes to build FileMaker-native analytical functions worth sharing. The desire comes up from time to time, but there isn't much in the way of publicly shared case studies to show the community that good things are possible and inspire folks to want more. So, please tell us more about what kind of archaeology analysis you want to do in FileMaker. Start another thread with something specific!

I realized the potential confusion with that this morning and re-named the parameters to "radius1" and "radius2". You should be passing ( width / 2 ) and ( length / 2 ) as the parameters (the order doesn't matter): EllipseCircumference ( width / 2 ; length / 2 )

So you're asking about how to copy my calculation into your file as a custom function? FileMaker's help documents how to make custom functions in general. Copy the calculation code for the EllipseCircumference custom function from GitHub. Open your file in FileMaker Pro Advanced. Open File > Manage > Custom Functions... In the "Manage Custom Functions" dialog, click the "New..." button in the bottom-left corner. In the "Edit Custom Function" dialog, paste the function code into the function calculation box. Set the "Function Name" and "Function Parameters" to match the header comment in the function calculation. Click "OK" in the "Edit Custom Function" dialog, and click "OK" in the "Manage Custom Functions" dialog to save the function. You can then use the EllipseCircumference custom function in any calculation (field, script, data viewer) in that file. The approximate calculations mentioned earlier in this thread are all wrong, by about a factor of two. The first reference link in the thread gave an incorrect formula. The correct form of the approximation is: 2 * Pi * Sqrt ( ( radius1 ^ 2 + radius2 ^ 2 ) / 2 ) (Or diameter/2 in place of the radiuses.) This approximation is exact when the axes are equal (i.e. for a circle). The relative error of this grows up to about 11% as the ellipse gets extremely stretched out. This is a much better approximation (now also available in custom function form), if you really want a simple formula instead of the more exact custom function: 4 * ( radius1 + radius2 ) * ( Pi / 4 ) ^ ( 4 * radius1 * radius2 / ( radius1 + radius2 ) ^ 2 ) This approximation is always within about 0.1% or better. Still, I'd only use this if the exact calculation is too slow for a particular application.

I believe the moderators of this forum prefer that transactional conversations happen in private messages. Aside from that, I'm not sure exactly what you mean by "teach me the function". Do you want to understand how the math works? I think that's much more than a 1-2 hour conversation, and there's plenty of other information linked to from the Wikipedia page for ellipses that can steer you in the right direction. Do you want a script version of the same thing? That could be done pretty easily, but this is a use where I think a custom function is a less clumsy solution to use. (I'll grant you that my implementation might look more complicated than absolutely necessary, but that's my pattern when I don't want any helper functions to depend on or extra parameters that are only there to pass information between recursions.)

Try this custom function for a reasonably exact calculation.

It looks like the calculation you found is approximating the circumference of an ellipse as the circumference of a circle with radius half-way between the circles that inscribe and circumscribe the ellipse. You could calculate upper and lower bounds as the circumferences of those 2 circles. The Wikipedia page includes some more bounds. Combining them in FileMaker might look like: lowerBound = Max ( Pi * ( width + height ) ; 4 * Sqrt ( width^2 + height^2 ) ) upperBound = Min ( 2 * Pi * Max ( width ; height ) ; 4 * ( width + height ) ; Pi * Sqrt ( 2 * ( width^2 + height^2 ) ) ) If those numbers are close enough for your comfort, then there you go. I imagine they might still be too loose with 500,000 pieces to cut. I thought the exact calculation looks a little fun to me, so I might take a crack at that later.

Calculating the circumference of an ellipse is complicated. Any simple formula you find is likely to be an approximation, so it's natural to expect that they won't all get the same answer. How close is close enough for your app?

What you're describing is the knapsack problem. The Wikipedia pages includes descriptions of several approaches to solving it for you to consider.

I made a solution once that works in the other direction: given a point, find which polygons in the database contain it. There are a couple ways to do this, but the way that executed fastest was to decompose each polygon into geohashes covering the same region in advance, then calculate the geohash for the point of interest and perform an exact match find on the geohash. (When it's important to keep the database small, you can decompose regions into different-precision geohashes depending on what's necessary to cover the shape of the region, then finding a region for a point consists of attempting finds at multiple resolutions until it gets a match.) This assumed that the regions of interest are already in the database. But you're asking for a solution to a different problem: given a polygon, find which points in the database are contained by it. Start by looking at the "point in polygon" problem. I specifically recommend starting with the winding number algorithm. That's fine if you already have a small number of records to check. If you have more than a handful of points in your database, it will probably be unacceptably slow to check every single one. You could solve that by doing a find for points within the bounding box of the polygon, then checking each record in that constrained set with the winding number algorithm. This blog post I wrote a few years ago discusses how to find within a bounding box. If it's worth it to you to experiment with more complication options to achieve better speed, look into the point location problem, and trapezoidal decomposition in particular. The idea is that you might decompose your polygons into strictly vertical slabs, perform finds for the bounding boxes of the slabs, and check if the found points are actually in those slabs (which is much faster than the general point in polygon solutions because we can take advantage of the known shape of the slabs). This gets gnarly very fast, especially when you start handling non-convex polygons. You also don't want to be performing a huge number of numerical range finds in a single find request. FileMaker starts to choke on the complexity of the request. Extending a found set for each slab would probably work better.

While you're editing the custom function, you need to set what the parameters are before the calculation will recognize the "bytes" and "precision" tokens, which would be undefined otherwise.

Try this custom function.

Then use stored number fields instead of calculation fields, and set each quartile field in a script that does separate sorts before setting each quartile field. A script also potentially makes it easier to handle ties, to address LaRetta's point.

Thank you for the update!

Is there an estimate of when a production version of a new build might be available? Weeks? Months?

If you sort the records by contribution, you can set the quartile for each record with the calculation: Ceiling ( Get ( RecordNumber ) / Get ( FoundCount ) * 4 )

The 5.06 build goes back to working normally in my tests on Mac & Windows clients. I'm reluctant to put this on the server without knowing how production-ready this build is.

Now running ScriptMaster v5.052, I get the exact same error in all three contexts (Mac client, Windows client, Windows server).

Is there a ScriptMaster version "5.052"? I just downloaded it fresh from 360Works yesterday, and it's only 5.05 (no 2).

This isn't working in FileMaker 16 anymore. Does anyone have any good ideas why? The full function decompresses the contents of a container field and puts the result in a file: // SM_DecompressGZIPContainerToFile ( containerFieldName ; resultFilePath ) import java.util.zip.GZIPInputStream; import java.io.FileOutputStream; import java.util.io.*; InputStream fieldStream = new BufferedInputStream(fmpro.getContainerStream(containerFieldName)); GZIPInputStream gunzip = new GZIPInputStream(fieldStream); FileOutputStream fileOutputStream = new FileOutputStream(resultFilePath); byte[] buffer = new byte[1024]; int bytes_read; while ((bytes_read = gunzip.read(buffer)) > 0) fileOutputStream.write(buffer, 0, bytes_read); gunzip.close(); fileOutputStream.close(); return 1; This worked great until FileMaker 16 and ScriptMaster 5.05. Now it doesn't. I've tried it from the client (16.0.2) on Mac (10.12.6) and Windows (Server 2012 R2), and in server schedules and Perform Script On Server (16.0.2.212). Mac and Windows clients both report this stack trace: java.lang.IllegalArgumentException: You tried to create a QuadChar with a string of length: 0 at com.prosc.fmkit.QuadChar.<init>(QuadChar.java:52) at com.prosc.fmkit.types.FMBinary.getStreamTypes(FMBinary.java:389) at com.prosc.fmkit.types.FMBinary.getBestQuadChar(FMBinary.java:404) at com.prosc.fmkit.types.FMBinary.getBestInputStream(FMBinary.java:443) at com.prosc.beanshell.FMPro.getContainerStream(FMPro.java:58) at com.prosc.beanshell.FMPro$getContainerStream.call(Unknown Source) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:120) at Script1.run(Script1.groovy:1) at com.prosc.beanshell.GroovyFunction._invoke(GroovyFunction.java:158) at com.prosc.beanshell.GroovyFunction.invoke(GroovyFunction.java:136) at com.prosc.fmkit.Plugin.invokeFunction(Plugin.java:398) at com.prosc.fmkit.RegisterablePlugin.invokeFunction(RegisterablePlugin.java:178) at com.prosc.fmkit.Plugin.invokeFunctionNoErrors(Plugin.java:374) at com.prosc.fmkit.PluginBridge2$1.run(PluginBridge2.java:1059) at com.prosc.fmkit.PluginBridge2.doFunction(PluginBridge2.java:1072) at com.prosc.fmkit.PipeChild.lambda$handleMessage$9(PipeChild.java:297) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Server schedules and PSOS both report this, instead: com.prosc.fmkit.FmCalculationException: 106 at com.prosc.fmkit.PluginContext.evaluateExpression(PluginContext.java:192) at com.prosc.beanshell.FMPro.getContainerStream(FMPro.java:57) at com.prosc.beanshell.FMPro$getContainerStream.call(Unknown Source) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:120) at Script1.run(Script1.groovy:1) at com.prosc.beanshell.GroovyFunction._invoke(GroovyFunction.java:158) at com.prosc.beanshell.GroovyFunction.invoke(GroovyFunction.java:136) at com.prosc.fmkit.Plugin.invokeFunction(Plugin.java:398) at com.prosc.fmkit.RegisterablePlugin.invokeFunction(RegisterablePlugin.java:178) at com.prosc.fmkit.Plugin.invokeFunctionNoErrors(Plugin.java:374) at com.prosc.fmkit.PluginBridge2$1.run(PluginBridge2.java:1059) at com.prosc.fmkit.PluginBridge2.doFunction(PluginBridge2.java:1072) at com.prosc.fmkit.PipeChild.lambda$handleMessage$9(PipeChild.java:297) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) I get the same results when I reduce the function to just this: InputStream fieldStream = fmpro.getContainerStream(containerFieldName); return 1; Any thoughts?

XSLT seems to me like it would probably be the fastest. Exporting the CSV to disc and importing can also be advantageous if your schema is already a convenient match for that or you're willing to adapt your schema for that. The built-in JSON functions have many virtues. Execution speed is not one of them. In every test I've run, GetValue on return-delimited lists leaves the built-in JSON functions in the dust. The MBS plug-in handles JSON differently in a way that makes it much faster (MBS does what I was hoping FileMaker would do with an in-memory data structure), if you're into plug-ins. If you like sticking with return-delimited lists, you can make the processing of a large list go faster by pruning the list as you go: Set Variable [ $value ; GetValue ( $list ; 1 ) ] Set Variable [ $list ; Replace ( $list ; 1 ; Length ( $value ) + 1 ; "" )] This makes the parsing of the whole list work in roughly O ( N^1.5 ) time, rather than O ( N^2 ).

The new JSON functions have many admirable advantages for performing certain operations. Speed is not one of them. (Unless the serialized data format is JSON, there's nothing you can do to change that, and the only alternative parsing method is other built-in FileMaker functions.)

I have now. The 16.0.2 update does not change the performance characteristics of the JSON functions relative to return-delimited lists or repeating variables.

The JSON functions in FileMaker 16 have constraints the SortValues and UniqueValues do not. The JSON parsing functions check that the input they're trying to parse is in fact completely valid JSON. FileMaker has to do this to be able to return an error result when appropriate. You can't know if the outer-most structure in a JSON string is correctly terminated without starting at the beginning, checking every inner structure, and getting all the way to the end. Due to how the JSON format works, this requires stepping through the entire string for every single operation. (This could be resolved by FileMaker using a more efficient internal data structure for JSON in place of the raw string, but performance tests from a few folks suggest that FileMaker isn't doing this.) In comparison, the GetValue function only has to parse as many newline characters as it takes to get to the value you want, and it can ignore everything in a string after that point. With repeating variables, there doesn't need to be any parsing at all (depending on how you do it) — you can just use the whole content of the variable (repetition). The JSON writing functions have a similar constraint. They validate any JSON you had to start with, and then the JSON writing functions are somewhat fastidious in organizing the contents of their JSON outputs. You can make parsing nested structures in JSON less slow by parsing out inner structures, then getting the details from the inner structures, rather than pulling from the outer structure for every inner detail. This reduces how much total work FileMaker spends validating the JSON for each read. When speed is more important than simple code, do this: Let ( [ _dictionary0 = JSONGetElement ( $json ; 0 ) ; _0a = JSONGetElement ( _dictionary0 ; "a" ) ; _0b = JSONGetElement ( _dictionary0 ; "b" ) ; _dictionary1 = JSONGetElement ( $json ; 1 ) ; _1a = JSONGetElement ( _dictionary1 ; "a" ) ; _1b = JSONGetElement ( _dictionary1 ; "b" ) ] ; ... ) Not this: Let ( [ _0a = JSONGetElement ( $json ; "[0].a" ) ; _0b = JSONGetElement ( $json ; "[0].b" ) ; _1a = JSONGetElement ( $json ; "[1].a" ) ; _1b = JSONGetElement ( $json ; "[1].b" ) ] ; ... ) And for writing JSON, add as much in each JSONSetElement call as you can, rather than using many separate calls. Do this: JSONSetElement ( "[]" ; [ "[0].a" ; 1 ; JSONNumber ] ; [ "[0].b" ; 2 ; JSONNumber ] ; [ "[1].a" ; 3 ; JSONNumber ] ; [ "[1].b" ; 4 ; JSONNumber ] ) Not this: Let ( [ _json = JSONSetElement ( "[]" ; "[0].a" ; 1 ; JSONNumber ) ; _json = JSONSetElement ( _json ; "[0].b" ; 2 ; JSONNumber ) ; _json = JSONSetElement ( _json ; "[1].a" ; 3 ; JSONNumber ) ; _json = JSONSetElement ( _json ; "[1].b" ; 4 ; JSONNumber ) ] ; ... ) While these patterns make FileMaker's JSON functions less slow, it isn't enough to make them competitive with return-delimited lists or repeating variables for speed.