Speed testing

[LaRetta]

When running speed tests in a file, will Refresh Window [ flush cached join results ] completely reset a file?  Is cache the only thing which might affect speed test?

 

Because of my anal nature, I've always closed FileMaker and I used to also restart my computer between each test.  Is there a way I can tell what resources each of these processes clear on Mac Mavericks, i.e. does closing FileMaker free more system resources which might affect the test than just flush cache?  Does restarting the computer influence?

 

I know that, during speed tests, all other processes should be shut down, such as backups, searching internet etc. or the effects can influence speed tests but how can I be absolutely sure all tests are equal to start?

[jbante]

There are two ways to see what you want through the hazing of confounding variables: control them, and average them out. The latter induces much less anxiety. You can only control what you know about (and spend a lot of time in the process), but you can average out even factors you don't know about. (This is not to say that you can average out all factors you don't know about, but almost always some of them.) In an A/B comparison, I might run each 5 times in random order. If one test run had a substantial benefit the others didn't, it will be an obvious outlier. I'll typically test smaller operations in much larger loops just so differences between solutions add up to something measurable. In this case, the central limit theorem takes care of any variation due to whatever else the computer is working on at the time.

 

One other thing I usually make a point of doing that more in the spirit of controlling than averaging is a "Null" test (not a null hypothesis) where I measure only how long it takes to run the test overhead, without the operation actually being tested. The overhead of running the test can otherwise make differences seem smaller than they actually are.

[LaRetta]

Hey Jeremy!  Thank you! 

 

Then I'll just begin my speed test with Refresh Window [ Flush cached...] and run many tests and take the average.  Ah, and the Null test makes sense although I'm not sure it would always be possible.  For example, I was comparing two scripts:

Refresh Window [ Flush cached join results ]
Set Error Capture [ On ]
Constrain Found Set [ OMIT ; uniqueID ; * ]

to this script:

Refresh Window [ Flush cached join results ]
Go To Record/Request/Page [ first ]
Omit Multiple [ Get ( FoundCount ) 

I could immediately tell that the Constrain would take longer (and much longer if it wasn't already indexed) because I wanted watched the indexing progress bar.   On a million records, the Omit Multiple took 2 seconds but the Constrain took 13 seconds.  :-)

If you could think of an example of using a null test, I would appreciate it.  I have a pretty good idea but just can't think of an example.  Maybe it's a looping script which does many things and you want to only test a single Set Field[] with a complex calculation, so you run the script first without that Set Field[] or with a Set Field [ num ; 1 ] just to get the base but NOT include the complex calc.  Something like that?

[jbante]

Maybe it's a looping script which does many things and you want to only test a single Set Field[] with a complex calculation, so you run the script first without that Set Field[] or with a Set Field [ num ; 1 ] just to get the base but NOT include the complex calc.  Something like that?

 

Exactly. A more common example for me is when I want to compare the execution time of an operation over a large range of inputs. I'll run an operation for all possible inputs in a loop and time that as one test, then I make the null test by simply copying and pasting the whole thing and disabling only the operation being tested to isolate the overhead from the timing and the loop.

[LaRetta]

Thank you for the example!  :-)