Jump to content

Speed testing


LaRetta
 Share

This topic is 2830 days old. Please don't post here. Open a new topic instead.

Recommended Posts

When running speed tests in a file, will Refresh Window [ flush cached join results ] completely reset a file?  Is cache the only thing which might affect speed test?

 

Because of my anal nature, I've always closed FileMaker and I used to also restart my computer between each test.  Is there a way I can tell what resources each of these processes clear on Mac Mavericks, i.e. does closing FileMaker free more system resources which might affect the test than just flush cache?  Does restarting the computer influence?

 

I know that, during speed tests, all other processes should be shut down, such as backups, searching internet etc. or the effects can influence speed tests but how can I be absolutely sure all tests are equal to start?

Link to comment
Share on other sites

There are two ways to see what you want through the hazing of confounding variables: control them, and average them out. The latter induces much less anxiety. You can only control what you know about (and spend a lot of time in the process), but you can average out even factors you don't know about. (This is not to say that you can average out all factors you don't know about, but almost always some of them.) In an A/B comparison, I might run each 5 times in random order. If one test run had a substantial benefit the others didn't, it will be an obvious outlier. I'll typically test smaller operations in much larger loops just so differences between solutions add up to something measurable. In this case, the central limit theorem takes care of any variation due to whatever else the computer is working on at the time.

 

One other thing I usually make a point of doing that more in the spirit of controlling than averaging is a "Null" test (not a null hypothesis) where I measure only how long it takes to run the test overhead, without the operation actually being tested. The overhead of running the test can otherwise make differences seem smaller than they actually are.

  • Like 1
Link to comment
Share on other sites

Hey Jeremy!  Thank you! 

 

Then I'll just begin my speed test with Refresh Window [ Flush cached...] and run many tests and take the average.  Ah, and the Null test makes sense although I'm not sure it would always be possible.  For example, I was comparing two scripts:

Refresh Window [ Flush cached join results ]
Set Error Capture [ On ]
Constrain Found Set [ OMIT ; uniqueID ; * ]

to this script:

Refresh Window [ Flush cached join results ]
Go To Record/Request/Page [ first ]
Omit Multiple [ Get ( FoundCount ) 

I could immediately tell that the Constrain would take longer (and much longer if it wasn't already indexed) because I wanted watched the indexing progress bar.   On a million records, the Omit Multiple took 2 seconds but the Constrain took 13 seconds.  :-)


If you could think of an example of using a null test, I would appreciate it.  I have a pretty good idea but just can't think of an example.  Maybe it's a looping script which does many things and you want to only test a single Set Field[] with a complex calculation, so you run the script first without that Set Field[] or with a Set Field [ num ; 1 ] just to get the base but NOT include the complex calc.  Something like that?

Link to comment
Share on other sites

Maybe it's a looping script which does many things and you want to only test a single Set Field[] with a complex calculation, so you run the script first without that Set Field[] or with a Set Field [ num ; 1 ] just to get the base but NOT include the complex calc.  Something like that?

 

Exactly. A more common example for me is when I want to compare the execution time of an operation over a large range of inputs. I'll run an operation for all possible inputs in a loop and time that as one test, then I make the null test by simply copying and pasting the whole thing and disabling only the operation being tested to isolate the overhead from the timing and the loop.

post-103690-0-49181200-1425442430_thumb.

Link to comment
Share on other sites

This topic is 2830 days old. Please don't post here. Open a new topic instead.

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

×
×
  • Create New...

Important Information

By using this site, you agree to our Terms of Use.