Benchmarking FM12 and FM11

[capsprojectos]

I would like to share with this forum some benchmarking results, between Filemaker 12 and Filemaker 11. And if possible to confirm this conclusions with other FM 12 users.

Scenario:

Hardware: Core i7, L620, 2GHz, 8Gb RAM.

OS: Windows 7 PRO 64bits

File: One layout, 180 fields, 1 script with 178 variables.

Test scenario: Run the script, with script debugger and Data Viewer (current tab) opened, showing 10 lines.

Results:

With FM11, using the Step Into shortcut key (F6), I finish this test in 1:52 (one minute and 52 seconds).

After convert the main file to the new FM 12 format, I did the same test:

With FM12, using the Step Into shortcut key (F6), I finish this test in 19:38 (nineteen minutes and 38 seconds).

Conclusion: FM12 spend 10 times longer than FM11 (17 minutes and 46 seconds).

Thanks,

test.zip

[Vaughan]

The same thing happened when FMP 7 replaced FMP 6: some things were significantly slower while some things were faster. When FMP 7 was first released there was no compelling reason to update.

So be patient, within 6 months most of the performance issues will be solved and there will be lots of compelling reasons to be getting everything onto FMP 12. :D

[genevieve charbon]

> capsprojectos

Please send this here :

http://forums.filemaker.com/posts/715ef37320

This os the thread FM Inc monitors for those speed issues

[capsprojectos]

genevieve,

First i would like to confirm this with some other FM12 user, because without corroboration I shouldn't affirm this. So for now, i wish to have other benchmarking tests.

If there is a real issue, then i will take your sugestion.

[jbante]

I could not reproduce your result. I ran the test, with some modifications to make the protocol more robust:

• To reduce measurement error from the lack of sub-second FileMaker-native time measurement, I wrote a script that pauses for the start of the next second, and called it at the beginning of your test.
  
• I ran the test without the script debugger or data viewer, to more closely reflect the demands placed on the system during the end-user experience.
  
• I enclosed the operations being tested in a loop that repeats 10,000 times to make the timing long enough to be measurable. (It ran in less than 1 second in both versions without this.)
  

My results? FileMaker 11 took 1:25 to run the test. FileMaker 12 took 1:23 to run the test — the difference is within what I might expect if another application was doing something extra in the background during the first test of FileMaker 11. If there is a performance difference, it's in the debugger; so I ran the test again as you described running it by stepping through it with the debugger. When I tested FileMaker 11, it took me 28 seconds; when I tested with FileMaker 12, it took me 35 seconds. It's a better test of how fast I can hit the F6 key when tired vs. fresh than a test of how fast FileMaker is executing the operations.

35 seconds is much faster than I ever need to get through a script that size step-by-step. The computer only has to be as fast as the human user is capable of perceiving and processing. It takes in the neighborhood of 0.1 seconds for a human to perceive that anything happened — longer to recognize what happened, and even longer to do anything about it. As long as the debugger can step through script steps close to that fast, I'm satisfied.

For those who care, I'm running a 2009-era MacBook Pro with 2.8 GHz Core 2 Duo and 4 GB RAM with Mac OS 10.7.3.

[grumbachr]

I did not see any slow down between fm 11 and 12.

if there is a difference it's imperceptible to me.

[capsprojectos]

Jeremy Bante,

Thanks for your review.

Let me explain better, because maybe I didn’t make my post clear, the aim of my test was to compare the performance of Debugger (dataviewer) in both Filemaker version, because when I was testing FM12, I found this last version much slower than previous (FM11).

With me this happens often, when there are too much variables or fields in a script, and I’m using dataviewer to review something, the FM12 crash and takes too long to go for the next step.

I can run the script and the performance is similar with FM11, but with dataviewer is very differente (10 times slower).

1. Did you test my files? Can you share the results?
   

In your second experiment, you used debugger and you didn’t found any difference?

1. Can you share your file here?
   
2. Do you have the opportunity to test my file in a Windows OS?
   

Thanks

grumbachr,

Which platform did you test these files?

Thanks,

[grumbachr]

capsprojectos,

The piece of info about having the Data Viewer open while stepping through is the key.

With Debugger and no Data Viewer the difference is no existent. Stepping through with Data Viewer open I see what you're talking about, that is painful ever step i see the spinning beach ball.

Is it odd that with Data Viewer open and no debugger it performs just as fast. It's as if Debugger and Data Viewer aren't getting along.

As for systems, i'm i5(dual core)/2.4 GHz /4GB/ OSX 10.7.3

Oops I replied to Jeremy, meant to reply to capsprojectos (fixed that).

Edited April 26, 2012 by grumbachr

[capsprojectos]

Jeremy, can you share your results?

With Fm11 how long and with FM12?

Thanks,

[jbante]

I tested again, this time with both the Data Viewer and Debugger open, instead of just the debugger or neither as before. It took me 26 seconds to step through the script in FileMaker 11. In FileMaker 12, it slowed down so much and so obviously that it wasn't worth my time to finish the test — several seconds per step. I also tested each script (without the modifications I mentioned earlier) with the Data Viewer open, but not the Debugger: both finished in less than 1 second. The results I got from my previous tests are the results I already shared above.

[capsprojectos]

After confirm this with other users. I did a post at FMTech - Performance Tuning, in other to send this information to FM tech guys.

Now we will have to wait.