Joshua Willing Halpern

Sync Check Speed Improvement - Part 2

23 posts in this topic

Hey again. In a previous thread I suggested using javascript in a webviewer to return the difference between the server and client UUID lists in order to increase the sync check speed. It was much faster, but the code I used included an algorithm with quadratic time complexity, meaning that as the size of the lists increased the time that sync check took would grow exponentially. @GisMo helped test confirm that this solution would not scale well.

However I think I've found a bit of javascript that solves this issue and will scale linearly. I've changed the script from the previous thread to use this new code instead. If you are using this javascript method for list comparison I highly recommend you change the sync check script to use this code. Simply open the new file, copy the calculation from set variable[ $url ] in "Sync Check Client", and paste it into your Set Variable[ $url ] step.

Otherwise if you're using EasySync and struggling with sync check, give my solution a try!

I've attached the new file below and linked to the old thread for continuity.

Best, Josh

EDIT: For anyone interested in a demonstration of just the algorithms' performance difference before updating sync check, I've uploaded a raw concept file.

 

EasySync_JWH_SyncCheckMod_LINEAR.zip

Linear_Array_Comparison.fmp12.zip

Screen Shot 2016-05-20 at 2.40.53 AM.png

Edited by Joshua Willing Halpern
Added a picture.
2 people like this

Share this post


Link to post
Share on other sites

Hi Joqshua

 

Thanks for the notification.

Will definitely give it a go, as my solutions are growing and any speed up at this point is very welcome.

Glad I am not the only one here who decided to go open-source.

 

J. how are you handling with new version deployment vs data set stored within the mobile part?

My mobile file with initial data is around 50MB now .. and seeing how goZync is handling the deployment is making me thinking about the future.

 

Thanks G.

Share this post


Link to post
Share on other sites

My pleasure. I hope it helps.

I agree that EasySync is a good open source project with strong bones. There is, of course, room for improvement but that is part of the fun and where we come in. :) 

As for deployment and version updating, I think you should start a new thread for that topic. But that is definitely a facet of EasySync I'm interested in discussing and working to improve.

Share this post


Link to post
Share on other sites

@Joshua Willing Halpern Thanks! I will give this a go! The previous jacascript solution is still working well for me and have not had any issues. That said, I'm always up for some performance boosts. I may have some questions when I try to implement this revision.

 

 

 

Share this post


Link to post
Share on other sites
26 minutes ago, GisMo said:

@Joshua Willing Halpern Thanks! I will give this a go! The previous jacascript solution is still working well for me and have not had any issues. That said, I'm always up for some performance boosts. I may have some questions when I try to implement this revision.

Glad to hear it. This version is exactly the same except for the actual $url set variable calculation. Should only take a minute to swap out the calculation and see if it's faster. You could just copy that step, and then replace the calc in one for a/b comparison. 

And I'm happy to try to answer any questions.

Share this post


Link to post
Share on other sites

I forgot to add just the javascript text for anyone who doesn't want to go digging in the attached fm file:

If you're already using the webviewer method, just copy this text into the Set Variable $url step 'Sync Check Client'.

JWH SyncCheck Mod 2.txt

1 person likes this

Share this post


Link to post
Share on other sites

Hi Joshua,

Apologie in advace for a newbie question. I am an accidental FM developer and newbie to FM.

I recently started using FMEasySync on a medium size database with 10 tables, 2 of them having container fields as well. Number of records are almsot 20000 and usually there are 200 - 300 record every 2 hours to sync. It takes around 10 mins to do this operation every 2 hours. 

my eyes lit up when i saw your post, I followed the steps as you mentioned, created a webviewer on payload layout on client, replaced Sync Check on Client and Server .

I am missing something here, any guidance would be highly appreciated.

 

Cheers

 

Share this post


Link to post
Share on other sites
4 hours ago, FM_Newbie said:

I am missing something here, any guidance would be highly appreciated.

 

Did you name the webviewer "WebViewer"? I may have forgotten to include that step in the instructions.

Also, did you copy the script 'Clear Script Result' into your script workspace?

Edited by Joshua Willing Halpern
Forgot something

Share this post


Link to post
Share on other sites

Ye, i did. Actually copied the same object from your layout . Also the script "Clear Script Result'' has been added on client as well.

 

just wondering if the variable 
$$use_psos_during_sync_check should be set to be on "Client" or "Server" it is currently set to "Server" in ping settings.

 

Share this post


Link to post
Share on other sites
8 hours ago, FM_Newbie said:

Ye, i did. Actually copied the same object from your layout . Also the script "Clear Script Result'' has been added on client as well.

 

just wondering if the variable 
$$use_psos_during_sync_check should be set to be on "Client" or "Server" it is currently set to "Server" in ping settings.

 

OK cool.

The variable $$use_psos_during_sync_check should be set to either 1 or 0. 1 = yes, perform script on server, 2 = no, perform script on client machine. Did you change any other settings in the settings script? Either is fine though performing the script on client can be very useful for using the debugger if you have filemaker pro advanced.

Can you describe the error/problem you're experiencing more? Where is EasySync failing? Is it getting stuck on the sync check or before that?

 

Also just to double check, did you copy the Sync_Check_Return_Diff script into your solution as well?

 

Edited by Joshua Willing Halpern

Share this post


Link to post
Share on other sites

Actually overall Sync is taking alot of time, i guess the bottleneck in my case is the internet speed at both sides, specially at the Host side.

EasySync is taking alot of time so i re-deployed it by taking identical copie of the solution and then as both copie update, i am trying to make sure sync is done every 2 hours or less.

Usually "Pushing Payload to the Server" takes alot of time, although there are not many changes on mobile side that are to be pushed. Mostly the update is on the Pull side. Occassionally it gives error of "Couldn't receive Payloa d Segements from customer."

To improve user experience, i have disabled Sync check as i am not using deletion support in the solution yet. Just trying to keep the steps to minimum to keep this working. Should i be using the Sync check anyway??

Share this post


Link to post
Share on other sites

Hey FM_Newbie. A 10 minute sync sounds like Sync Check is the culprit.

Whether you disable sync check is up to you. But if you do, any records you deleted from the server will have to be deleted manually or via a different script. Disabled, you'll need physical access to the client file/device to delete superfluous records or records which no longer "belong" to that device/user.

By using sync check, the administrator has control over what remains on the devices.

If you never/rarely need to delete device records, then go ahead and disable Sync Check. Otherwise, you'll need it. If you're able to get my mod working, sync check will not be a huge burden. On a database with 15 large tables and 30,000 records, sync check takes me ~7 seconds to collect all IDs from server and client and ~0.3 seconds to compare the lists (it used to take about 15 minutes).

It just depends on your needs. ;)

j

Edited by Joshua Willing Halpern
misspoke.

Share this post


Link to post
Share on other sites

Thanks Josh,

I am not currently doing deletion on Mobile side, plan to do it later and will use your version of Sync Check, it certainly takes lesser time.

Appreciate your help and guidance in this regard.

Cheers

Share this post


Link to post
Share on other sites

Good call. No need to spend time engineering a function you don't need yet.

Let me know if you have any issues when you go to implement.

Share this post


Link to post
Share on other sites

Thanks mate,

appreciate your help.

Share this post


Link to post
Share on other sites

Thanks for posting this. It really helped.

I made few changes. First, I made the UUIDs from the host as part of the payload in the "pull payload" / "prepare payload for client" scripts which gets triggered by a flag in additional pull info. If the client doesn't receive the uuids then it continues as before. If the server doesn't receive the flag then it doesn't send the UUIDs. So things stay compatible both ways.

I also changed the script to work on windows (Internet Explorer 11), old iOS (version 8) and version 9. I found the very hard way that Explorer 11 doesn't fully support ECMAscript 6 which your new code uses and apparently iOS 8 Safari doesn't either. See the attached files.

Also the parameter length limitation in Internet Explorer 11. May be Filemaker on windows 10 is using Edge? I can't verify this now.

One more thing, I found that busy waiting for Script result to come back didn't work on windows. So instead, I halt the script and let the return script continue everything

If [Get(SystemPlatform) = -2] set variable [$url; Value: $windowsScript] Else set variable [$url; Value: $iOSscript]End if

the callback script goes like this

Set Variable [$diff; Value: Get(ScriptParameter)]

If [$diff="Clipboard"]

Paste [No style; EasySync_Payloads::SyncDiff]

Commit Records/Requests [No dialog]

Set Variable[$diff; Value: EasySync_Payloads::SyncDiff]

End If

iOS8Syncdiff.txt

windowsSyncdiff.txt

iOS9Syncdiff.txt

Edited by tmr_slh

Share this post


Link to post
Share on other sites
8 minutes ago, tmr_slh said:

I made few changes.

Hey, I'm going to take a look at this tonight; they all sound like great changes, thanks for taking the time to do this.

I believe reigning in the Sync Check speed was the biggest obstacle to EasySync being competitive with Gozync and Mirrorsync and cross-platform compatibility is paramount.

J

Share this post


Link to post
Share on other sites

Hey @tmr_slh, thanks again for your input.

1. Can you clarify for us OSX users what the IE parameter limit is? I see Todd Geist said it was 2048 characters concurrent with version 13.0v2. Is this still the case?

2. Also, do you know why the looping and waiting for script result doesn't work on windows?

3. Also, it appears you're accessing the clipboard in the windows version. I try to avoid this unless we warn the user first. Is there another way to store/pass on the list diff?

4. Could I see a copy of your file with the modified scripts?

Share this post


Link to post
Share on other sites
23 hours ago, Joshua Willing Halpern said:

Hey @tmr_slh, thanks again for your input.

1. Can you clarify for us OSX users what the IE parameter limit is? I see Todd Geist said it was 2048 characters concurrent with version 13.0v2. Is this still the case?

2. Also, do you know why the looping and waiting for script result doesn't work on windows?

3. Also, it appears you're accessing the clipboard in the windows version. I try to avoid this unless we warn the user first. Is there another way to store/pass on the list diff?

4. Could I see a copy of your file with the modified scripts?

I'll post the whole file soon.

1. The limit is not from FileMaker itself but from internet explorer. Total URL length around 2048 characters as you mentioned. If FileMaker is using Microsoft Edge instead in the newer version, things will be different but I don't know different in what way.

2. The only way to explain why or why not it works is multi threading. After spending a whole day trying to get it to work, my only conclusion is that FileMaker on Windows doesn't launch the web viewer in a separate thread while it does just that on Mac. Extremely annoying and I have no way of confirming this other than the fact that it doesn't work. However, may be you will agree that a callback that continues processing is better than loop-pause.

3. Not that I know of. The commercial products from seed code and Todd Geist are using the clipboard and I copied it from one of their blog posts.

Share this post


Link to post
Share on other sites
15 hours ago, tmr_slh said:

3. Not that I know of. The commercial products from seed code and Todd Geist are using the clipboard and I copied it from one of their blog posts.

In that case, perhaps we should wrap the previous clipboard contents, recopy and pass that along to the callback script as well. Then we could restore the clipboard after retrieving the UUID diff.

1 person likes this

Share this post


Link to post
Share on other sites
On 5/31/2016 at 5:25 PM, Joshua Willing Halpern said:

I forgot to add just the javascript text for anyone who doesn't want to go digging in the attached fm file:

If you're already using the webviewer method, just copy this text into the Set Variable $url step 'Sync Check Client'.

JWH SyncCheck Mod 2.txt

@Joshua Willing Halpern A+! I just implemented the new code. Significantly faster! Thank you for your awesome work!

Share this post


Link to post
Share on other sites
1 hour ago, GisMo said:

@Joshua Willing Halpern A+! I just implemented the new code. Significantly faster! Thank you for your awesome work!

Glad it's working for you! You may want to read tmr_slh's comment a couple entries up and verify that this modification works on all platforms you need it for. If not his suggestions look promising though I haven't tried them and I don't know that they all have the same linear complexity. Cheers.

Share this post


Link to post
Share on other sites
On 7/19/2016 at 1:09 AM, tmr_slh said:

I'll post the whole file soon.

@tmr_slh would love to see your version, tmr. We too ran into failure on Windows.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now

  • Similar Content

    • By Gianluca D'Aquino
      Hello there,
      I've set up 2 files syncing with Easysync. I've set up just a single table in sync called ES_TRIP (without pull or push, because I want to sync in both directions, is that correct).
      I put the same cfg on mobile and on host file and the process start working but it throws me an error:
      "The attempt to push data to the server has failed. (The message received from the server was: "111A01C0-27AE-414B-AC1C-DC56C83F877E"
      The hosted file is on a fm server 15 that is reachable and working - I've tried the hosted link and it is correctly working.
      The TRIP child table is set to add record (but not delete) on both files.
      What should I check on it?
      Thanks!
       
    • By TJ53
      Let’s say we have two related tables: “Invoice” and “Invoice_Item”. We could create a calculation field in the “Invoice” table called “total_amount” with this formula:
      total_amount = Sum (Invoice_Item::amount)
      This field would have a negative impact in performance when appearing in the layout, since it would have to be defined as unstored, because it’s referencing a field from a related table.
      Now let’s suppose this field is not used for any scripts, tooltips, conditional format, etc … would the performance of the database be negatively affected ONLY when this field appeared in a layout? 
      In other words, would adding an unstored calculation field to a table involve a performance penalty, even in the “unreal” case where this field didn’t appear in any layout, script, conditional format, etc.? thanks in advance!
    • By TJ53
      Is there any difference in terms of performance between a calculation field (stored and indexed) and field defined as auto-enter calculated value (indexed)?
      For example, we have an “INVOICE” table, with a field called “date_invoice_sent”, and we’d like to have a boolean field called “is_sent”.
      The calculation would be “not IsEmpty(date_invoice_sent)”
      So we have two options here:
      - Calculation field (stored, number result).
      - Number field defined as “auto-enter / calculated value / do not replace … unchecked”.
      Would there be any difference in performance between the two options? thanks in advance!
    • By FM_Newbie
      Hi Guys,
      I had been using FM EasySync for few months now, so it had been working fine till now.
      Without having any change in the table structure, i got the error " The attempt to push data into server has failed ( The mesage recevied rom the server was "852" )
      852 Error is :  Cannot write a file to the external storage
      I believe this is related to container fileds I am using. There has been no change in the Container field options. They are still being stored "Externally" on the server with Open Storage option. Also there hasn't been any change on Server policies as well.
      I have no clue as to what could have cause this. Anyone having idea about this ??
      Thanks much.
    • By JerrySalem
      I have a system being hosted using FMS14.  I also have a duplicate database hosted for testing/exporting using FMS15. PSOS is much slower in my hands with FMS15.
      Both servers are physical boxes, with similar (maybe identical) specs (Cores, RAM HD Space, HD Space available)
      I also have a Server Side script that creates a found set, exports data to excel and emails it to a user.  (This is based on https://www.skeletonkey.com/restoring-filemaker-clients-found-set-within-server-side-script/).  The exports have a couple of related fields, but no unstirred calculations.
      Using FMS14 this has been working just fine.  Using FMS15 it has been significantly slower.
      As a test, I exported 13,000 records from both FMS14 and FMS15.
      Using FMS14, it takes about 10 Seconds to recreate the found set, then 70 Seconds to create the export.  
      Using FMS15, it takes about 20 Seconds to recreate the (same) found set, then 175 Seconds to export the data.
      In another series of tests, this time exporting 30,000 records (more realistic in my scenario) I found;
      Using FMS14, it takes about 25 Seconds to recreate the found set, then 168 Seconds to create the export.  (2.5x longer to find/2x longer to export 3x records)
      Using FMS15, it takes about 90 Seconds to recreate the (same) found set, then 825 Seconds to export the data. 9x longer to find/11x longer to export 3x records)
      In the last series of tests, this time exporting 50,000 records I found;
      Using FMS14, it takes about 59 Seconds to recreate the found set, then 262 Seconds to create the export.  (6x longer to find/4x longer to export 5x records)
      Using FMS15, it takes about 160 Seconds to recreate the (same) found set, then 1700 Seconds to export the data.(16x to find/24x to export 5x records)
      Any ideas?  Anyone see similar results with PSOS on FMS15?  
      Again, the machines are identical, the databases are identical and the scripts are identical.  the only difference is FMS14 vs. FMS15.  I am also letting another user pull data using ODBC.  That has also gotten extremely slow using FMS15, but that is a discussion for another thread.  I am not using WebD on this server.  In general the FMS15 database performs find using FMP Clients.  But the FMS15 functions are not very impressive.  I am petrified to move my production database to FMS15, and even considering moving the test server back to FMS14.  Help!
       
      TIA
      Jerry