Sync Check Speed Improvement - Part 2

[Joshua Willing Halpern]

Hey again. In a previous thread I suggested using javascript in a webviewer to return the difference between the server and client UUID lists in order to increase the sync check speed. It was much faster, but the code I used included an algorithm with quadratic time complexity, meaning that as the size of the lists increased the time that sync check took would grow exponentially. @GisMo helped test confirm that this solution would not scale well.

However I think I've found a bit of javascript that solves this issue and will scale linearly. I've changed the script from the previous thread to use this new code instead. If you are using this javascript method for list comparison I highly recommend you change the sync check script to use this code. Simply open the new file, copy the calculation from set variable[ $url ] in "Sync Check Client", and paste it into your Set Variable[ $url ] step.

Otherwise if you're using EasySync and struggling with sync check, give my solution a try!

I've attached the new file below and linked to the old thread for continuity.

Best, Josh

EDIT: For anyone interested in a demonstration of just the algorithms' performance difference before updating sync check, I've uploaded a raw concept file.

 

EasySync_JWH_SyncCheckMod_LINEAR.zip

Linear_Array_Comparison.fmp12.zip

Edited May 20, 2016 by Joshua Willing Halpern
Added a picture.

[iOllivier]

Hi Joqshua

 

Thanks for the notification.

Will definitely give it a go, as my solutions are growing and any speed up at this point is very welcome.

Glad I am not the only one here who decided to go open-source.

 

J. how are you handling with new version deployment vs data set stored within the mobile part?

My mobile file with initial data is around 50MB now .. and seeing how goZync is handling the deployment is making me thinking about the future.

 

Thanks G.

[Joshua Willing Halpern]

My pleasure. I hope it helps.

I agree that EasySync is a good open source project with strong bones. There is, of course, room for improvement but that is part of the fun and where we come in.  

As for deployment and version updating, I think you should start a new thread for that topic. But that is definitely a facet of EasySync I'm interested in discussing and working to improve.

[GisMo]

@Joshua Willing Halpern Thanks! I will give this a go! The previous jacascript solution is still working well for me and have not had any issues. That said, I'm always up for some performance boosts. I may have some questions when I try to implement this revision.

 

 

 

[Joshua Willing Halpern]

26 minutes ago, GisMo said:

@Joshua Willing Halpern Thanks! I will give this a go! The previous jacascript solution is still working well for me and have not had any issues. That said, I'm always up for some performance boosts. I may have some questions when I try to implement this revision.

Glad to hear it. This version is exactly the same except for the actual $url set variable calculation. Should only take a minute to swap out the calculation and see if it's faster. You could just copy that step, and then replace the calc in one for a/b comparison. 

And I'm happy to try to answer any questions.

[Joshua Willing Halpern]

I forgot to add just the javascript text for anyone who doesn't want to go digging in the attached fm file:

If you're already using the webviewer method, just copy this text into the Set Variable $url step 'Sync Check Client'.

JWH SyncCheck Mod 2.txt

[FM_Newbie]

Hi Joshua,

Apologie in advace for a newbie question. I am an accidental FM developer and newbie to FM.

I recently started using FMEasySync on a medium size database with 10 tables, 2 of them having container fields as well. Number of records are almsot 20000 and usually there are 200 - 300 record every 2 hours to sync. It takes around 10 mins to do this operation every 2 hours. 

my eyes lit up when i saw your post, I followed the steps as you mentioned, created a webviewer on payload layout on client, replaced Sync Check on Client and Server .

I am missing something here, any guidance would be highly appreciated.

 

Cheers

 

[Joshua Willing Halpern]

4 hours ago, FM_Newbie said:

I am missing something here, any guidance would be highly appreciated.

 

Did you name the webviewer "WebViewer"? I may have forgotten to include that step in the instructions.

Also, did you copy the script 'Clear Script Result' into your script workspace?

Edited June 16, 2016 by Joshua Willing Halpern
Forgot something

[FM_Newbie]

Ye, i did. Actually copied the same object from your layout . Also the script "Clear Script Result'' has been added on client as well.

 

just wondering if the variable 
$$use_psos_during_sync_check should be set to be on "Client" or "Server" it is currently set to "Server" in ping settings.

 

[Joshua Willing Halpern]

8 hours ago, FM_Newbie said:

Ye, i did. Actually copied the same object from your layout . Also the script "Clear Script Result'' has been added on client as well.

 

just wondering if the variable 
$$use_psos_during_sync_check should be set to be on "Client" or "Server" it is currently set to "Server" in ping settings.

 

OK cool.

The variable $$use_psos_during_sync_check should be set to either 1 or 0. 1 = yes, perform script on server, 2 = no, perform script on client machine. Did you change any other settings in the settings script? Either is fine though performing the script on client can be very useful for using the debugger if you have filemaker pro advanced.

Can you describe the error/problem you're experiencing more? Where is EasySync failing? Is it getting stuck on the sync check or before that?

 

Also just to double check, did you copy the Sync_Check_Return_Diff script into your solution as well?

 

Edited June 17, 2016 by Joshua Willing Halpern

[FM_Newbie]

Actually overall Sync is taking alot of time, i guess the bottleneck in my case is the internet speed at both sides, specially at the Host side.

EasySync is taking alot of time so i re-deployed it by taking identical copie of the solution and then as both copie update, i am trying to make sure sync is done every 2 hours or less.

Usually "Pushing Payload to the Server" takes alot of time, although there are not many changes on mobile side that are to be pushed. Mostly the update is on the Pull side. Occassionally it gives error of "Couldn't receive Payloa d Segements from customer."

To improve user experience, i have disabled Sync check as i am not using deletion support in the solution yet. Just trying to keep the steps to minimum to keep this working. Should i be using the Sync check anyway??

[Joshua Willing Halpern]

Hey FM_Newbie. A 10 minute sync sounds like Sync Check is the culprit.

Whether you disable sync check is up to you. But if you do, any records you deleted from the server will have to be deleted manually or via a different script. Disabled, you'll need physical access to the client file/device to delete superfluous records or records which no longer "belong" to that device/user.

By using sync check, the administrator has control over what remains on the devices.

If you never/rarely need to delete device records, then go ahead and disable Sync Check. Otherwise, you'll need it. If you're able to get my mod working, sync check will not be a huge burden. On a database with 15 large tables and 30,000 records, sync check takes me ~7 seconds to collect all IDs from server and client and ~0.3 seconds to compare the lists (it used to take about 15 minutes).

It just depends on your needs.

j

Edited June 20, 2016 by Joshua Willing Halpern
misspoke.

[FM_Newbie]

Thanks Josh,

I am not currently doing deletion on Mobile side, plan to do it later and will use your version of Sync Check, it certainly takes lesser time.

Appreciate your help and guidance in this regard.

Cheers

[Joshua Willing Halpern]

Good call. No need to spend time engineering a function you don't need yet.

Let me know if you have any issues when you go to implement.

[FM_Newbie]

Thanks mate,

appreciate your help.

[tmr_slh]

Thanks for posting this. It really helped.

I made few changes. First, I made the UUIDs from the host as part of the payload in the "pull payload" / "prepare payload for client" scripts which gets triggered by a flag in additional pull info. If the client doesn't receive the uuids then it continues as before. If the server doesn't receive the flag then it doesn't send the UUIDs. So things stay compatible both ways.

I also changed the script to work on windows (Internet Explorer 11), old iOS (version 8) and version 9. I found the very hard way that Explorer 11 doesn't fully support ECMAscript 6 which your new code uses and apparently iOS 8 Safari doesn't either. See the attached files.

Also the parameter length limitation in Internet Explorer 11. May be Filemaker on windows 10 is using Edge? I can't verify this now.

One more thing, I found that busy waiting for Script result to come back didn't work on windows. So instead, I halt the script and let the return script continue everything

If [Get(SystemPlatform) = -2] set variable [$url; Value: $windowsScript] Else set variable [$url; Value: $iOSscript]End if

the callback script goes like this

Set Variable [$diff; Value: Get(ScriptParameter)]

If [$diff="Clipboard"]

Paste [No style; EasySync_Payloads::SyncDiff]

Commit Records/Requests [No dialog]

Set Variable[$diff; Value: EasySync_Payloads::SyncDiff]

End If

iOS8Syncdiff.txt

windowsSyncdiff.txt

iOS9Syncdiff.txt

Edited July 13, 2016 by tmr_slh

[Joshua Willing Halpern]

8 minutes ago, tmr_slh said:

I made few changes.

Hey, I'm going to take a look at this tonight; they all sound like great changes, thanks for taking the time to do this.

I believe reigning in the Sync Check speed was the biggest obstacle to EasySync being competitive with Gozync and Mirrorsync and cross-platform compatibility is paramount.

J

[Joshua Willing Halpern]

Hey @tmr_slh, thanks again for your input.

1. Can you clarify for us OSX users what the IE parameter limit is? I see Todd Geist said it was 2048 characters concurrent with version 13.0v2. Is this still the case?

2. Also, do you know why the looping and waiting for script result doesn't work on windows?

3. Also, it appears you're accessing the clipboard in the windows version. I try to avoid this unless we warn the user first. Is there another way to store/pass on the list diff?

4. Could I see a copy of your file with the modified scripts?

[tmr_slh]

23 hours ago, Joshua Willing Halpern said:

Hey @tmr_slh, thanks again for your input.

1. Can you clarify for us OSX users what the IE parameter limit is? I see Todd Geist said it was 2048 characters concurrent with version 13.0v2. Is this still the case?

2. Also, do you know why the looping and waiting for script result doesn't work on windows?

3. Also, it appears you're accessing the clipboard in the windows version. I try to avoid this unless we warn the user first. Is there another way to store/pass on the list diff?

4. Could I see a copy of your file with the modified scripts?

I'll post the whole file soon.

1. The limit is not from FileMaker itself but from internet explorer. Total URL length around 2048 characters as you mentioned. If FileMaker is using Microsoft Edge instead in the newer version, things will be different but I don't know different in what way.

2. The only way to explain why or why not it works is multi threading. After spending a whole day trying to get it to work, my only conclusion is that FileMaker on Windows doesn't launch the web viewer in a separate thread while it does just that on Mac. Extremely annoying and I have no way of confirming this other than the fact that it doesn't work. However, may be you will agree that a callback that continues processing is better than loop-pause.

3. Not that I know of. The commercial products from seed code and Todd Geist are using the clipboard and I copied it from one of their blog posts.

[Joshua Willing Halpern]

15 hours ago, tmr_slh said:

3. Not that I know of. The commercial products from seed code and Todd Geist are using the clipboard and I copied it from one of their blog posts.

In that case, perhaps we should wrap the previous clipboard contents, recopy and pass that along to the callback script as well. Then we could restore the clipboard after retrieving the UUID diff.

[GisMo]

On 5/31/2016 at 5:25 PM, Joshua Willing Halpern said:

I forgot to add just the javascript text for anyone who doesn't want to go digging in the attached fm file:

If you're already using the webviewer method, just copy this text into the Set Variable $url step 'Sync Check Client'.

JWH SyncCheck Mod 2.txt

@Joshua Willing Halpern A+! I just implemented the new code. Significantly faster! Thank you for your awesome work!

[Joshua Willing Halpern]

1 hour ago, GisMo said:

@Joshua Willing Halpern A+! I just implemented the new code. Significantly faster! Thank you for your awesome work!

Glad it's working for you! You may want to read tmr_slh's comment a couple entries up and verify that this modification works on all platforms you need it for. If not his suggestions look promising though I haven't tried them and I don't know that they all have the same linear complexity. Cheers.

[bcooney]

On 7/19/2016 at 1:09 AM, tmr_slh said:

I'll post the whole file soon.

@tmr_slh would love to see your version, tmr. We too ran into failure on Windows.

[sfpx]

How's the speed of EasySync compared to Mirrorsync ?

We like Mirrorsync but we had troubles recently and the fact that it's an external app makes it harder to debug .

We're looking at other products while we're waiting for a fix.

 

[Joshua Willing Halpern]

I'd say Mirrorsync is the quickest sync tool I've come across. But Easysync can be a great solution--how many records will be in your sync payloads, and across how many tables?

Edited July 5, 2017 by Joshua Willing Halpern
word for clarity