Sync Check Speed Improvement

[Joshua Willing Halpern]

Hey Fellow Easy Sync-ers. I have an idea for improving the speed of the Sync Check step.

Instead of comparing UUID lists in loops, we can retrieve a UUID list from server, and compare it to the client UUIDs using a couple lines of JavaScript in a webviewer. I've tested this and my Sync check time comparing ~23000 records, over 18 sync tables on a shared server has gone from 5-10 minutes down to a few seconds. No plugins necessary.

Process:

1. Get UUIDs_client.
2. Get full UUIDs_host list from server
3. Format both as JavaScript arrays and put them into a web viewer to be compared using 'filter'
4. Have the webviewer call another script using FMP url to return the resulting diff array as a list.
5. Process the diff like normal (The way Tim wrote it).

I also updated the Sync Check process to pull the Server UUIDs in segments if they exceed the $$max_pull_segment_size setting.

Here are my example files. Try adding a bunch of records and then run the sync to see how quick the sync check is.

PLEASE NOTE: This method requires a named webviewer on the client EasySync_Payloads layout.

 

EasySync v1.3 mod1 (jwillinghalpern@gmail).zip

[dansmith65]

I didn't test it, but it sounds promising. I added an issue so I don't forget to look into adding this concept to my fork of EasySync: https://github.com/dansmith65/FileMaker-EasySync/issues/29

[Joshua Willing Halpern]

Hey Dan, I've been following your posts in this thread and have visited your repo once or twice. I intended to add this to your issue list (so thanks) and contribute but wanted to see if anyone had insight first. My main concern is finding a way to handle if the webviewer url doesn't successfully call the Sync_Check_Return_Diff script. I'm doing it with a loop that exits when a script result is found with some basic timeout logic.

Lemme know if you get a chance to test it. It's working wonders for me and I'm already running it on one of my client's systems.

J

[GisMo]

On 1/23/2016 at 9:32 PM, Joshua Willing Halpern said:

Hey Dan, I've been following your posts in this thread and have visited your repo once or twice. I intended to add this to your issue list (so thanks) and contribute but wanted to see if anyone had insight first. My main concern is finding a way to handle if the webviewer url doesn't successfully call the Sync_Check_Return_Diff script. I'm doing it with a loop that exits when a script result is found with some basic timeout logic.

Lemme know if you get a chance to test it. It's working wonders for me and I'm already running it on one of my client's systems.

J

Hey Josh,

I really want to try this. Can you foresee any issues?

Also, can you please post a little step-by-step on what to modify/add in both hosted and local database files to make this work?

 

Thanks!

 

 

 

 

[Joshua Willing Halpern]

Hey @GisMo, this method is working great for me and my clients. Honestly, for us it is the difference between easysync being usable and not. I've had no issues.

The only issue I can imagine is this. The javascript runs in a web viewer and then calls a script using fmp url with the diff as a parameter. The called script simply returns the Parameter as a ScriptResult to the original script. Because fmp url calls scripts in a new thread, the only way (that I know) to wait for the called script to complete is to loop in the original script and wait for a new script result. If for some reason the fmp url fails to call the script we'll be looping forever. Therefore we need a timeout provision.

The way I addressed this was to give the javascript 35 seconds to find the diff and call the script which returns that diff. I also gave it 3 chances to do this (It's never exceeded the first chance though and this may be overkill). However, 35 seconds is an arbitrary number, so while in my case a 35 second limit is way more than enough, Much larger record sets might require a longer timeout limit. I don't know because I haven't tested.

I'll be interested to hear if you had to set this number differently to accommodate longer ID lists.

AS FOR STEP-BY-STEP: I think the easiest way to try this out is to copy my attached example file. Here's the basics.

1. Place a webviewer on your easysync payloads layout and call it "WebViewer". I'm not sure if it will work as an offscreen object.
2. Update both your "Sync Check Client" and "Sync Check" scripts like I did in the example file. Be sure to hook up your Hosted Database PSOS steps if the external connection broke. You can probably just copy paste my work.

That's it. Please let me know what you think and if you have any questions/ideas for improvement.

Best, Josh

EasySync_JWH_SyncCheckMod.zip

[GisMo]

Hey Josh,

What's the Admin user/pass for this file?

Edited February 15, 2016 by GisMo

[Joshua Willing Halpern]

account: "admin"

password is blank.

[GisMo]

Hey Josh,

Thanks. I'm in the file.

I've noticed you've renamed some of the scripts so updating some of the existing scripts in my heavily modified client file place is going to be tricky and I don't want to miss something. 

Here are the scripts what I've found to be modified,renamed, or added for the client side:

• EasySync Settings Client
• Sync With Server
• Sync Check Client
• Clear Script Result
• Sync_Check_Return_Diff

Can you confirm these are the ONLY modified or added scripts on the client side?

For the guys out there like me looking to upgrade an in-place setup, we need to be careful.

I haven't looked at the server scripts yet.

[Joshua Willing Halpern]

15 minutes ago, GisMo said:

Hey Josh,

Thanks. I'm in the file.

I've noticed you've renamed some of the scripts so updating some of the existing scripts in my heavily modified client file place is going to be tricky and I don't want to miss something. 

Here are the scripts what I've found to be modified,renamed, or added for the client side:

• EasySync Settings Client
• Sync With Server
• Sync Check Client
• Clear Script Result
• Sync_Check_Return_Diff

Can you confirm these are the ONLY modified or added scripts on the client side?

For the guys out there like me looking to upgrade an in-place setup, we need to be careful.

I haven't looked at the server scripts yet.

I suppose I wasn't very clear with the instructions haha. I believe there are literally only 3 things you need to do:

1. Add the webviewer named "Webviewer" to the payloads layout on the client.
2. Modify your client-side sync check script. Use the script steps in my "Sync Check Client" script.
3. Modify your server-side sync check script using the steps in my "Sync Check" script.

The only two scripts that need to be modified are flagged in the script workspace with ***UPDATE THIS SCRIPT***. The renamed scripts are so that I don't need dedicated files for server and client. You can disregard those and if you have dedicated files, you don't need to rename your scripts as long as the correct scripts are being called.

This probably goes with out saying, but I don't know what "in-place setup" means, so I'd caution against testing this on your live file unless you can afford some sync downtime. If you are working on a non-dev file, just make sure you back everything up.

J

Edited February 15, 2016 by Joshua Willing Halpern

[GisMo]

Hey Josh,

I do see some additional code you added to "Sync With Server" after the Sync Check Script is called.

If GetValue(GetScriptResult);1)=0

Show Custom Dialog...

End If

This seems harmless.

Also, need to add the script  "Clear ScriptResult" to the client database.

Seems like there are some modifications to "EasySync Delete Record" Scipt As well. Checking the timestamp, but it seems to be bypassed.

 

Are there any other little bits like this added elsewhere?

Edited February 16, 2016 by GisMo

[Joshua Willing Halpern]

Oh, good catch. The clear script result is required. The dialog in sync with server is only if you want to tell them sync check failed, if I remember.

You can ignore the delete mod. I added it so client doesn't tell server to try to delete records it doesn't even have yet. 

Good detective work on the clear scriptresult script. I think that was the only missing piece.

i like to use the monkeybread software plugin to search the script text for "unknown" and "missing" when hooking up copied scripts. It should reveal any other loose ends like that missing script.

Best, j 

[GisMo]

Hey Josh,

I just tested this on backup copies of the working files. To get an idea of the records involved I've attached an image. It takes two minutes for the sync check to complete with your method! A vast improvement! Bravo. It seems to be working correctly as well. I've only tested this on desktop, not FMGO.

We may want to consider an 'allow user abort' set to False during the sync check. I'm not sure where the right place to put that script step would be.

 

Edited February 16, 2016 by GisMo

[Joshua Willing Halpern]

Hey, glad it's working!

For some context, I'm syncing 27,000 records over 17 tables with 10 - 65 fields each. My time for the sync check alone is 15 seconds. It sounds like the completion time does not grow linearly with more records, but rather exponentially.

I created a $$sync_check_time_log variable where I time the different steps of the sync check. E.g. client ID collection, server ID collection, time to return diff, and time to process deletions. my breakdown is:

• Sync check start: 2/16/2016 11:36:39 AM
• Client ID collection finished: 2/16/2016 11:36:40 AM (1 second)
• Server Sync Check script finished: 2/16/2016 11:36:47 AM (7 seconds)
• Server ID collection finished: 2/16/2016 11:36:50 AM (3 seconds)
• Diff returned: 2/16/2016 11:36:54 AM (4 seconds)

My bottleneck appears to be the connection speed to the server. I'd be interested to hear a breakdown of your runtime to see if the culprit is indeed in the webviewer javascript portion.

And yes, I agree that giving the user an option to stop the sync check if it's taking too long might be nice, but I'm not sure where that would go either. Perhaps a Cancel Button appears on the layout during the loop waiting for the diff to be returned? That button might just halt the sync check script.

Best, Josh

Edited February 16, 2016 by Joshua Willing Halpern

[GisMo]

Hey Josh,

Where do you put the sync_check_time_log? I'll post my results.

 

[Joshua Willing Halpern]

45 minutes ago, GisMo said:

Hey Josh,

Where do you put the sync_check_time_log? I'll post my results.

 

I updated the variable in a few spots. I don't think it needs to be exact as long as the placement is after the big chunks of the script.

I've attached a text file with the log steps in bold if you want. Don't mind the funny "synch" spelling.

J

Sync Check Log Placement.rtf

[GisMo]

Thanks!

 

Here are my results. As expected, the diff it consuming most of the time

Sync check start: 2/17/2016 9:45:35 AM
Client ID collection finished: 2/17/2016 9:45:36 AM
Server Sync Check script finished: 2/17/2016 9:45:45 AM
Server ID collection finished: 2/17/2016 9:45:45 AM
Diff returned: 2/17/2016 9:47:37 AM

 

Edited February 17, 2016 by GisMo

[Joshua Willing Halpern]

Ah, alright. It seems that the current concept of sync check comparing all db IDs is not particularly scalable. Unless someone can provide a JavaScript algorithm superior to a nested loop or the JS filter I used, then we'll see this exponential slowing.  

Anyway, at least this dramatically speeds things up for now!

[GisMo]

Hey Josh,

It's certainly faster. I'm no JS expert. Would you consider tabling that issue of increasing performance on Stackoverflow.com ? If so, let me know and I'll up vote your question. 

Edited February 17, 2016 by GisMo

[Joshua Willing Halpern]

1 minute ago, GisMo said:

Hey Josh,

It's certainly faster. I'm no JS expert. Would you consider tabling that issue of increasing performance on Stackoverflow.com ? If so, let me know and I'll up vote your question. 

Neither am I. Yeah, I'll post that when I get a chance. Searching Stack Overflow is how I found the current javascript function in the first place.

Thanks, J

[Joshua Willing Halpern]

Hey guys, I've updated the sync check javascript to use a new algorithm with linear time complexity. The new version should handle larger/growing quantities of record UUIDs much better.

If you're using this javascript method, all you need to do is replace the Set Variable [$url] calculation with the calculation text in the new version.

@GisMo, @iOllivier, @dansmith65

 

New Thread: