Jump to content
Sign in to follow this  
Joshua Willing Halpern

Sync Check Speed Improvement - Part 2

Recommended Posts

Hey again. In a previous thread I suggested using javascript in a webviewer to return the difference between the server and client UUID lists in order to increase the sync check speed. It was much faster, but the code I used included an algorithm with quadratic time complexity, meaning that as the size of the lists increased the time that sync check took would grow exponentially. @GisMo helped test confirm that this solution would not scale well.

However I think I've found a bit of javascript that solves this issue and will scale linearly. I've changed the script from the previous thread to use this new code instead. If you are using this javascript method for list comparison I highly recommend you change the sync check script to use this code. Simply open the new file, copy the calculation from set variable[ $url ] in "Sync Check Client", and paste it into your Set Variable[ $url ] step.

Otherwise if you're using EasySync and struggling with sync check, give my solution a try!

I've attached the new file below and linked to the old thread for continuity.

Best, Josh

EDIT: For anyone interested in a demonstration of just the algorithms' performance difference before updating sync check, I've uploaded a raw concept file.

 

EasySync_JWH_SyncCheckMod_LINEAR.zip

Linear_Array_Comparison.fmp12.zip

Screen Shot 2016-05-20 at 2.40.53 AM.png

Edited by Joshua Willing Halpern
Added a picture.
  • Like 2

Share this post


Link to post
Share on other sites

Hi Joqshua

 

Thanks for the notification.

Will definitely give it a go, as my solutions are growing and any speed up at this point is very welcome.

Glad I am not the only one here who decided to go open-source.

 

J. how are you handling with new version deployment vs data set stored within the mobile part?

My mobile file with initial data is around 50MB now .. and seeing how goZync is handling the deployment is making me thinking about the future.

 

Thanks G.

Share this post


Link to post
Share on other sites

My pleasure. I hope it helps.

I agree that EasySync is a good open source project with strong bones. There is, of course, room for improvement but that is part of the fun and where we come in. :) 

As for deployment and version updating, I think you should start a new thread for that topic. But that is definitely a facet of EasySync I'm interested in discussing and working to improve.

Share this post


Link to post
Share on other sites

@Joshua Willing Halpern Thanks! I will give this a go! The previous jacascript solution is still working well for me and have not had any issues. That said, I'm always up for some performance boosts. I may have some questions when I try to implement this revision.

 

 

 

Share this post


Link to post
Share on other sites
26 minutes ago, GisMo said:

@Joshua Willing Halpern Thanks! I will give this a go! The previous jacascript solution is still working well for me and have not had any issues. That said, I'm always up for some performance boosts. I may have some questions when I try to implement this revision.

Glad to hear it. This version is exactly the same except for the actual $url set variable calculation. Should only take a minute to swap out the calculation and see if it's faster. You could just copy that step, and then replace the calc in one for a/b comparison. 

And I'm happy to try to answer any questions.

Share this post


Link to post
Share on other sites

I forgot to add just the javascript text for anyone who doesn't want to go digging in the attached fm file:

If you're already using the webviewer method, just copy this text into the Set Variable $url step 'Sync Check Client'.

JWH SyncCheck Mod 2.txt

  • Like 1

Share this post


Link to post
Share on other sites

Hi Joshua,

Apologie in advace for a newbie question. I am an accidental FM developer and newbie to FM.

I recently started using FMEasySync on a medium size database with 10 tables, 2 of them having container fields as well. Number of records are almsot 20000 and usually there are 200 - 300 record every 2 hours to sync. It takes around 10 mins to do this operation every 2 hours. 

my eyes lit up when i saw your post, I followed the steps as you mentioned, created a webviewer on payload layout on client, replaced Sync Check on Client and Server .

I am missing something here, any guidance would be highly appreciated.

 

Cheers

 

Share this post


Link to post
Share on other sites
4 hours ago, FM_Newbie said:

I am missing something here, any guidance would be highly appreciated.

 

Did you name the webviewer "WebViewer"? I may have forgotten to include that step in the instructions.

Also, did you copy the script 'Clear Script Result' into your script workspace?

Edited by Joshua Willing Halpern
Forgot something

Share this post


Link to post
Share on other sites

Ye, i did. Actually copied the same object from your layout . Also the script "Clear Script Result'' has been added on client as well.

 

just wondering if the variable 
$$use_psos_during_sync_check should be set to be on "Client" or "Server" it is currently set to "Server" in ping settings.

 

Share this post


Link to post
Share on other sites
8 hours ago, FM_Newbie said:

Ye, i did. Actually copied the same object from your layout . Also the script "Clear Script Result'' has been added on client as well.

 

just wondering if the variable 
$$use_psos_during_sync_check should be set to be on "Client" or "Server" it is currently set to "Server" in ping settings.

 

OK cool.

The variable $$use_psos_during_sync_check should be set to either 1 or 0. 1 = yes, perform script on server, 2 = no, perform script on client machine. Did you change any other settings in the settings script? Either is fine though performing the script on client can be very useful for using the debugger if you have filemaker pro advanced.

Can you describe the error/problem you're experiencing more? Where is EasySync failing? Is it getting stuck on the sync check or before that?

 

Also just to double check, did you copy the Sync_Check_Return_Diff script into your solution as well?

 

Edited by Joshua Willing Halpern

Share this post


Link to post
Share on other sites

Actually overall Sync is taking alot of time, i guess the bottleneck in my case is the internet speed at both sides, specially at the Host side.

EasySync is taking alot of time so i re-deployed it by taking identical copie of the solution and then as both copie update, i am trying to make sure sync is done every 2 hours or less.

Usually "Pushing Payload to the Server" takes alot of time, although there are not many changes on mobile side that are to be pushed. Mostly the update is on the Pull side. Occassionally it gives error of "Couldn't receive Payloa d Segements from customer."

To improve user experience, i have disabled Sync check as i am not using deletion support in the solution yet. Just trying to keep the steps to minimum to keep this working. Should i be using the Sync check anyway??

Share this post


Link to post
Share on other sites

Hey FM_Newbie. A 10 minute sync sounds like Sync Check is the culprit.

Whether you disable sync check is up to you. But if you do, any records you deleted from the server will have to be deleted manually or via a different script. Disabled, you'll need physical access to the client file/device to delete superfluous records or records which no longer "belong" to that device/user.

By using sync check, the administrator has control over what remains on the devices.

If you never/rarely need to delete device records, then go ahead and disable Sync Check. Otherwise, you'll need it. If you're able to get my mod working, sync check will not be a huge burden. On a database with 15 large tables and 30,000 records, sync check takes me ~7 seconds to collect all IDs from server and client and ~0.3 seconds to compare the lists (it used to take about 15 minutes).

It just depends on your needs. ;)

j

Edited by Joshua Willing Halpern
misspoke.

Share this post


Link to post
Share on other sites

Thanks Josh,

I am not currently doing deletion on Mobile side, plan to do it later and will use your version of Sync Check, it certainly takes lesser time.

Appreciate your help and guidance in this regard.

Cheers

Share this post


Link to post
Share on other sites

Thanks for posting this. It really helped.

I made few changes. First, I made the UUIDs from the host as part of the payload in the "pull payload" / "prepare payload for client" scripts which gets triggered by a flag in additional pull info. If the client doesn't receive the uuids then it continues as before. If the server doesn't receive the flag then it doesn't send the UUIDs. So things stay compatible both ways.

I also changed the script to work on windows (Internet Explorer 11), old iOS (version 8) and version 9. I found the very hard way that Explorer 11 doesn't fully support ECMAscript 6 which your new code uses and apparently iOS 8 Safari doesn't either. See the attached files.

Also the parameter length limitation in Internet Explorer 11. May be Filemaker on windows 10 is using Edge? I can't verify this now.

One more thing, I found that busy waiting for Script result to come back didn't work on windows. So instead, I halt the script and let the return script continue everything

If [Get(SystemPlatform) = -2] set variable [$url; Value: $windowsScript] Else set variable [$url; Value: $iOSscript]End if

the callback script goes like this

Set Variable [$diff; Value: Get(ScriptParameter)]

If [$diff="Clipboard"]

Paste [No style; EasySync_Payloads::SyncDiff]

Commit Records/Requests [No dialog]

Set Variable[$diff; Value: EasySync_Payloads::SyncDiff]

End If

iOS8Syncdiff.txt

windowsSyncdiff.txt

iOS9Syncdiff.txt

Edited by tmr_slh

Share this post


Link to post
Share on other sites
8 minutes ago, tmr_slh said:

I made few changes.

Hey, I'm going to take a look at this tonight; they all sound like great changes, thanks for taking the time to do this.

I believe reigning in the Sync Check speed was the biggest obstacle to EasySync being competitive with Gozync and Mirrorsync and cross-platform compatibility is paramount.

J

Share this post


Link to post
Share on other sites

Hey @tmr_slh, thanks again for your input.

1. Can you clarify for us OSX users what the IE parameter limit is? I see Todd Geist said it was 2048 characters concurrent with version 13.0v2. Is this still the case?

2. Also, do you know why the looping and waiting for script result doesn't work on windows?

3. Also, it appears you're accessing the clipboard in the windows version. I try to avoid this unless we warn the user first. Is there another way to store/pass on the list diff?

4. Could I see a copy of your file with the modified scripts?

Share this post


Link to post
Share on other sites
23 hours ago, Joshua Willing Halpern said:

Hey @tmr_slh, thanks again for your input.

1. Can you clarify for us OSX users what the IE parameter limit is? I see Todd Geist said it was 2048 characters concurrent with version 13.0v2. Is this still the case?

2. Also, do you know why the looping and waiting for script result doesn't work on windows?

3. Also, it appears you're accessing the clipboard in the windows version. I try to avoid this unless we warn the user first. Is there another way to store/pass on the list diff?

4. Could I see a copy of your file with the modified scripts?

I'll post the whole file soon.

1. The limit is not from FileMaker itself but from internet explorer. Total URL length around 2048 characters as you mentioned. If FileMaker is using Microsoft Edge instead in the newer version, things will be different but I don't know different in what way.

2. The only way to explain why or why not it works is multi threading. After spending a whole day trying to get it to work, my only conclusion is that FileMaker on Windows doesn't launch the web viewer in a separate thread while it does just that on Mac. Extremely annoying and I have no way of confirming this other than the fact that it doesn't work. However, may be you will agree that a callback that continues processing is better than loop-pause.

3. Not that I know of. The commercial products from seed code and Todd Geist are using the clipboard and I copied it from one of their blog posts.

Share this post


Link to post
Share on other sites
15 hours ago, tmr_slh said:

3. Not that I know of. The commercial products from seed code and Todd Geist are using the clipboard and I copied it from one of their blog posts.

In that case, perhaps we should wrap the previous clipboard contents, recopy and pass that along to the callback script as well. Then we could restore the clipboard after retrieving the UUID diff.

  • Like 1

Share this post


Link to post
Share on other sites
On 5/31/2016 at 5:25 PM, Joshua Willing Halpern said:

I forgot to add just the javascript text for anyone who doesn't want to go digging in the attached fm file:

If you're already using the webviewer method, just copy this text into the Set Variable $url step 'Sync Check Client'.

JWH SyncCheck Mod 2.txt

@Joshua Willing Halpern A+! I just implemented the new code. Significantly faster! Thank you for your awesome work!

Share this post


Link to post
Share on other sites
1 hour ago, GisMo said:

@Joshua Willing Halpern A+! I just implemented the new code. Significantly faster! Thank you for your awesome work!

Glad it's working for you! You may want to read tmr_slh's comment a couple entries up and verify that this modification works on all platforms you need it for. If not his suggestions look promising though I haven't tried them and I don't know that they all have the same linear complexity. Cheers.

Share this post


Link to post
Share on other sites
On 7/19/2016 at 1:09 AM, tmr_slh said:

I'll post the whole file soon.

@tmr_slh would love to see your version, tmr. We too ran into failure on Windows.

Share this post


Link to post
Share on other sites

How's the speed of EasySync compared to Mirrorsync ?

We like Mirrorsync but we had troubles recently and the fact that it's an external app makes it harder to debug .

We're looking at other products while we're waiting for a fix.

 

Share this post


Link to post
Share on other sites

I'd say Mirrorsync is the quickest sync tool I've come across. But Easysync can be a great solution--how many records will be in your sync payloads, and across how many tables?

Edited by Joshua Willing Halpern
word for clarity

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Sign in to follow this  

  • Similar Content

    • By Richard Carlton
      Testing FileMaker Network Performance Part 1
      Get up to speed with the FileMaker Video Training Course! 
      FileMaker is a cross-platform relational database application from FileMaker Inc. https://en.wikipedia.org/wiki/FileMaker
      Top Rated Course by FileMaker Expert, Richard Carlton.  
      http://learningfilemaker.com/fmpro17.php
      Customer relationship management, processes implemented to manage a company's interactions with customers and prospects
      https://en.wikipedia.org/wiki/CRM
      Experience Richard's dynamic and exciting teaching format, while learning both basic, intermediate, and advanced FileMaker development skills. With 28 years of FileMaker experience and a long time speaker at FileMaker's Developer Conference, Richard will teach you all the ins and outs of building FileMaker Solutions.  The course is 50 hours of video content!
      Richard has been involved with the FileMaker platform since 1990 and has grown RCC into one of the largest top tier FileMaker consultancies worldwide. Richard works closely with RCC's staff: a team of 28 FileMaker developers and supporting web designers. 
      He has offices in California, Nevada, and Texas.
      Richard has been a frequent speaker at the FileMaker Developers Conference on a variety of topics involving FileMaker for Startups and Entrepreneurs, and client-server integration.  
      Richard is the Product Manager for FM Starting Point, the popular and most downloaded free FileMaker CRM Starter Solution.
      Looking for FM Starting Point free software download: 
      http://www.fmstartingpoint.com
      Richard won 2015 Excellence Award from FileMaker Inc (Apple Inc) for outstanding video and product creation, leading to business development.
      RCC, FileMaker Videos, and LearningFileMaker.com are headquartered in Santa Clara, CA.
      http://www.rcconsulting.com/
      Contact us at support@rcconsulting.com
      FileMaker Pro is simply a powerful software used to create custom apps that work seamlessly across iPad, iPhone, Windows, Mac, and the web
      Transform your business with the FileMaker Platform.   Workforce Innovation Platform
      https://sites.google.com/site/filemakerprotrainingd2n/home/filemaker-training
      Free FileMaker Training Videos Channel https://www.youtube.com/user/FileMakerVideos
      FileMaker Training -FileMaker Video Training Course-FileMaker  News-FileMaker Experts
      https://www.youtube.com/watch?v=RO3Do8QIQOE
      https://www.youtube.com/watch?v=RO3Do8QIQOE&index=2&list=PLjTvUZtwtgBTWVT38kPvgjCfkZmMmvCOk&t=25s
      FileMaker What's New-Top Ten Things To Know About FileMaker -FileMaker  News
      https://youtu.be/6ZcnQEOF9XE
      https://www.youtube.com/watch?v=6ZcnQEOF9XE&index=1&list=PLjTvUZtwtgBTWVT38kPvgjCfkZmMmvCOk&t=25s
      Sharing a FileMaker Solution-The Basics-Sharing FileMaker Database-FileMaker Experts
      https://youtu.be/a5o5Mt8DnFI
      https://www.youtube.com/watch?v=a5o5Mt8DnFI&index=9&list=PLjTvUZtwtgBQukMafXx60CF5rcXwNmGWl&t=0s
      A database management system (DBMS) is a computer software application that interacts with the user, other applications, and the database itself to capture and analyze data
       
      Free FileMaker videos check out ...http://www.filemakervideos.com
      Download the FileMaker Pro & FileMaker GO for mobile devices training videos at http://www.learningfilemaker.com
      Download FileMaker Go video training at http://learningfilemaker.com/FMGO-17/index.php
      Download FileMaker Full Video Training Bundle at http://learningfilemaker.com/subscription.php
      Learn how to use FileMaker to create an app with the FileMaker Training Series
      Comment, Like & Share All of Our Videos.
      Feel Free to Embed any of Our Videos on Your Blog or Website.
      Follow Us on Your Favorite Social Media
      https://www.facebook.com/FileMakerVideos
      https://twitter.com/filemakervideos
      https://plus.google.com/+FileMakerVideos/videos
      #FileMakerNetworkPerformance
      #FileMakerNetworkLinkConditioner
      #FileMakerTesting
      #FileMakerFreeVideoTraining
      #FileMakerTrainingVideos
      #FilemakerProTraining
      #FileMakerVideoTutorial
      #FileMakerProVideos
    • By Richard Carlton
      FileMaker Coaches' Corner - Tip 11 - Improve Performance - FileMaker - FileMaker Experts
      https://youtu.be/5D2qoA_S7u8
      Get up to speed with the FileMaker Pro 17 Video Training Course! 
      Top Rated Course by FileMaker Expert, Richard Carlton.  
      http://learningfilemaker.com/fmpro16.php
      Experience Richard's dynamic and exciting teaching format, while learning both basic, intermediate, and advanced FileMaker development skills. With 27 years of FileMaker experience and a long time speaker at FileMaker's Developer Conference ,Richard will teach you all the ins and outs of building FileMaker Solutions.  The course is 50 hours of video content!
      Richard has been involved with the FileMaker platform since 1990 and has grown RCC into one of the largest top tier FileMaker consultancies worldwide. 
      Richard works closely with RCC's staff: a team of 28 FileMaker 
      developers and supporting web designers. He has offices in California, Nevada, and Texas.

      Richard has been a frequent speaker at the FileMaker Developers Conference on a variety of topics involving 
      FileMaker for Startups and Entrepreneurs, and client server integration.

      Richard is the Product Manager for FM Starting Point, the popular and most downloaded free FileMaker CRM Starter Solution.

      Richard won 2015 Excellence Award from FileMaker Inc (Apple Inc) for outstanding video and product creation, leading to business development.

      RCC and LearningFileMaker.com are headquartered in Santa Clara, CA.

      http://www.rcconsulting.com/
      Please feel free to contact us at support@rcconsulting.com
      If you want to explore building I.O.S apps for I Phone or I Pad and deploying those out to the Apple App Store.
       
      Here is a video introduction to our iOS App Training https://www.youtube.com/watch?v=cVxQe_yAshw
      Looking for FM Starting Point free software download: http://www.fmstartingpoint.com
      For More Free FileMaker Videos Check out Http://www.filemakerfree.com
      Visit http://www.learningfilemaker.com for all facets of FileMaker Award Winning Video Training.
      Please Visit Our Channel: https://www.youtube.com/user/FileMakerVideos Please Subscribe While There.

      Please Comment, Like & Share All of Our Videos.

      Feel Free to Embed any of Our Videos on Your Blog or Website.
      Follow Us on Your Favorite Social Media
      https://www.facebook.com/FileMakerVideos
      https://twitter.com/filemakervideos
      https://plus.google.com/+FileMakerVideos/videos

      Filemaker Pro 17 Training Videos
      FileMaker 17 Videos
      Filemaker Pro 17 Video Course
      #FileMakerVideos
      #FileMakerTrainingVideos
      #WhatisFilemaker17
      #FilemakerPro17Training
      #Filemaker17VideoTutorial
      #FilemakerPro17Videos
      #FileMakerCoaches'CornerTip11
    • By MSPJ
      So I have a hosted solution, with a dashboard with a couple portals. When I go to scroll a portal, scrolling even one portal screen has a delay of a few seconds, ie I click the scroll bar or try to drag it, and it takes about 3 seconds before it moves. Running locally, it's instantaneous. Running on a dev server on my LAN, there's a slight delay, but minimal.
      Having read various threads about this situation, my first thought was it's due to a filtered portal, so I removed filtering, but that made no difference.
      I also read various posts about unstored calcs and other factors that would cause all the data to have to be transferred to the client over the WAN.
      But here's the thing - at the moment, there are about 20 records in the database, related to the particular portal. 
      Clearly, moving that data can't be the primary issue - right? 
      I do have a lot of Execute SQL calculations in related tables, and I've read that can cause poor performance. But wouldn't that only be a factor when there are a lot of records?  
      Are there design /schema choices that would cause significant lag regardless of the amount of data? 
      Thanks,
      Michael
    • By lan
      Environment:
      FMS 15 Windows Server 2012 host on Azure Plug in: 360Works ScriptMaster Purpose for using the Plug in: zip the pdf files. Problems We have:
      There are a large amount temp files created on the client machine which run the fmp to connect the server , the temp files won't removed. What I am looking for:
      Any script step can avoid these temp files stay in temp folder. How these files created? I attached a screenshot.
       
      Thanks

       
    • By TJ53
      Let’s say we have two related tables: “Invoice” and “Invoice_Item”. We could create a calculation field in the “Invoice” table called “total_amount” with this formula:
      total_amount = Sum (Invoice_Item::amount)
      This field would have a negative impact in performance when appearing in the layout, since it would have to be defined as unstored, because it’s referencing a field from a related table.
      Now let’s suppose this field is not used for any scripts, tooltips, conditional format, etc … would the performance of the database be negatively affected ONLY when this field appeared in a layout? 
      In other words, would adding an unstored calculation field to a table involve a performance penalty, even in the “unreal” case where this field didn’t appear in any layout, script, conditional format, etc.? thanks in advance!
  • Who Viewed the Topic

    9 members have viewed this topic:
    H  melmitts707  Steve Martino  Lee Smith  doughemi  dpwilton  fmsavey  BruceR  Spidey 
×
×
  • Create New...

Important Information

By using this site, you agree to our Terms of Use.