Sync Check Error: sync check's call to the server has failed

[GisMo]

I'm having some difficulty tracking down this issue. The whole setup was working correctly, then I seeded the mobile database with the hosted database data(exact duplicate of the data). Then I made a change to one record and deleted another using the easy sync delete function and tested the sync.. Now I received the error message: "The sync check's call to the server has failed". I've enabled Sync Check debug and disable PSOS for sync check, but I am still having trouble getting this to work correctly. I can't actually debug/step through the Sync Check script in the Hosted_Database and it seems to run endlessly on the Sync Check script when running locally.

 

At the Perform Script On Server: Sync Check from Hosted_Database the error code is 513: Record must have a value in some field to be created OR No validation was specified but data cannot fit into the field - There errors don't make sense in this context. There are no fields being accessed/written in the script, only SQL and variables.

 

The value count of $UUIDs is 213,485  - from what gather, this is EVERY es_uuid on the local machine. correct? 

 

Debugging/Data Viewer is painfully slow..I think due to the size of $UUIDs variable.

 

Any ideas?

 

Edit:

Does Device ID have any affect on Sync Check?

 

Thanks

[GisMo]

An Update: I decided to disable PSOS for Sync Check in the settings and decided to be patient instead of quitting the process:

 7 Hours 22 minutes! ..Is there an alternative to this or a way to get PSOS to work? Is Sync Check only used for deleting records? Maybe there's a way to do it without having to do the Diff?

[timdietrich]

GisMo --

 

Based on your earlier post, it sounds like you have a large number of records on the mobile device. 

 

When EasySync reaches the Sync Check phase of the sync process, the UUIDs of all of the records (in all of the tables being synced) that are currently stored on the mobile device are gathered into a list and sent to the server. The server evaluates that list, checking each UUID to determine whether or not a particular record should still be on the mobile device. It does this because it is possible that a user has a record on their device that has been deleted by another user. Another possibility is that the business rules have changed in such a way that a record that the user was previously allowed to access should no longer be available to them (such as a customer account being reassigned to a different sales rep, a project or task that has been reassigned, etc).

 

In any case, when the server has completed its part of the Sync Check, it sends back a list of the UUIDs that the mobile device should no longer have. The mobile device then deletes those records.

 

The Sync Check is an important, although admittedly inefficient part of the overall process. And when a large number of records are being synced, the inefficiencies become much more apparent. And I think that's what you're seeing.

 

I think the problem that you are running into is being caused by the large number of records that you are attempting to sync. It could very well be that you absolutely need all 200k+ records to be available on the mobile device. If that's the case, EasySync might not be a good solution for you. However, if you do not need all of those records, and can instead work with just a subset of them, then I think you'll see much faster and more reliable performance when syncing.

 

Good luck with your project.

 

-- Tim

[GisMo]

Thanks for the reply. Yea it's A LOT, but the users NEED the whole database. I've seeded the mobile databases with the data so the initial sync was nothing..Normal syncing is working ok. it's about 100 records or so when updated on a daily basis. I would like to find an alternative solution for deletion sync...

 

Maybe we can create another table which logs the ES UUID and ES table which is maintained on the both the local and remote databases. Then upon a sync, we can send the deletion table to the server and find and remove those records. Then If another client syncs, they pull down the deletion table and the client will locally delete those records. Something a little more light weight. On the local table we flag the record as deletedLocally=1. I think it needs more thought, but maybe doing a DIFF on this table versus the entire database might be faster?

[timdietrich]

Perhaps some of the records are in an archived state, and as a result, unlikely to change.

 

If so, you could exclude those from the Sync Check entirely - and that might improve things.

 

-- Tim

[GisMo]

What's the best way to mark those records? On the host? On the client? Both?

[GisMo]

I've tested by setting certain records based on date to ES_Exclude = 1. I did this in both the local and hosted file. This reduced the $UUIDs variable to about 38000 records. It's a lot smaller. BUT, I still get the same error with PSOS for sync check. Error 513. 

 

I did some more digging: PSOS has a parameter size limitation to 1,000,000 characters. My smaller $UUIDs set is 1.4million characters. Is is possible to break up the $UUIDs into 20k chunks and run the diff check check for each set?

 

Also, I just tried to share the $UUIDs via a global field in the hosted file. I needed to created a table occurrence from the Hosted_Databased in the local database and then I can write $UUIDs to the global field to be used in the Sync Check on the server. This seems to work.

[GisMo]

I'm still looking for an alternative or viable way to sync deletions with my "large" dataset.

[GisMo]

I did some more digging: PSOS has a parameter size limitation to 1,000,000 characters. My smaller $UUIDs set is 1.4million characters. Is is possible to break up the $UUIDs into 20k chunks and run the diff check check for each set?

 

Also, I just tried to share the $UUIDs via a global field in the hosted file. I needed to created a table occurrence from the Hosted_Databased in the local database and then I can write $UUIDs to the global field to be used in the Sync Check on the server. This seems to work.

 

Does anyone else care to try this out? I'm open to discussion for the first method as I think there are holes in it, and while the second method seems to work, is this a "bad" idea?

[m_fitzgerald]

I did some more digging: PSOS has a parameter size limitation to 1,000,000 characters. My smaller $UUIDs set is 1.4million characters. Is is possible to break up the $UUIDs into 20k chunks and run the diff check check for each set?​

I'm getting 513 errors too. The length of my $UUIDs string is 2,343,143. Where did you find the PSOS parameter size limit? It's not in Tech Specs for v13?

Tim, 

Assuming that there is a limit of 1MB for a parameter, that creates an upper limit of 27027 records that can be handled using the current code base. 

In my situation I have 60,000 records in the table but only one or two will need to be pushed to / pulled from the server. I am guessing that the reason that I'm getting such such large sets is that I haven't done the initial config properly. Otherwise, the mod timestamp would be used to isolate the few records that need to be passed, correct?

 

[m_fitzgerald]

I've modified the Sync Check script on the Mobile side. I've set up variables that will split $UUIDs into chunks and loop through them. The script on the server side is unchanged.  

This is the code that is changed. It's not the entire script, just the significant bit that has changes.

 

 

#Refresh sync status.

Set Variable [ $$EasySync_Status; Value:"Performing sync check with server..." ] 

Refresh Window
#
#If the UUIDs exceed max size.

Absolute max for a parameter value is 1MB. Suggested max is 0.5MB or slightly more than 13500 UUIDs

Set Variable [ $slice_size; Value:13500 ]
Set Variable [ $UUID_valuecount; Value:ValueCount ( $UUIDs ) ] 

Set Variable [ $n; Value:Ceiling ( $UUID_valuecount / $slice_size ) ] 

Set Variable [ $i; Value:0 ]
#
Loop

Exit Loop If [ Let($i = $i+1; $i > $n) ]

Set Variable [ $slice_positions; Value:Let( [ $slice_start = 1 + ( $i * $slice_size ) - $slice_size ; $slice_end = Min ($UUID_valuecount; $i * $slice_size )] ; "" ) ]

Set Variable [ $UUID_slice; Value:MiddleValues ( $UUIDs ; $slice_start ; $slice_end ) ] #Send list to Server for analysis. Server will return the diff.

If [ $$use_psos_during_sync_check ]

#*** Remote Script: Sync Check

Perform Script on Server [ “Sync Check” from file: “Hosted_Database”; Parameter: $$additional_sync_check_info & ¶ & $UUID_slice ]

[ Wait for completion ]

Set Variable [ $error; Value:Get ( LastError ) ]

Else

#*** Remote Script: Sync Check

Perform Script [ “Sync Check” from file: “Hosted_Database”; Parameter: $$additional_sync_check_info & ¶ & $UUID_slice ]

Set Variable [ $error; Value:Get ( LastError ) ] 

End If

#

#If there was an error... 

If [ $error ]

Show Custom Dialog [ Title: "Sync Check Error"; Message: "The sync check's call to the server has failed."; Default Button: “OK”, Commit: “Yes” ]

Exit Script [ ] 

End If

#
#Get the DIFF list.
Set Variable [ $diff; Value:List ( $diff ; Get ( ScriptResult ) ) ]

End Loop

Close File [ “Hosted_Database” ]
#

[iOllivier]

On 5/8/2015 at 1:05 AM, m_fitzgerald said:

I've modified the Sync Check script on the Mobile side. I've set up variables that will split $UUIDs into chunks and loop through them. The script on the server side is unchanged.  

This is the code that is changed. It's not the entire script, just the significant bit that has changes.

 

 

#Refresh sync status.

Set Variable [ $$EasySync_Status; Value:"Performing sync check with server..." ] 

Refresh Window
#
#If the UUIDs exceed max size.

Absolute max for a parameter value is 1MB. Suggested max is 0.5MB or slightly more than 13500 UUIDs

Set Variable [ $slice_size; Value:13500 ]
Set Variable [ $UUID_valuecount; Value:ValueCount ( $UUIDs ) ] 

.......

Thanks m_fitzgerald for the quick script amendment. It works great with my large data set.

For those who are also using Joshua Willing Halpern Java script solution for quick UUIDs diff check in a webviewer to speed up comparing process.

You need to ignore this part:

#
#Get the DIFF list.
Set Variable [ $diff; Value:List ( $diff ; Get ( ScriptResult ) ) ]

End Loop

Close File [ “Hosted_Database” ]
#

 

and add only this bit after Josh's JavaScript magic part:

#------------------------------------------------------- m_fitzgerald's part2 ----------------------------------------------------------

End Loop

Close File [ “Hosted_Database” ]

 

#------------------------------------------------------- m_fitzgerald's end part2 ----------------------------------------------------------