Server to Server Syncing

[kim.m2]

I’ve just finished setting up a server-to-server sync involving several files (most work in a separated model, although some are not, and they all have references to each other).  I'm using the XML sync option and I don't need all the data to sync to the spoke server, so I have implemented constraining script steps to isolate just the data that does need to sync.  Overall, The Hub server needs to house all the data and be the master source, the spoke is set up in another part of the country to offer better performance to local users who need access to some of the data.

I ran the first sync over the weekend and have since found 3 issues:

First Issue: How do I safely update files on the Spoke Server?

I ran into a concern with the ID field on one table and once I straightened it out decided to replace that particular data file on the spoke server with a clone (which had been created in the middle of last week).  I was rather surprised to find that running the sync after replacing the file on the spoke caused ALL the syncing data to be deleted from my hub server, instead of re-syncing from the Hub to the Spoke as I expected (yikes!).  Fortunately, with our separated model, only a few tables were affected; and with my configurations only some of the records were set up to sync at all; and I happened to have replaced the clone within a few minutes of our last backup, so I was able to recover all the affected records.

While this instance resolved ok, this highlights a significant issue that I need to understand better:

In imagining how we plan to support changes on the second server, my intent is that if we need to update files, that we could at any time replace the files on the spoke server with clones and re-run the sync.  (About half the files we are syncing are built in a separated model, but several are not, so the changes in the clone copy may be related to layouts and scripts instead of new tables or fields – so I wouldn’t expect to need to rebuild the mirror sync configuration).

However, I need to trust that in doing so I’m not risking data being deleted from the hub.  Does anyone have a recommendation on how to do this safely/correctly?  

Second Issue: Modifying the Set of Syncing records?

This one is less of an actual issue, and more about confirming that the behavior I'm seeing.  I'm controlling the records that sync based on a field selection: if a project record has the syncing checkbox enabled, all the records related to that project are included in the found set that syncs to the spoke server.  If I uncheck the sync field on a project record from the hub server, I noticed that the records are automatically removed from the spoke server.  Pretty cool.  This is intentional, right? (as opposed to the records just staying on the spoke server as stale data).  

And - just to be very sure, if a the sync were unchecked from the project record on the spoke server, that won't cause those records to delete from the hub server, correct?  (I need the Hub to be a source of truth to house all the data, and not delete if we turn a project sync off or update files on the spoke server).

Third Issue: Fields that did not sync

I haven't thoroughly checked all tables yet(as I have 82 syncing tables altogether), but on at least one table I noticed that while much of the data on the record did sync, a particular number field did not sync.  (This is not an primary key field for the table, but it is important to other relationships).  The field is included on the layout in the file that has all the mirror sync set up.  Any suggestions on how I can fix this (or further troubleshoot why the field didn't sync when other fields on the same record did)?

 

[joshua-rivers-digital]

Hello Kim,

 

23 minutes ago, kim.m2 said:

First Issue: How do I safely update files on the Spoke Server?

If your sync is setup for Bi-Directional and at any point after syncing all the syncing data from the spoke server is missing (by being replaced by an empty clone) then yes..MirrorSync will look at this as that data was deleted from the spoke and should delete from the Hub.  I would recommend using 360Deploy 2.  This will allow you to make development changes to the spoke server and then deploy those changes to the server, then it imports all the data within those files into the newly deployed solution(s) resulting in updating the spoke server with the development changes but keeping all the production data.  If you go this route I would recommend you stop syncing during this process and re-enable any auto syncs after deploying.

28 minutes ago, kim.m2 said:

Second Issue: Modifying the Set of Syncing records?

This is expected behavior.  Basically what is happening is by unchecking the syncing checkbox that record, and if setup correctly related data will then not be included in the found set during the sync...which results in the spoke not being able to "see" that data anymore and therefore deletes from the spoke.  

I will need to do some testing on the second part with unchecking the box on the spoke side before giving you a definite answer.

32 minutes ago, kim.m2 said:

Third Issue: Fields that did not sync

If you send us the logs from a day where you noticed this behavior we can get a better sense of what may be going on.  You can do this by going to your MirrorSync launch page and clicking the Send problem report and log files link.  Filling out the form and then sending it our way.

In the meantime we typically see this when fields don't pass field validation, have correct privileges, field cannot be modified etc.  Those are some things you can check pretty easily.

 

Let us know if you have any other questions and I'll get back to you on the customizations.  Thank you!

 

Joshua Rivers

360Works Support

[kim.m2]

Thanks for the feedback Josh.

Regarding my first issue, updating files on the spoke server:  I'm not prepared for exploring formal data migration tools at this time.  We aren't running FileMaker 17 yet, and initially I'd like to start up our second server without it in order to evaluate the need (depending on how often we need updates and the use of the system from the new server).  If this goes really well and sees a lot of traffic, it may be a next step.  For now, how do I force a Sync to start over?  (Is there a setting somewhere, or would I need to rename the configuration....any other suggestions?)

 

 

Regarding my second issue, regarding the found set of the spoke server causing file deletions on the hub server, I did a some testing on my system: I unchecked the test project record on the spoke server and then ran the sync and the records were not deleted on the Hub server, and in fact I could continue editing records and those edited records continued to sync.  In this case my project records are not part of a mirror sync (just a one-way push to the spoke set up manually).  I also tested with a different type of record where the table is both part of the mirror sync configuration and has a the field that enables or disables syncing.  In this case I created the record on the spoke, confirmed it synced to the hub server, and tested changing the fields that controlled filtering.  Ultimately, the record synced to the hub and deleted from the spoke, but remained in the hub, unless explicitly deleted from the spoke by the user (or an empty clone file).  In my current situation, this is exactly what I wanted.  It suggests to me that the filtering of records to sync or not sync happens only on the hub server; all records & changes sync to the hub always, and the hub controls what records add, stay, or delete on the spoke.  Please let me know if your testing and understanding of 360 Works can confirm my conclusions. 

 

 

Regarding the fields that did not sync, what information is contained in the logs?  Schema is clearly involved, but is the data included as well?  If it is, can we schedule a call/screenshare to discuss this further directly?  (I can't send out the contents of our databases).

If it helps, the field in question is a number field with a calculation to enforce number of digits. The account mirror sync uses has filemaker's default [Data Entry Only] permissions, with no field customization applied.  There is some data validation applied, only during data entry, to accept the numeric only data type.  (While the calculation enforcing a number of digit may add leading 0s to the number, my particular example included numbers 101-110 , so the values should not have failed validation. 

Please let me know if there is more you can suggest.

 

Thanks for your help.

Kim

 

Quick follow up: I checked a smaller (easier to peruse) log from my recent testing.  This does include our database field data, so I can't send you the logs.  Could we schedule a call instead?  If it helpful, I can screenshare to research anything we need to check, but I can't send out the data itself.

Thanks for understanding.

[joshua-rivers-digital]

Hey Kim,

 

I have replied to your open ticket in our support system.  Please follow up there.  Thank you!

 

Regards,

Joshua
360Works Support