bi-directional sync with NAS: doable?

[m4rk0]

Hello,

I need to find a solution capable of bi-directional syncing.
I was thinking to buy a NAS or two, until I found out that there is no such a feature because of the impossibility to track changes to files which could be potentially changed in two different locations at the same exact time: conflicts. Am I right?

So here is the scenario, maybe someone has some ideas.

In the location A, there is a network of PCs connected through a router and a 4G internet connection.
These computers will access & process data on one of them, let's call it the main workstation. Data could be stored on a NAS locally as well. By process, I mean open, rename and/or delete pictures, mostly. The data must be accessible through the local network as if it were stored on a DAS, nothing to be copied to the computer themselves (no local sync).
In the location B, there is a computer (MAC or PC), maybe with a local NAS, which needs to access the same data located on the workstation (or on the NAS), either by having a local copy or by opening the files as if it were on a local hard disk (as opposite to the location A, both these features will be used on different times).

Random solutions (or not):

Originally, I thought to buy two QNAPs, one to install in the location A and one to install in the location B. Unfortunately, as I wrote above, it seems there is no way to have a bi-directional syncing between the two. A workaround could be two folders on the two QNAPs syncing mono-directionally but that isn't really practical.

Synology offers a feature called Cloud Station which could work but I am afraid will be too slow. That's what I was reading somewhere.

Reading in online forums, I found these two tiny softwares: Syncthing and Resilio Sync. The latter can be installed on a NAS as well, but I am not sure how well they work.

What do you think guys?
Whichever solution I use, it shall be as straight forward as possible.

Please let me know your thoughts

Thanks!

 

[RMerlin]

QNAP's RTRR is two-way between two NAS.

 

[HeMaN]

I use SyncThing on my Thecus NAS and pc's

 

[m4rk0]

Gents, thanks for your answers. Took me a while to get back to you because I ordered some hardware to do my testing.

QNAP's RTRR is two-way between two NAS.

Great, I'll keep it in mind.

I am having some speed issues trying to setup a TS-253a: its download speed over the local network is ~20MB.
Do you have a clue why?

Here's my test setup:
MBP 2016 connected via 5Ghz to an ac88u (1300Mbps stable link)
TS-253a w/ Samsung EVO 850 connected via link aggregation to the Asus router.

Why am I getting just 20MB/s when transferring files over wifi to the NAS? Is wifi really that bad in terms of net transfers or I am missing something? By the way tomorrow I'll get the ethernet adapter (damn you Apple) so I'll find it out soon.

I use SyncThing on my Thecus NAS and pc's

Thanks, I'll do some testing soon.

 

[L&LD]

Yes, WiFi can be that bad. Have you tried testing for a better channel for your specific equipment?

 

[sfx2000]

Here's my test setup:
MBP 2016 connected via 5Ghz to an ac88u (1300Mbps stable link)
TS-253a w/ Samsung EVO 850 connected via link aggregation to the Asus router.

Why am I getting just 20MB/s when transferring files over wifi to the NAS? Is wifi really that bad in terms of net transfers or I am missing something? By the way tomorrow I'll get the ethernet adapter (damn you Apple) so I'll find it out soon.

Should be a bit faster, but rsync can be very chatty, and WiFi is half-duplex compared to ethernet...

Just a bit curious about that MBP2016 - since that's all USB-C stuff, so please follow thru on this thread.... and which GBE adapter you might have picked up

(I have both MBP2012 (non-Retina, so it has GBE onboard) and an MBA2014 with Thunderbolt - and the TB-GBE adapter there is pretty fast, e.g. same as onboard ethernet on the MBP2012)

 

[m4rk0]

Yes, WiFi can be that bad. Have you tried testing for a better channel for your specific equipment?

Nope, I haven't. I am aware it's wise to try out other channels and find the one which works best for the environment but I have such a limited time. I'll try to trouble shoot it when I can. Thanks for the hint tho.

Should be a bit faster, but rsync can be very chatty, and WiFi is half-duplex compared to ethernet...

Just a bit curious about that MBP2016 - since that's all USB-C stuff, so please follow thru on this thread.... and which GBE adapter you might have picked up

(I have both MBP2012 (non-Retina, so it has GBE onboard) and an MBA2014 with Thunderbolt - and the TB-GBE adapter there is pretty fast, e.g. same as onboard ethernet on the MBP2012)

And here's the thing. I wasn't really using rsync.
I did that test transferring a few GBs from my MBP over wifi to the TS-253a to find out which speed I would get. I am getting 1/8th of the link speed over wifi, it's depressing.

I have another TS-253a configured with a 850 EVO which will be used to simulate locally the working environment where they will be located. But I am not too sure they are the best solution in terms of efficiency (there is a quite a bit of lag when working over a local LAN with several files). On the other hand, I tried quickly SyncThing and I have been pleasantly surprised.

I need to do so much more testing tho.

And yes, I'll report back as soon as I get the USB-C to Ethernet thing.

Thanks both of you

 

[m4rk0]

Hello guys.

Sorry I have been missing for a while but I got super busy with work and unable to put my hands on for some proper tests.

Anyhow.

@sfx2000
So far, it's been impossible to get a hold of an Apple USB-C to ethernet adapter: out of stock in London. I think I found one in Italy so I might just get it while I am here.
Also, it looks like I can virtually pair two ethernet interfaces aggregating them. Have you ever heard anything about it? Not necessary at the moment, just curious for now.

@everyone
I had a look at this Qnap page which talks about RTRR sync.
Can anybody answer these questions please?

1. How efficient is QTS handling changes to files? Let's suppose both NASes sync'ed a folder with 1000 files. In the local NAS, all of the files are renamed in a batch (in Windows, select all files and then hit F2 to rename progressively). Are the changes being applied 'instantly' to the remote NAS or does it take a while before the system realises which files have changed because of some sort of calculation?
2. Question number one leads directly to question number two - if only the name of a file (or more files) has changed, is the change reflected in the remote NAS as file name change or does the system consider the file(s) as new file(s) thus requiring transferring over the whole file(s)?
3. Scenario: local and remote NAS configured identically with two drives: SSDs in the slots #1 and mechanical HDDs in the slots #2; each drive has a shared folder synced remotely on the other twin NAS. A (sync'ed) folder is being moved from SSD to HDD. Does the remote NAS replicate the action or does it require transferring files all over uploading them as a 'new' through internet connection? I think the latter happens.

I'll need to do my own tests once I get back home.
However, it would be amazing if someone, who has experience in RTRR, could answer these questions.

Thank you.

 

[sfx2000]

So far, it's been impossible to get a hold of an Apple USB-C to ethernet adapter: out of stock in London. I think I found one in Italy so I might just get it while I am here.
Also, it looks like I can virtually pair two ethernet interfaces aggregating them. Have you ever heard anything about it? Not necessary at the moment, just curious for now.

Try looking for a USB-C to Thunderbolt 2 adapter (should be able to get this direct from Apple - then you have many options (even 10GB ethernet if so desired)

Yes, one can bind multiple interfaces, but it's usually not worth the trouble - you'd also need a switch that support LAG as well - can help with capacity, but the speed is the same (as this is how ethernet works, two lanes, but each ethernet has to be on a single lane, and same with the ack for that frame)

 

[m4rk0]

Try looking for a USB-C to Thunderbolt 2 adapter (should be able to get this direct from Apple - then you have many options (even 10GB ethernet if so desired)

Yes, one can bind multiple interfaces, but it's usually not worth the trouble - you'd also need a switch that support LAG as well - can help with capacity, but the speed is the same (as this is how ethernet works, two lanes, but each ethernet has to be on a single lane, and same with the ack for that frame)

Yup, but those are expensive adapters which I don't need at the moment. Anyhow, for the sake of information: Akitio adapter 10Gbe bundled w/ 3T3.

Do you have any clue regarding those RTRR questions?

 

[sfx2000]

Can't say much about RTRR... don't use it myself.

rsync and snapshots is what I use...

 

[m4rk0]

Can't say much about RTRR... don't use it myself.

rsync and snapshots is what I use...

Okay. I'll do my own testing at some point.
Thanks!

 

[RMerlin]

I only used RTRR in scheduled mode, not in realtime-mode, sorry.

 

[m4rk0]

Gents, I have been able to do some testing: what a disappointment.

QTS 4.2.4 & 4.3.3

Basically, RTRR (real time remote replication, as they like to call it) is not really real time.
Oh well, it is real time while being scheduled to happen, so to speak: this 2-ways sync can be done only on a schedule, be it every now and X minutes/hours/days, on a specified time, etc. It does NOT really happen in real time.
In fact, the new Hybrid Backup Station app doesn't state anything being real time when selecting 2-way sync, something which they were saying on the previous backup app.
On the other hand, one way sync seems to be real time but I haven't done much testing just because I do need it to be two ways.

The best compromise I found is in the app Resilio, which can be downloaded for free on various OSes, including Qnap's. It is somewhat slow, I am guessing it calculates files' hash comparing both directions. CPU usage is pretty intense for a while and even simple actions happens after severals minutes. But they happen.

Thanks for your feedbacks, I appreciate your help.

@sfx2000
At the moment I doubt I'll get a type C to ethernet adapter but I'll keep you posted as soon as I do.

 

[sfx2000]

@sfx2000
At the moment I doubt I'll get a type C to ethernet adapter but I'll keep you posted as soon as I do.

Sounds good - the Apple thunderbolt Gbe adapter is pretty good, all told... as good as their USB2 to Fast Ethernet adapter - just works...