Remote backup with Rsync fails

[huotg01]

I'm trying to automate my remote backup process by using rsync. The actual command is the first line below:

Admin@1080-Router:/jffs/scripts#[B] rsync -avz -e "ssh -p xxxx -i /jffs/dropbear/dropbear_rsa_host_key" /mnt/My_Book/Backup/ admin@xxxxx.asuscomm.com:/mnt/My_Book/Backup-1080[/B]
sending incremental file list
NUC-Utilisateur/NUC/Data/$OF/12246/11916 (2014_11_13 23_54_41 UTC).jpg
[B]ssh: Couldn't set SO_PRIORITY (Bad file descriptor)[/B]

ssh: Connection to admin@xxxxx.asuscomm.com:1220 exited: Remote closed the connection
rsync: writefd_unbuffered failed to write 4 bytes to socket [sender]: Broken pipe (32)
rsync: connection unexpectedly closed (297 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(605) [sender=3.0.9]

At the terminal prompt it starts, begins the transfer (there is messages on the terminal) and suddenly just "suspend" its progression. So far I was under the impression that there was no errors reported, but I think that I was not patient enough...I let the command going on all night and this time I have the above message on the terminal. Does that means that I have a problem with one specific file, and this stops the whole process? Could someone help me to debug this ?

Thanks

 

[huotg01]

For debugging purposes I decided to make the command simpler by copying on the same router USB disk (ssh and the remote router are not involved):

rsync -avz /mnt/My_Book/Backup/ /mnt/My_Book/Backup-Doris

Here is the result:

[...hundreds of lines...] 
NUC-Utilisateur/NUC/Data/$OF/112197/112257 (2014_11_13 23_54_41 UTC).htm
NUC-Utilisateur/NUC/Data/$OF/112197/112258 (2014_11_13 23_54_41 UTC).htm
NUC-Utilisateur/NUC/Data/$OF/112197/112259 (2014_11_13 23_54_41 UTC).png
NUC-Utilisateur/NUC/Data/$OF/112197/112260 (2014_11_13 23_54_41 UTC)
NUC-Utilisateur/NUC/Data/$OF/112197/112261 (2014_11_13 23_54_41 UTC)
NUC-Utilisateur/NUC/Data/$OF/112197/112262 (2014_11_13 23_54_41 UTC)
NUC-Utilisateur/NUC/Data/$OF/112197/112263 (2014_11_13 23_54_41 UTC)
NUC-Utilisateur/NUC/Data/$OF/112197/22834 (2014_11_13 23_54_41 UTC).gif
NUC-Utilisateur/NUC/Data/$OF/112197/230 (2014_11_13 23_54_41 UTC).css
NUC-Utilisateur/NUC/Data/$OF/112197/35970 (2014_11_13 23_54_41 UTC).js
ERROR: out of memory in map_ptr [sender]
rsync error: error allocating core memory buffers (code 22) at util.c(117) [sender=3.0.9]
rsync: writefd_unbuffered failed to write 78 bytes to socket [generator]: Broken pipe (32)

Would it be possible that the RT-AC68u is not powerfull enough to handle the process ?

 

[huotg01]

...and:

NUC-Utilisateur/NUC/Data/$OF/113476/113480 (2014_11_13 23_54_41 UTC).manifest
NUC-Utilisateur/NUC/Data/$OF/113567/113480 (2014_11_13 23_54_41 UTC).manifest
NUC-Utilisateur/NUC/Data/$OF/11460/11461 (2014_11_13 23_54_41 UTC).htm
NUC-Utilisateur/NUC/Data/$OF/11475/11478 (2014_11_13 23_54_41 UTC).doc
NUC-Utilisateur/NUC/Data/$OF/11475/11479 (2014_11_13 23_54_41 UTC).pdf
NUC-Utilisateur/NUC/Data/$OF/11475/11712 (2014_11_13 23_54_41 UTC).htm
ERROR: out of memory in flist_expand [sender]
rsync error: error allocating core memory buffers (code 22) at util.c(117) [sender=3.0.9]
rsync: writefd_unbuffered failed to write 78 bytes to socket [generator]: Broken pipe (32)

 

[RMerlin]

Try skipping the trailing slash on the source path. I remember running into issues on a customer's server while trying to rsync a directory with too many files, I can't remember however if the issue was because of a trailing slash or a trailing wildcard.

 

[nairn62]

Hi,

I had a similar problem copying data from a Linux ext4 file system (on my RT-AC68U) to NTFS disk, using rsync 3.0.9-1 (ipkg) and the following rsync command:

rsync -rltDvu --modify-window=10 --progress --delete-after --exclude=Incomplete /mnt/SAMSUNG/Transmission /mnt/MYCLONE/TRANSMISSION

In the end, I had to recompile rsync 3.1.1 and the new code stopped it running out of memory! The rsync ipkg (bluering package) is 3 years old and way out of date IMHO.

 

[huotg01]

In the end, I had to recompile rsync 3.1.1 and the new code stopped it running out of memory! The rsync ipkg (bluering package) is 3 years old and way out of date IMHO.

How do we re-compile Rsync for the Asus?
Thanks. Is it something you can provide me?

 

[huotg01]

Try skipping the trailing slash on the source path. I remember running into issues on a customer's server while trying to rsync a directory with too many files, I can't remember however if the issue was because of a trailing slash or a trailing wildcard.

The trailing slash has to be there in order to have the backup done the way I need it. What I could maybe do is split the backup in smaller pieces (but just after trying a recompiled Rsync, if possible).

 

[nairn62]

Placing the following 'make code' into the Makefile for the router type you are building is all that you need to do:

rsync-3.1.1/stamp-h1:
@test -d rsync-3.1.1 || wget -O - http://rsync.samba.org/ftp/rsync/rsync-3.1.1.tar.gz|gzip -d|tar xvf -
cd rsync-3.1.1 && \
$(CONFIGURE) --disable-debug --disable-iconv --disable-iconv-open
touch $@

rsync-3.1.1: rsync-3.1.1/stamp-h1
@$(MAKE) -C rsync-3.1.1 $(PARALLEL_BUILD)

rsync-3.1.1-install: rsync-3.1.1
@$(SEP)
install -D rsync-3.1.1/rsync $(INSTALLDIR)/rsync-3.1.1/usr/bin/rsync
$(STRIP) -s $(INSTALLDIR)/rsync-3.1.1/usr/bin/rsync
chmod 0500 $(INSTALLDIR)/rsync-3.1.1/usr/bin/rsync

rsync-3.1.1-clean:
-@$(MAKE) -C rsync-3.1.1 clean
@rm -f rsync-3.1.1/stamp-h1

 

[huotg01]

Placing the following 'make code' into the Makefile for the router type you are building is all that you need to do:

This is very kind of you, but...

At first I thought that I could use the Entware Rsync. Entware Rsync is at 3.1.1. No luck: Entware is not for my router: RT-AC68U (and my partner router is a RT-AC56u, same CPU...).

Therefore I have to go back at the re-compile solution, but I would need just a little bit more information. Could you add another layer of info to those already sent ?

Thanks.

 

[huotg01]

For the record (http://rsync.samba.org/FAQ.html#5):

out of memory

The usual reason for "out of memory" when running rsync is that you are transferring a _very_ large number of files. The size of the files doesn't matter, only the total number of files. If memory is a problem, first try to use the incremental recursion mode: upgrade both sides to rsync 3.0.0 or newer and avoid options that disable incremental recursion (e.g., use --delete-delay instead of --delete-after). If this is not possible, you can break the rsync run into smaller chunks operating on individual subdirectories using --relative and/or exclude rules.

The solution is 3.1.1...

 

[RMerlin]

http://www.mediafire.com/download/ffxocc5vg9e2o9d/rsync_arm.zip

This was a quick five minutes build, so I haven't tested it beyond making sure it actually ran. Let me know how it goes.

 

[huotg01]

This was a quick five minutes build, so I haven't tested it beyond making sure it actually ran. Let me know how it goes.

Do I just replace the actual Rsync by the one in the zip file ?

Thanks.

 

[RMerlin]

Do I just replace the actual Rsync by the one in the zip file ?

Thanks.

Copy it wherever you want, and make sure your script uses the correct path. You can put it in /jffs for example, so it will always be available. Make sure you do specify the fully qualified path to it in your scripts.

You can also simply uninstall the Optware version to avoid any confusion.

 

[huotg01]

Copy it wherever you want, and make sure your script uses the correct path. You can put it in /jffs for example, so it will always be available. Make sure you do specify the fully qualified path to it in your scripts.

You can also simply uninstall the Optware version to avoid any confusion.

NUC-Utilisateur/NUC/Data/$OF/113298/52772 (2014_11_13 23_54_41 UTC).css
NUC-Utilisateur/NUC/Data/$OF/113298/52844 (2014_11_13 23_54_41 UTC).js
NUC-Utilisateur/NUC/Data/$OF/113298/58675 (2014_11_13 23_54_41 UTC).js
NUC-Utilisateur/NUC/Data/$OF/113298/69793 (2014_11_13 23_54_41 UTC).gif
ERROR: out of memory in flist_expand [sender]
rsync error: error allocating core memory buffers (code 22) at util2.c(102) [sender=3.1.1]
ERROR: out of memory in flist_expand [receiver]
rsync error: error allocating core memory buffers (code 22) at util2.c(102) [receiver=3.1.1]
Admin@1080-Router:/tmp/home/root# rsync -avz /mnt/My_Book/Backup/ /mnt/My_Book/Backup-Doris
Admin@1080-Router:/tmp/home/root# rsync --version
rsync  version 3.1.1  protocol version 31
Copyright (C) 1996-2014 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
    64-bit files, 64-bit inums, 32-bit timestamps, 64-bit long ints,
    no socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
    append, no ACLs, no xattrs, iconv, symtimes, prealloc

It take some time but finally gave an error. Having the last version, I will now try a few other scenarios.

 

[RMerlin]

My guess is, you simply have too many files to sync, for the router's limited amount of memory. You might have to ressort to creating a swap on your disk. Possibly using a lower swapiness value to ensure that the swap only gets used when truly needed, so not to swap out actual services.

 

[noric]

How many files are you syncing, to have an idea?
Thanks.

 

[huotg01]

My guess is, you simply have too many files to sync, for the router's limited amount of memory. You might have to ressort to creating a swap on your disk. Possibly using a lower swapiness value to ensure that the swap only gets used when truly needed, so not to swap out actual services.

I have to admit that I don't understand very well the way Rsync works (and I'm not talking about the algorithm (that parts is ok ;-)).

I'm talking about the way Rsync interacts with the receiver.

If I do a Rsync from A to B on the same server, does one process handle "both" sides: sender and receiver ? Two Rsync processes, one acting as sender and one acting as a receiver ?

If I do a Rsync from A to [C on a remote server], is there just the Rsync sender working on both sides (like "cp"), or the Rsync sender is talking to a Rsync receiver on the remote, each of them doing its work ? If it is the latter, how does the remote Rsync know that it has to interact with the sender ? Do we have to start a daemon process (I use the word but don't even know what it means...)?

If a "working files list" has to be maintained during the backup process on both sides (sender and receiver), then doing a backup with Rsync on the same server would put more "stress" on the memory of that server (even more if it is a router with not so much memory...).

If Rsync 1 sender is doing is job of sender on server 1 and Rsync 2 is doing is job of receiver on server 2, then less memory would be used on each of them ?

Anyways, I know that this out of the scope of this forum, but if someone knows about these questions, it would help a lot those who wants to use Rsync on a Asus router.

Thanks.

 

[huotg01]

[...] You might have to ressort to creating a swap on your disk. Possibly using a lower swapiness value to ensure that the swap only gets used when truly needed, so not to swap out actual services.

• Is there a difference in performance between a swap file and a swap partition ?
• Does it make a difference to have the swap space either on the usb 2 flash drive either on the usb 3 hard drive ? If the latter is better, am I right to think that a ntfs partition can coexist with a ext2-3-4 partition ?
• For the swap space, does it make a difference to format it ext 2, 3 or 4?

Thanks

 

[RMerlin]

• Is there a difference in performance between a swap file and a swap partition ?
• Does it make a difference to have the swap space either on the usb 2 flash drive either on the usb 3 hard drive ? If the latter is better, am I right to think that a ntfs partition can coexist with a ext2-3-4 partition ?
• For the swap space, does it make a difference to format it ext 2, 3 or 4?

Thanks

Swap partition should in theory provide better performance, but it's more work to get set up, and performance isn't really what you are after here, as the swap would probably only be used during syncs. For your specific usage scenario, I'd go with a swap file.

Definitely go with the HDD - much faster performance, and less prone to wearing out as swaps can generate a fair amount of writes.

A swap partition is formatted as swap, not as a filesystem. If you are going with a swap file, it shouldn't really matter with version of ext you chose.

 

[huotg01]

Anyways, I know that this out of the scope of this forum, but if someone knows about these questions, it would help a lot those who wants to use Rsync on a Asus router.

For those interested by the questions asked above, I found this web page with a lot a informations on the mechanic of rsync:
http://rsync.samba.org/how-rsync-works.html

Still unclear for me the question of daemon ou not daemon, but other than that it is an excellent document.