huotg01
Senior Member
About 500 000 files in 310 000 folders for ~1.5 Tb
Remote backup with Rsync fails
- Thread starter huotg01
- Start date
About 500 000 files in 310 000 folders for ~1.5 Tb
From what I understand, rsync can work in two different ways:
1) with a daemon. This is how I have it setup at home, where I run an rsync daemon (DelteCopy) on my desktop, and a cron job on the HTPC runs an rsync client on the HTPC every hour, syncing the music between the two. The client talks to the daemon, and the two decide together what to transfer, using their own protocol. The client read/writes on the HTPC, and the daemon reads/writes on the desktop.
2) standalone client. rsync then reads the content on both ends through a user-specific access method (SMB share, ssh tunnel, NFS share, etc...). rsync then has to read everything on both ends, make the decisions, and handle copying/deleting files on both ends.
1) with a daemon. This is how I have it setup at home, where I run an rsync daemon (DelteCopy) on my desktop, and a cron job on the HTPC runs an rsync client on the HTPC every hour, syncing the music between the two. The client talks to the daemon, and the two decide together what to transfer, using their own protocol. The client read/writes on the HTPC, and the daemon reads/writes on the desktop.
2) standalone client. rsync then reads the content on both ends through a user-specific access method (SMB share, ssh tunnel, NFS share, etc...). rsync then has to read everything on both ends, make the decisions, and handle copying/deleting files on both ends.
That's a LOT of files, especially for a router with less than 256 MB of RAM.
You might want subdivising the sync job in folder groups to reduce the load.
huotg01
Senior Member
Ok then I will swap on the usb 3 disk. This disk is formatted NTFS and, unless forced, I had no intention to change that.
I suppose that I cannot create a swap file on my NTFS partition, can I ?
If I can't, then I will have to create a small extX partition on the USB 3 disk ?. If so, at that point, better to create a "swap partition" ? How (if not too complex)?
Thanks
No idea if swapfiles can be used on an NTFS filesystem, you will have to try.
huotg01
Senior Member
...it is not a bed of roses!
Ok my usb3 disk has now 2 partitions:
and the /dev/sdc2 is now a swap space (the swappiness is set to 10):
I should be in position to check is Rsync will now behave the way it should...
Ok my usb3 disk has now 2 partitions:
- NTFS (backup from my windows machines)
- ext (for the 1Gb swap): /dev/sdc2
Code:
Disk /dev/sdc: 4000.7 GB, 4000752599040 bytes
255 heads, 63 sectors/track, 60799 cylinders
Units = cylinders of 16065 * 4096 = 65802240 bytes
Device Boot Start End Blocks Id System
/dev/sdc1 1 60784 3905928192 7 HPFS/NTFS
/dev/sdc2 60784 60800 1048576 83 Linux
Code:
Admin@1080-Router:/tmp/home/root# [B]mkswap [/B]/dev/sdc2
Setting up swapspace version 1, size = 1073737728 bytes
UUID=a5cb6b25-ab75-40a7-b6ea-5c755df5c135
Admin@1080-Router:/tmp/home/root# [B]swapon [/B]/dev/sdc2
Admin@1080-Router:/tmp/home/root# [B]free[/B]
total used free shared buffers
Mem: 255760 56964 198796 0 416
-/+ buffers: 56548 199212
Swap: 1048572 0 1048572
Admin@1080-Router:/tmp/home/root# [B]echo 10 > /proc/sys/vm/swappiness[/B]huotg01
Senior Member
I'm now investigating this error:
If someone has an idea (other than splitting the backup, for now...), you are very welcome.
Code:
Admin@1080-Router:/jffs/scripts# rsync -avz -e "ssh -p xxxx -i /jffs/dropbear/dropbear_rsa_host_key" /mnt/My_Book/Backup/ admin@xxxx.asuscomm.com:/mnt/My_Book/Backup-1080
sending incremental file list
NUC-Utilisateur/NUC/Data/$OF/12246/11916 (2014_11_13 23_54_41 UTC).jpg
rsync: [sender] write error: Broken pipe (32)
ERROR: out of memory in flist_expand [generator]
rsync error: error allocating core memory buffers (code 22) at util2.c(102) [generator=3.1.1]Monitor your memory usage while rsync is running, you'll see what's going on there, now that there's a 1 GB swap up.
Also confirm the swap setup is actually working with the "free -m" command.
huotg01
Senior Member
These are some random samplings of the actual run, a test made with a simpler configuration on the same server (in fact on the same disk...), but with the real source. No problem so far:
Code:
Admin@1080-Router:/tmp/home/root# free
total used free shared buffers
Mem: 255760 205416 50344 0 244
-/+ buffers: 205172 50588
Swap: 1048572 116 1048456
Admin@1080-Router:/tmp/home/root# free
total used free shared buffers
Mem: 255760 228248 27512 0 536
-/+ buffers: 227712 28048
Swap: 1048572 13880 1034692
Admin@1080-Router:/tmp/home/root# free
total used free shared buffers
Mem: 255760 227212 28548 0 656
-/+ buffers: 226556 29204
Swap: 1048572 11240 1037332
Admin@1080-Router:/tmp/home/root# free
total used free shared buffers
Mem: 255760 227196 28564 0 644
-/+ buffers: 226552 29208
Swap: 1048572 8216 1040356
Admin@1080-Router:/tmp/home/root# free
total used free shared buffers
Mem: 255760 231620 24140 0 500
-/+ buffers: 231120 24640
Swap: 1048572 54852 993720The problem reported earlier was with the real remote backup. Even if the error message doesn't say so, it could be caused by the remote router not having yet a swap space ? I will do it now.
Thanks.
huotg01
Senior Member
@RMerlin: yes it is possible to create a swap file on a NTFS partition. Here is what I did on the remote site:
Now one side is on a swap partition and the other side (remote) on a swap file (to test both).
I just re-launched the real backup, the one going to the remote site, and everything seems to go well. Even more interesting, just for fun, at the same time I let the test backup running on the source router. So there is now 2 huge backups running at the same time (4 rsync processes on the source) and everything goes well. So far so good. I will keep my fingers crossed...
Next step is to make the swap spaces "permanent"(survive to a boot). I know that there is a lot of threads on that matter but comments are welcome.
Code:
admin@RT-AC56U:/tmp/home/root# dd if=/dev/zero of=/mnt/My_Book/myswapfile bs=1M count=1024
1024+0 records in
1024+0 records out
admin@RT-AC56U:/tmp/home/root# [B]chmod 600[/B] /mnt/My_Book/myswapfile
admin@RT-AC56U:/tmp/home/root# [B]mkswap [/B]/mnt/My_Book/myswapfile
Setting up swapspace version 1, size = 1073737728 bytes
UUID=0ea6c42d-ce7b-48dd-8c7d-c517f0531f87
admin@RT-AC56U:/tmp/home/root# [B]swapon [/B]/mnt/My_Book/myswapfile
admin@RT-AC56U:/tmp/home/root# [B]free[/B]
total used free shared buffers
Mem: 255776 85644 170132 0 768
-/+ buffers: 84876 170900
Swap: 1048572 0 1048572I just re-launched the real backup, the one going to the remote site, and everything seems to go well. Even more interesting, just for fun, at the same time I let the test backup running on the source router. So there is now 2 huge backups running at the same time (4 rsync processes on the source) and everything goes well. So far so good. I will keep my fingers crossed...
Next step is to make the swap spaces "permanent"(survive to a boot). I know that there is a lot of threads on that matter but comments are welcome.
huotg01
Senior Member
Code:
TTT-Admin@1080-Router:/tmp/home/root# free
total used free shared buffers
Mem: 255760 225508 30252 0 1032
-/+ buffers: 224476 31284
Swap: 1048572 107460 941112It's quite a good machine (RT-AC68U), and by the way, it still acts as a router. I'm impressed.
Attachments
Last edited:
I probably go with a post-mount user script, so it can be executed after the disk is mounted by the router.
huotg01
Senior Member
Thanks. That part works well now: the swap is always back after a boot, with the right happiness,.
That said, none of the commands in services-start seems to work. As an example, I have a script to copy the rsa private key and the known-hosts file from jffs back to ~/.ssh and it doesn't work, neither in post-mount neither in services-start. When the script is run with a command line on the terminal there is no problem. Is there a way to debug these situations involving the "special scripts" ?
Thanks
What I usually do is, right after the shebang, I insert a touch /tmp/000script-name. That way, if I see an empty 000script-name appear in /tmp, I know the script did get executed.
huotg01
Senior Member
Sorry for that...I should have read the wiki before asking...
I used the touch and the logger commands and I can confirm that EDIT: SERVICES-START RUNS.
Here is the "services-start" script:
Code:
#!/bin/sh
touch /tmp/services-start.txt
logfile_path="/mnt/My_Book/Backup-logs/services-start.log"
logger logfile_path="/mnt/My_Book/Backup-logs/services-start.log"
# write some stuff to the log file
echo " " >> $logfile_path
echo " " >> $logfile_path
echo "***** new log entry services-start *****" >> $logfile_path
echo "$action_1" >> $logfile_path
date >> $logfile_path
#cédule le backup externe sur le disque de Doris
logger cru a backuptodoris "30 23 * * * /jffs/scripts/backuptodoris.sh"
cru a backuptodoris "30 23 * * * /jffs/scripts/backuptodoris.sh" >> $logfile_path
#les diverses clés SSH sont recopiées à chaque boot vers /root/.ssh car cet espace est "volatil" sur le routeur ASUS
logger /jffs/scripts/copySSHkey.sh
/jffs/scripts/copySSHkey.sh
#fait un backup de la partition permanente jffs
logger /jffs/scripts/backupJFFS.sh
/jffs/scripts/backupJFFS.sh
exit $?
Last edited:
huotg01
Senior Member
In fact, the one and only one proof I have that services-start runned is the trace let by the touch command in the /tmp folder.
Other than that there is no trace in the syslog (logger is used in the script), cru -l is empty, no files are copied and there is no trace either on the specific log files. Does that mean someting (other that I'm confused...)?
Other than that there is no trace in the syslog (logger is used in the script), cru -l is empty, no files are copied and there is no trace either on the specific log files. Does that mean someting (other that I'm confused...)?
USB disks aren't mounted yet when services-start runs. That step only happens later on during boot time.
huotg01
Senior Member
Thanks RMerlin. I think I have now a better understanding. I was seeing the whole thing as happening as a time schedule just at boot time, or at a shotdown, in an orderly manner. As it is said the scripts are triggered by events, and the scripts are runned whenever these events happen, and these events can happen any time during the router uptime. As an example is it ok to say that if on day two I unmount a disk/partition, and then mount the same disk or another one, the unmount and the post-mount scripts will be runned ? By the way I was also using , as you suggested, the post-mount, but not a good understanding of what was goint on. I will make the post-mount script check if the event is related to the mounting of the disk I need, and, if yes, execute the copies and other things that have to be done. Thanks again for your patience.
GH
ps: other than that the rsync remote backup has been running all night without any problem...
huotg01
Senior Member
Thanks @joegreat for the logging tips (http://forums.smallnetbuilder.com/showthread.php?p=62259) I have now more information, which brings more questions...
Here we see that there is 3 mounts, and 2 on AsusflashDrive, the usb2 stick formated in ext2:
Here is the syslog:
...and here is the result of a fdisk -l:
The post-mount sript is:
Here is the result of a mount command:
In the post-mount specific log I have 3 runs:
Thanks
Here we see that there is 3 mounts, and 2 on AsusflashDrive, the usb2 stick formated in ext2:
Code:
START_post-mount started [/tmp/mnt/AsusflashDrive]
Postmount execution OK. Script: /jffs/scripts/post-mount
START_post-mount started [/tmp/mnt/My_Book]
Postmount execution OK. Script: /jffs/scripts/post-mount
START_post-mount started [/tmp/mnt/AsusflashDrive]
Postmount execution OK. Script: /jffs/scripts/post-mount
Code:
Dec 14 11:16:15 rc_service: ntp 1415:notify_rc restart_upnp
Dec 14 11:16:15 rc_service: ntp 1415:notify_rc restart_diskmon
Dec 14 11:16:15 rc_service: waitting "restart_upnp" via ntp ...
Dec 14 11:16:15 miniupnpd[1276]: shutting down MiniUPnPd
Dec 14 11:16:15 miniupnpd[1439]: HTTP listening on port 44633
Dec 14 11:16:15 miniupnpd[1439]: Listening for NAT-PMP/PCP traffic on port 5351
Dec 14 11:16:17 disk monitor: be idle
[B]Dec 14 11:16:18 START_post-mount: started [/tmp/mnt/My_Book][/B]
Dec 14 11:16:19 kernel: Adding 1048572k swap on /dev/sdc2. Priority:-1 extents:1 across:1048572k
[B]Dec 14 11:16:19 STOP_post-mount: return code 0[/B]
Dec 14 11:16:19 rc_service: hotplug 1205:notify_rc restart_nasapps
Dec 14 11:16:19 iTunes: daemon is stopped
Dec 14 11:16:19 FTP Server: daemon is stopped
Dec 14 11:16:20 Samba Server: smb daemon is stopped
Dec 14 11:16:20 kernel: gro disabled
Dec 14 11:16:21 Timemachine: daemon is stopped
Dec 14 11:16:21 kernel: gro enabled with interval 2
Dec 14 11:16:21 dnsmasq[1208]: nameserver 70.38.99.32 refused to do a recursive query
Dec 14 11:16:21 Samba Server: daemon is started
[B]Dec 14 11:16:22 hotplug[1223]: USB ext2 fs at /dev/sda5 mounted on /tmp/mnt/AsusflashDrive
Dec 14 11:16:22 kernel: EXT2-fs (sda5): warning: mounting unchecked fs, running e2fsck is [/B]recommended
[B]Dec 14 11:16:23 hotplug[1223]: USB partition unmounted from /tmp/mnt/AsusflashDrive[/B]
[B]Dec 14 11:16:41 hotplug[1223]: USB ext2 fs at /dev/sda5 mounted on /tmp/mnt/AsusflashDrive[/B]
[B]Dec 14 11:16:44 START_post-mount: started [/tmp/mnt/AsusflashDrive]
Dec 14 11:16:44 STOP_post-mount: return code 0[/B]
Dec 14 11:16:44 rc_service: hotplug 1223:notify_rc restart_nasapps
Dec 14 11:16:44 iTunes: daemon is stopped
Dec 14 11:16:44 FTP Server: daemon is stopped
Dec 14 11:16:45 Samba Server: smb daemon is stopped
Dec 14 11:16:45 kernel: gro disabled
Dec 14 11:16:46 Timemachine: daemon is stopped
Dec 14 11:16:46 kernel: gro enabled with interval 2
Dec 14 11:16:47 Samba Server: daemon is started
Dec 14 11:17:09 crond[821]: time disparity of 2078896 minutes detected
Code:
Disk /dev/mtdblock0: 0 MB, 524288 bytes
255 heads, 63 sectors/track, 0 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/mtdblock0 doesn't contain a valid partition table
Disk /dev/mtdblock1: 1 MB, 1572864 bytes
255 heads, 63 sectors/track, 0 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/mtdblock1 doesn't contain a valid partition table
Disk /dev/mtdblock2: 31 MB, 31457280 bytes
255 heads, 63 sectors/track, 3 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/mtdblock2 doesn't contain a valid partition table
Disk /dev/mtdblock3: 29 MB, 29783552 bytes
255 heads, 63 sectors/track, 3 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/mtdblock3 doesn't contain a valid partition table
Disk /dev/mtdblock4: 99 MB, 99352576 bytes
255 heads, 63 sectors/track, 12 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/mtdblock4 doesn't contain a valid partition table
Disk /dev/mtdblock5: 1 MB, 1310720 bytes
255 heads, 63 sectors/track, 0 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/mtdblock5 doesn't contain a valid partition table
Disk /dev/sda: 2021 MB, 2021654528 bytes
228 heads, 47 sectors/track, 368 cylinders
Units = cylinders of 10716 * 512 = 5486592 bytes
Device Boot Start End Blocks Id System
/dev/sda1 1 368 1966111+ f Win95 Ext'd (LBA)
Partition 1 has different physical/logical beginnings (non-Linux?):
phys=(0, 31, 33) logical=(0, 42, 12)
Partition 1 has different physical/logical endings:
phys=(244, 227, 47) logical=(367, 30, 26)
/dev/sda5 1 368 1966080 83 Linux
Note: sector size is 4096 (not 512)
Disk /dev/sdc: 4000.7 GB, 4000752599040 bytes
255 heads, 63 sectors/track, 60799 cylinders
Units = cylinders of 16065 * 4096 = 65802240 bytes
Device Boot Start End Blocks Id System
/dev/sdc1 1 60784 3905928192 7 HPFS/NTFS
/dev/sdc2 60784 60800 1048576 83 Linux
Code:
#!/bin/sh
#based on http://forums.smallnetbuilder.com/showthread.php?p=62259 from @joegreat
touch /tmp/postmount.txt
/bin/sleep 2s # give the disk x sec to really be ready
/usr/bin/logger -t START_$(basename $0) "started [$@]" # Log the start of the script int syslog.log
SCRLOG=/tmp/$(basename $0).log # define the own log file for this script
touch $SCRLOG # create my own log file for the messages of this script
echo "START_$(basename $0) started [$@]" >> $SCRLOG # log the start message
if [ $1 = "/tmp/mnt/My_Book" ] # check if My_Book drive is mounted
then
#Active le swap space
swapon UUID=36151d7f-56a0-4eef-b835-de7f5daf94de
#les diverses clés SSH sont recopiées à chaque boot vers /root/.ssh car cet espace est "volatil" sur le routeur ASUS
/jffs/scripts/copySSHkey.sh
#fait un backup de la partition permanente jffs
/jfss/scripts/backupJFSS.sh
#Fixe le swappiness qui est le paramètre qui détermine le degré d'agressivité du processus de swap. Peut varier de 0 à 100. À 0 le routeur ne va swapper que si manque total de mémoire. Par défaut: 60
echo 20 > /proc/sys/vm/swappiness
wait # wait for the script to finish
fi
if [ "$?" -ne 0 ] # If error in the execution of the script
then
echo "Error in postmount execution! Script: $0" >> $SCRLOG # Log the error
else
echo "Postmount execution OK. Script: $0" >> $SCRLOG # Log the success
fi
/usr/bin/logger -t STOP_$(basename $0) "return code $?" # Log that the script has finished into syslog.log
exit $?
Code:
TTT-Admin@1080-Router:/tmp/home/root# mount
rootfs on / type rootfs (rw)
/dev/root on / type squashfs (ro,relatime)
devtmpfs on /dev type devtmpfs (rw,relatime,size=127772k,nr_inodes=31943,mode=755)
proc on /proc type proc (rw,relatime)
tmpfs on /tmp type tmpfs (rw,relatime)
sysfs on /sys type sysfs (rw,relatime)
devpts on /dev/pts type devpts (rw,relatime,mode=600)
/dev/mtdblock4 on /jffs type jffs2 (rw,noatime)
usbfs on /proc/bus/usb type usbfs (rw,relatime)
/dev/sdc1 on /tmp/mnt/My_Book type tntfs (rw,nodev,relatime,uid=0,gid=0,umask=00,nls=utf8,min_prealloc_size=64k,max_prealloc_size=3905928188,readahead=1M,user_xattr,case_sensitive,reset_journal,nocache,fail_safe,hidden=show,dotfile=show,errors=continue,mft_zone_multiplier=1)
/dev/sda5 on /tmp/mnt/AsusflashDrive type ext2 (rw,nodev,relatime,errors=continue,user_xattr)- 1 for My_book
- 2 for AsusflashDrive (why 2?)
Thanks
Last edited:
Similar threads
Similar threads
-
Connection to router by ip fails while remote over OpenVPN
- Started by cone
- Replies: 17
-
-
Isolating home security cameras and providing remote access?
- Started by MadPup
- Replies: 5
-
-
Backup and Restore Procedures for Diversion and SkyNet
- Started by dragon84
- Replies: 3
-
Script to detect backup wan and disable certain devices from internet access
- Started by RandomUser777
- Replies: 0
-