syncthing ac68u

[voipair]

It works, i didn't see this posted
ac68u
asuswrt-merlin 378.55
This is what i did
install debian
I don't think this is required but i do have it installed so if you find it is needed entware
I followed the steps to install syncthing on this page into chroot debian substituting the url for the current arm download link on the syncthing website - Not sure what the difference is but i also installed from inside debian to /tmp, the instructions say to use /opt as dir.
Add iptables rule

 nano /jffs/scripts/firewall-start
iptables -I INPUT -p tcp --destination-port 8384 -j ACCEPT
/jffs/scripts/firewall-start

edit syncthing listening host

debian enter
cd $HOME/.config/syncthing
nano config.xml

Change this line from 127.0.0.1 to your hostname, this worked on mine

<address>RT-AC68U:8384</address>

I installed in temp instead of the opt dir in the linked post so modify the path below to install dir and syncthing version

cd /tmp/syncthing/syncthing-linux-arm-v0.11.24
./syncthing

access via browser on computer on lan https://rt-ac68u:8384/
first couple of tests seem to be working,
Data on platter drive connected to usb3 port syncing from router to 1 client (dell venue 8 pro win 10 lan wifi 5g),
1) 10k files 3g ~2 hours to sync
2) 32 files 10g ~30mins to index (i let router fully index before connecting client)

45 minutes into second sync 4.8 gb remaining
speed is steady just above 2 MiB/s
syncthing reports ram usage at 73.8 MiB
run on router

top - 01:52:17 up 17:16,  0 users,  load average: 1.44, 1.45, 1.48
Tasks:  93 total,  1 running,  92 sleeping,  0 stopped,  0 zombie
%Cpu(s): 53.6 us,  3.0 sy,  0.0 ni, 33.0 id,  3.8 wa,  0.0 hi,  6.6 si,  0.0 st
KiB Mem:  255732 total,  219948 used,  35784 free,  5440 buffers
KiB Swap:  1572860 total,  3404 used,  1569456 free.  83656 cached Mem

sync completed in about 2.5 hours

todo
syncthing shouldn't run as root
startup script/ auto start - this might be a useful page
run in background
confirm syncing works from wan

 

[kvic]

If it can work over WAN (does it?), one good use scenario is for online offsite backup.

Two-way/multi-way sync will inevitably give rise to conflicts. For backup purpose, we can assume the source (master) always override the destination (slave). Does it allow setting up such a relationship between, say, two devices?

 

[martinr]

Following your links to discover more about Syncthing, which I hadn't heard of, I read on their FAQ page

"Is Syncthing my ideal backup application?
No, Syncthing is not a backup application because all changes to your files (modification, deletion, etc) will be propagated to all your devices. You can enable versioning, but we encourage the use of other tools to keep your data safe from your (or our) mistakes."

If it's not recommended as a backup application, is its usefulness not severely limited?

 

[kvic]

@martinr I can't comprehend the rationale they putting up such a disclaimer! lol

 

[martinr]

@martinr I can't comprehend the rationale they putting up such a disclaimer! lol

Thanks. I thought I was being dense. And if all the sync'ing devices were located under one roof I can see risks of having all one's eggs in one basket.

 

[john9527]

"Is Syncthing my ideal backup application?
No, Syncthing is not a backup application because all changes to your files (modification, deletion, etc) will be propagated to all your devices

This is the same reasoning as to why a RAID solution isn't a backup solution. If you make an 'oops', the 'oops' is propagated...you no longer have a backup to recover from.

 

[martinr]

This is the same reasoning as to why a RAID solution isn't a backup solution. If you make an 'oops', the 'oops' is propagated...you no longer have a backup to recover from.

Thanks. And I'd be sure to make an 'oops' at some stage. (And there's also susceptibility to malware that encrypts every sync'ing device.). But I guess with a backup & sync system such as, for example, Dropbox, with versioning, you could at least revert to a pre-oops version. I hope it doesn't look as if I'm trying to knock Syncthing: I warm to their philosophy and I think the clarity of the website is something their open-source community should be really proud of. I also think it decent of volpair to publish his info.

 

[kvic]

@martinr I like this success story. Of cos I also appreciate @voipair's post for introducing the community to this interesting open source project...better yet it can somewhat run on asuswrt-merlin.

@john9527 I'm actually running RAID1 on my desktop for... backup...Personally I consider RAID1 part of a bigger backup strategy. lol

 

[voipair]

you are correct syncthing is not designed for backups, check out crashplan on the router for backup.

 

[voipair]

----edit 09242015 ignore this entry it didn't do anything-----

So trying to figure out autostart
syncthing help on auto starting
If i try to start it outside of debian then it goes through and starts generating new ssh keys so don't want that

This is what i have tried on the router

cp /opt/etc/init.d/S99debian /opt/etc/init.d/S98syncthing

edit the file so it reads

#!/bin/sh

ENABLED=yes
PROCS=syncthing
ARGS="/opt/debian/tmp/syncthing/syncthing-linux-arm-v0.11.24/syncthing"
PREARGS=""
DESC=$PROCS
PATH=/opt/sbin:/opt/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

. /opt/etc/init.d/rc.func

and reboot router, but that didn't work, i think i need to look at the sytemctl option on the syncthing help page, but it seems like we might already have some sort of means to do this in the debain install, since systemctl isn't installed on my chroot debian i don't want to install additional pkg if not required.

I am not sure what is the best way to integrate this foss into my debian install so it always runs and starts on reboot as a service. Anyone have any suggestions ?

 

[kvic]

Your first post mentioned you need to run synching inside a chrooted debian (is my understanding right?). So the script as-is in #10 won't work.

You need a custom script in place of "PROCS=synctihng". In this custom script, you fork a child process and then run a script which will first chroot to debian and then start synching from there. I think that's the right direction to try.

 

[voipair]

From this page
put the debian script where they say to, i didn't move my syncthing file to usr/local as is suggested and changed the link in the script, the two lines i changed are below

 debian enter
nano /etc/init.d/syncthing
#these are the two lines i changed
syncthing_USERS="root"
DAEMON=/tmp/syncthing/syncthing-linux-arm-v0.11.24/syncthing

i also had to chmod the file

 chmod +x  /etc/init.d/syncthing

now from within debian i can start it as a service
Then added it to the services list page not sure why but this page said it would be a great idea

exit
nano /opt/etc/chroot-services.list
#this is the only text in the file, it is a new file for my system
syncthing

Now when debian is started syncthing is started.

I now have a new problem that the directory i am mounting isn't getting mounted inside debian at boot. I have to restart debian after the router has booted in order for the external directory to mount. I think it is the same issue as is fixed here. Also synthing doesn't actually start when debian is started at boot time. it only starts when debian is started after the router boots, not sure why
There is already a /jffs/scripts/post-mount file on my system, i tried adding the bottom lines to it but it didn't seem to work on reboot (ie external directory were not mounted in chroot debian when started from router booting). here is my whole file
i tried debian, debian start, debian restart as the second to last line in the file, none worked

#!/bin/sh
if [ $1 = "/tmp/mnt/sda1" ]
then
  ln -nsf $1/entware.arm /tmp/opt
fi
swapon /opt/swap

#Following is what i added to the file
mountpoint=/tmp/mnt/My_Book/
# the disk you use for media server

#dlna_config=/tmp/mnt/DISK1/Scripts/minidlna.conf
# the configuration file created for minidlna to use the custom settings.


if [ $1 = $mountpoint ]
then
  debian start
fi

any thoughts?

this is the ouput from restarting debian, no idea why syncthing has two ii

debian restart
Stopping chroot'ed Debian services...
umount: can't forcibly umount /tmp/mnt/sda1/entware.arm/debian/dev/pts: No such file or directory
Restarting chroot'ed Debian services...
Starting syncthiing for root
Done.

 

[kvic]

Also synthing doesn't actually start when debian is started at boot time. it only starts when debian is started after the router boots, not sure why

Congrats. I believe you worked out 99% of the problem.

The remaining issue is answered in #11 already. You need to spawn off a child process and then "debian start" (if it's right way of starting chrooted debian or else the equivalent chrooting and starting debian..)

One way you can spawn a child is to use a package from Entware, called Daemonize..

 

[voipair]

Congrats. I believe you worked out 99% of the problem.

Thanks very much.

The remaining issue is answered in #11 already.

I finally understand that debian services ARE NOT starting at boot on my system (like you said). You could very well have the perfect answer. but i haven't found much online about it that i can copy and modify to use here. could be i just don't understand.

another road leads....
From this page seems like the services-start script is the best one to use to start services at boot, but when adding commands to mine still couldn't get debian to start at boot.
Then i notice the first line in the file is

#!/bin/sh
script="/opt/etc/init.d/rc.unslung"

So i go look at /opt/etc/init.d/rc.unslung

#!/bin/sh

# Start/stop all init scripts in /opt/etc/init.d including symlinks
# starting them in numerical order and
# stopping them in reverse numerical order

#logger "Started $0${*:+ $*}."

ACTION=$1
CALLER=$2

if [ $# -lt 1 ]; then
  printf "Usage: $0 {start|stop|restart|reconfigure|check|kill}\n" >&2
  exit 1
fi

[ $ACTION = stop -o $ACTION = restart -o $ACTION = kill ] && ORDER="-r"

for i in $(/opt/bin/find /opt/etc/init.d/ -perm '-u+x' -name 'S*' | sort $ORDER ) ;
do
  case "$i" in
  S* | *.sh )
  # Source shell script for speed.
  trap "" INT QUIT TSTP EXIT
  #set $1
  #echo "trying $i" >> /tmp/rc.log
  . $i $ACTION $CALLER
  ;;
  *)
  # No sh extension, so fork subprocess.
  $i $ACTION $CALLER
  ;;
  esac
done

Well that looks perfect, already programmed to do what i want.

ls -l /opt/etc/init.d/
-rwxr-xr-x  1 admin  root  215 Aug 12 06:24 S80lighttpd
-rwxr-xr-x  1 admin  root  246 Sep 19 10:29 S98syncthing
-rwxr-xr-x  1 admin  root  2572 Sep  5 22:12 S99debian
-rw-r--r--  1 admin  root  2786 Aug  1 07:01 rc.func
-rwxr-xr-x  1 admin  root  841 Aug  1 07:01 rc.unslung

so it is already trying to start these files, but for some reason is not. I checked the log in the asus web interface and nothing of note there. Attached is a reboot log, not sure why the time is reverts to dec 31.
i searched the log and found this line but no errors after it so not helpful, no mention of rc.unslung in system log

Dec 31 16:00:27 custom script: Running /jffs/scripts/services-start

executing from command line works perfectly

/opt/etc/init.d/rc.unslung start
Starting lighttpd...  done.
Starting syncthing...  failed.
Starting chroot'ed Debian services...
Starting syncthiing for root
Done.

So why will it not run at boot when called from /jffs/scripts/services-start ?

This is how the system is set up to start services, but it isn't doing it at boot. works perfectly from command line.
I put a couple of touch /tmp/filename before and afer the line to call it in the /jffs/scripts/services-start file and the files are created so i know it is executing those lines. But any touches i put in the rc.unslung file are only created when i run it from command line.
the rc.unslung file has the word fork in it so maybe it is already doing this

you fork a child process and then run a script which will first chroot to debian and then start synching from there. I think that's the right direction to try.

i tried modifying my /services-start script, touch works but rc.unslung doesn't

#!/bin/sh
touch /tmp/aaazero
script="/opt/etc/init.d/rc.unslung start"
/opt/etc/init.d/rc.unslung start
touch /tmp/aaaastart
script="/opt/etc/init.d/S99debian start"
#/etc/init.d/S98syncthing start
touch /tmp/aaaamiddle

 

[kvic]

System clock

On reboot AC68's system clock reverts back to default (in your case is Dec 31 something..). Because home routers do not come with a RTC - a battery powered real-time clock which continues to run when a device is cut off from power supply. This design decision is not flawed as routers are expected to work 24/7. Also being able to access Internet means a router has better way to keep system time.

On reboot, AC68 re-initializes system time through NTP. But the system need minimal facilities ready e.g. kernel loaded, network interfaces up and WAN address acquired before it can run NTP. So there is a short delay..not by much about 30s after a router comes back to live..

Why rc.unslung not run

The issue that your rc.unslung doesn't run is because your Entware disk is not ready. Here are the excerpts from your syslog:

Dec 31 16:00:27 custom script: Running /jffs/scripts/services-start
Dec 31 16:00:32 custom script: Running /jffs/scripts/post-mount (args: /tmp/mnt/sdb1)
Sep 26 00:18:37 custom script: Running /jffs/scripts/post-mount (args: /tmp/mnt/My_Book)
Sep 26 00:18:41 custom script: Running /jffs/scripts/post-mount (args: /tmp/mnt/sda1)

services-start is run at 27s after AC68 reboots. However, all your disks are mounted after that. The first disk (sdb1) is mounted at 32s after reboots. The other two are mounted after NTP finishes adjusting system clock. So this explains why rc.unslung was never able to be executed.

One solution

The proper place to call rc.unslung is inside the custom script post-mount. Here is mine

$ cat /jffs/scripts/post-mount
#!/bin/sh

tag=$(basename $0)

logger -t $tag $1

if [ "$1" == "/tmp/mnt/sandisk" ]
then
    ## workaround firmware bug
    ## the bug: mount in mount_r() instead of mount_fstab()
    mount /tmp/mnt/sandisk -o remount,rw,nodev,noatime,barrier=1,data=writeback
 
    ## turn on swap file
    swapon /tmp/mnt/sandisk/swapfile

    ## link entware.arm to /opt
    ln -sf $1/entware.arm /tmp/opt

    ## run Entware INIT
    /opt/etc/init.d/rc.unslung start
fi

Replace "/tmp/mnt/sandisk" with one of your external disks where you installed Entware. If your Entware disk is a pen drive, then you may want to keep my firmware bug workaround lines. That speeds up disk access by not updating last access attributes. Otherwise, you can remove those lines. Same for swapfile. If you don't have, you can comment out for now.

Get synching run on reboot

Next, back to synching. Since your debian+syncthing is not part of Entware, it's not necessary to have rc.unslung to start for you. Though it's perfectly doable. So I would start with easier first.

• Install the package "daemonize" from Entware (you need it to spawn a child)
• Then write a little script that contains the lines you use to run Debian+syncthing from command prompt. Save the script as, say, /jffs/bin/debian_syncthing.sh (give it correct permission).
• Now add this line to the end of the above post-mount (maybe after rc.unslung)
  • daemonize /jffs/bin/debian_syncthing.sh

Hope this shall get it started. Or else we need to take a closer look at debian_syncthing.sh.

 

[voipair]

The proper place to call rc.unslung is inside the custom script post-mount

that was all i needed to do, thank you very much for following along and all your advice

cat /jffs/scripts/post-mount
#!/bin/sh

if [ $1 = "/tmp/mnt/sda1" ]
then
  ln -nsf $1/entware.arm /tmp/opt
fi
swapon /opt/swap

/opt/etc/init.d/rc.unslung start

services-start is run at 27s after AC68 reboots. However, all your disks are mounted after that. The first disk (sdb1) is mounted at 32s after reboots.

Thanks for pasting those lines from the log, it helped me understand how to read the log file.

Though it's perfectly doable. So I would start with easier first.

Easier ended up being adding to post-mount

Since your debian+syncthing is not part of Entware, it's not necessary to have rc.unslung to start for you.

Interesting, thanks for the explanation.

• Install the package "daemonize" from Entware (you need it to spawn a child)

• Then write a little script that contains the lines you use to run Debian+syncthing from command prompt. Save the script as, say, /jffs/bin/debian_syncthing.sh (give it correct permission).

• Now add this line to the end of the above post-mount (maybe after rc.unslung)
  • daemonize /jffs/bin/debian_syncthing.sh

Thanks, this is essentially the answer handed to me. I haven't tried it for syncthing but may use it for something else i am working on.

to expand on the solution i used, i followed the instructions that i used to install debian and they told me how to start debian services. basically add the name to /opt/etc/chroot-services.list. It looks to me like they have it set up sort of if not actually doing what kvic suggested. to add another service just add a line to start, for ex to add crashplan to start with the router at boot (all i had to do was add crashplan to that file).

cat /opt/etc/chroot-services.list
syncthing
crashplan

you also need the startup script in /opt/debian/etc/init.d/ or if you are in debian /etc/init.d/

 

[kvic]

I skimmed the instructions link (2nd last) you provided in #16. Seems this chrooted debian is nicely setup for booting within itself and integrated to launch it by rc.unslung. Interesting stuff (btw the author is a member of this forum). I wonder if SSHD could also be run within there so that you could interact with this chrooted environment after its birth..

Glad to hear you get it working!

 

[voipair]

SSHD could also be run within there so that you could interact with this chrooted environment after its birth

ssh into the router and they have script for starting, stoping, restarting debian
so to interact

debian enter

as i was testing this to see if the debian services were started i would ssh into the router and type debian status, to see if it was running.
oddly, now when i run top to see what is running the services started in the /opt/etc/chroot-services.list show in the router top and not in the debian top, previously i would run the executables after typing debian enter and they would show in the debian top, and i don't think they were listed in the router top (could be wrong).
i don't understand chroot enough to know what that change means, both crashplan and syncthing appear to be running normally.

 

[vmkogeretla]

Why install Debian, it is fine if Syncthing set to Entware-ng? It is necessary to unpack, and run with the -h. And see the available parameters. Transfer to /opt/bin/, to create S89syncthing /opt/etc/init.d/ code. Files are attached.

https://github.com/syncthing/syncth.../v0.12.17/syncthing-linux-arm-v0.12.17.tar.gz

You can make a script easy-install.sh, who can and will be taken, there is not difficult.

 

[cuekay]

Hi all. I tried to run /opt/bin/syncthing per the advice above using three different versions with the same results: 1) the version on entware-ng's ARM repo 2) the attachment from vmkogeretla and 3) a most current version as of today 0.12.23. The result is as follows:

[monitor] 04:24:19 INFO: Starting syncthing
[monitor] 04:24:19 FATAL: fork/exec /opt/bin/syncthing: cannot allocate memory

But my top isn't showing any spikes between execution of the binary and the error messages. It more or less looks like this:

Mem: 117628K used, 138088K free, 0K shrd, 748K buff, 15848K cached
CPU:  0.0% usr  0.0% sys  0.0% nic 99.9% idle  0.0% io  0.0% irq  0.0% sirq
Load average: 0.00 0.01 0.05 1/104 14078
  PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
 1561     1 admin S    49552 19.3   0  0.0 radiusd
  915     1 admin S    25444  9.9   1  0.0 asterisk
 1010  1009 admin S    10800  4.2   1  0.0 /opt/bin/php-fcgi
 1023  1010 admin S    10800  4.2   1  0.0 /opt/bin/php-fcgi
 1022  1010 admin S    10800  4.2   1  0.0 /opt/bin/php-fcgi
13306 13305 admin S N   9248  3.6   1  0.0 minidlna -f /etc/minidlna.conf -R
13167     1 admin S     9248  3.6   0  0.0 minidlna -f /etc/minidlna.conf -R
13305 13167 admin S     9248  3.6   1  0.0 minidlna -f /etc/minidlna.conf -R
13113     1 admin S     6512  2.5   0  0.0 httpd
  459     1 admin S     6404  2.5   0  0.0 disk_monitor
  964     1 admin S     6380  2.4   1  0.0 ntpd -c /opt/etc/ntp.conf -f /opt/
    1     0 admin S     6324  2.4   0  0.0 /sbin/preinit
  450     1 admin S     6228  2.4   0  0.0 watchdog
  451     1 admin S     6224  2.4   1  0.0 sw_devled
 1033     1 admin S     6224  2.4   1  0.0 ntp
 1066   450 admin S     6224  2.4   1  0.0 hour_monitor
13094     1 admin S     6224  2.4   0  0.0 /sbin/wanduck
  433     1 admin S     6224  2.4   1  0.0 wpsaide
13139     1 admin S     6224  2.4   1  0.0 usbled
  460     1 admin S     6224  2.4   1  0.0 bwdpi_check

Thanks.