Rclone (Entware-NG) issues configuring and syncing

[MarCoMLXXV]

I'm trying to find a way to sync my Dropbox folder to an USB 3.0 drive connected to my router. I'm mostly using Linux and as the Linux Dropbox client has several issues since the last update around two years ago, and Dropbox doesn't seem to intend to fix the (rather simple) issues, I was looking for a cli solution (and use Dropbox as a network share mounted on my several devices through sftp). There are several fixes, but haven't found one that actually stays working, so I'm somewhat fed up with Dropbox by now. Tried to install the Dropbox daemon (the non-gui version, just solely the daemon and the python script to control it), which should work, as I have Python 2 .7.x installed, but it fails at line 1 after every single attempt.

So I started looking for a different solution and found rclone in the entware-ng packages repository. However, after I installed it successfully, and entered

rclone config

as described here, it returns

marco@router:/tmp/home/root# rclone config
runtime: this CPU has no floating point hardware, so it cannot run
this GOARM=7 binary. Recompile using GOARM=5.

Looks like it's assuming it's not being ran at an ARMv7 architecture, but as my router is an RT-AC68U (rev E1) with an ARMv7l cpu , that's seems like a wrong assumption. Did some searching on the forum as well on the net, but could not find any one with similar issues.

Any suggestions? A working alternative to sync Dropbox to my router's external drive will do fine too.

 

[Jack Yaz]

Is rsync available on entware?

 

[MarCoMLXXV]

It is:

marco@router:/tmp/home/root# opkg list | grep "rsync"
librsync - 2.0.0-1 - librsync implements the rolling-checksum algorithm of remote file synchronization that was popularized by the rsync utility and is used in rproxy. This algorithm transfers the differences between 2 files without needing both files on the same system.
rsnapshot - 1.4.2-1 - rsnapshot is a filesystem snapshot utility for making backups of local and remote systems.  Using rsync and hard links, it is possible to keep multiple, full backups instantly available. The disk space required is just a little more than the space of one full backup, plus incrementals.
rsync - 3.1.2-2 - rsync is a program that allows files to be copied to and from remote machines in much the same way as rcp. It has many more options than rcp, and uses the rsync remote-update protocol to greatly speed up file transfers when the destination file already exists.  The rsync remote-update protocol allows rsync to transfer just the differences between two sets of files across the network link.
rsyncd - 3.1.2-2 - rsyncd is a configuration file and initscript to utilize rsync as a daemon. It uses the same binary as rsync.

Downloaded the latest version of rclone from the website http://rclone.org, needed to install the arm version for my router and linux version for my netbook as you need a (graphical) browser to get rclone authorized during setup. However, rclone is unable to safe it's config for some reason, but doesn't say where it's trying too. Bit frustrating, as the whole binary is portable (which is great), but it wants to store it's config elsewhere, presumably in a read-only part of the router's filesystem.

Ps. I've read the documentation you've sent me. Very interesting, but out of my league for now. Bookmarked it though for future reference

 

[Jack Yaz]

Looking at the source, it should end up storing it within the rclone dir

https://rclone.org/docs/#config-config-file

// Default to ./.rclone.conf (current working directory)
    Errorf(nil, "Couldn't find home directory or read HOME or XDG_CONFIG_HOME environment variables.")
    Errorf(nil, "Defaulting to storing config in current directory.")
    Errorf(nil, "Use -config flag to workaround.")
    Errorf(nil, "Error was: %v", err)
    return hiddenConfigFileName

 

[MarCoMLXXV]

Strange, I double checked permissions. Will check again or maybe see if I can find a sample config. It's supposed to be copied to /usr/bin, but that's read only, so I created a directory on my thumbdrive. Config went well, got a token to copy, but config doesn't get stored. After running rclone config, you can only add, edit, delete or exit and to start using the remote connection you just configured, you need to exit. That's when your freshly made config vanishes.

Wondering why you asked whether rsync was available in entware-ng? Rsync won't work with Dropbox, will it? Or can it handle oAuth tokens?

 

[Jack Yaz]

Strange, I double checked permissions. Will check again or maybe see if I can find a sample config. It's supposed to be copied to /usr/bin, but that's read only, so I created a directory on my thumbdrive. Config went well, got a token to copy, but config doesn't get stored. After running rclone config, you can only add, edit, delete or exit and to start using the remote connection you just configured, you need to exit. That's when your freshly made config vanishes.

Wondering why you asked whether rsync was available in entware-ng? Rsync won't work with Dropbox, will it? Or can it handle oAuth tokens?

I think I saw the "Dropbox as FTP share" but missed the "it doesnt stay working" bit, sorry!

 

[Jack Yaz]

Having re-read again, you want to do something on the router to pull Dropbox from the cloud and then serve as a share for clients from router, is that right?

I'm tired >.<

 

[MarCoMLXXV]

I'm tired >.<

Me too ~.~

Simply put: Dropbox on Linux is a hassle and I have no time yet to set up my own cloud service. So, as I have 64Gb USB 3.0 sticking out of the back of my router, I was wondering whether there would be an easy way (cli-only) to sync my entire dropbox onto the router and add the folder as a share on the machines I use personally as an SFTP-share. That way only one client needs to be synced with dropbox, with more space than my dropbox allows (around 15 Gbit) and I finally got rid of the horrible Linux client for Dropbox. Unfortunately the cli daemon from dropbox doesn't work and the latest version of rclone doesn't either. Time to call it day and have a fresh look tomorrow.

 

[MarCoMLXXV]

Okay, one step further. It does save in the default directory (which isn't persistent, as it's /root by default) and therefore useless, but with the --config <PATH><FILENAME> flag I managed to put it elsewhere:

marco@router:/tmp/mnt/usb/rclone# ./rclone config --config /mnt/usb/rclone/config
2017/09/02 21:04:24 NOTICE: Config file "/mnt/usb/rclone/config" not found - using defaults
No remotes found - make a new one
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
name> Dropbox
Type of storage to configure.
Choose a number from below, or type in your own value
 1 / Amazon Drive
   \ "amazon cloud drive"
 2 / Amazon S3 (also Dreamhost, Ceph, Minio)
   \ "s3"
 3 / Backblaze B2
   \ "b2"
 4 / Dropbox
   \ "dropbox"
 5 / Encrypt/Decrypt a remote
   \ "crypt"
 6 / FTP Connection
   \ "ftp"
 7 / Google Cloud Storage (this is not Google Drive)
   \ "google cloud storage"
 8 / Google Drive
   \ "drive"
 9 / Hubic
   \ "hubic"
10 / Local Disk
   \ "local"
11 / Microsoft OneDrive
   \ "onedrive"
12 / Openstack Swift (Rackspace Cloud Files, Memset Memstore, OVH)
   \ "swift"
   \ "sftp"
14 / Yandex Disk
   \ "yandex"
15 / http Connection
   \ "http"
Storage> 4
Dropbox App Key - leave blank normally.
app_key>
Dropbox App Secret - leave blank normally.
app_secret>
Remote config
Use auto config?
 * Say Y if not sure
 * Say N if you are working on a remote or headless machine
y) Yes
n) No
y/n> n
For this to work, you will need rclone available on a machine that has a web browser available.
Execute the following on your machine:
    rclone authorize "dropbox"
Then paste the result below:

Downside is that I have to specify that each time I need to change or check something,* but hopefully I can set it up as a service, using cron to periodically sync it if there are any changes detected.

* = only when running rclone config, after that it will just look for it in it's own directory.

To be continued...

Edit: After pasting the auth token, the config file was created at the specified location. Be aware you do need a second system (with rclone installed) to get an auth token from Dropbox. I tried it with lynx on the router, but it won't work. The config script wants to open a browser to login to Dropbox, which it can't, so I opened a second ssh connection and manually pasted the url it wanted to open, but wasn't able to login and get a token. This could also be caused by 2FA, can't say for sure. With a second (non-headless) system it's a breeze, it takes less than 30 seconds to get the token and paste it in your ssh session.

 

[Odkrys]

you should to install softfloat version.

opkg install rclone_nohf

and run rclone config

 

[RMerlin]

Doesn't AiCloud's Smartsync support Dropbox? Been a while since I've looked at the AiCloud webui, but I know the dropbox_client is definitely there in the firmware sources.

 

[MarCoMLXXV]

Thanks for your help so far.

@Odkrys, not sure how often the Entware-ng packages are updated, but in the past five months I've been running Asuswrt-Merlin, haven't seen a single update of the several packages I have installed, so not sure whether the repository is still actively maintained. As for the rclone_nohf package, I thought the ARMv7l does have hardware floating point, or did I misread that part (see link to ARM's website) in the first post.

I've let rclone run overnight to see how it turned out and just discovered it did sync, but only partially. It apparently has an issue with naming conventions used on Dropbox's side, with appears to have been solved with the latest beta. I do get a feeling rclone is far from stable yet.

Currently running rclone-v1.37-117-g58f7b4edβ-linux-arm (built 2017-09-01) for the past 20 minutes in verbose mode and it appears to be syncing the missing files, with no errors so far.

Will update this thread as my journey continues.

As for AICloud @RMerlin, actually never looked into it, tbh. When I started using your firmware I saw many AICloud related (user) errors being posted that I decided, as I didn't need another cloud service, not to give it a try. But will look into it now, wasn't aware it might be able to sync different cloud services.

In addition to my last comment above: I do want to see if I can, over time, set up OwnCloud as I still have three WD network drives with 5 Tb of capacity in total gathering dust here, while paying for three different cloud (storage) services monthly (Microsoft 365 / Dropbox / iCloud).

 

[MarCoMLXXV]

Update: sync completed successfully with the beta mentioned in my previous post, it downloaded the missing ~6 GB without any errors in a little more than half an hour, will now verify both locations to see if it actually synced everything.

<snip>
Transferred:   6.114 GBytes (2.918 MBytes/s)
Errors:                 0
Checks:             26434
Transferred:         3717
Elapsed time:    35m45.4s

 

[MarCoMLXXV]

@RMerlin: Looking at ASUS Support, it seems like you're absolutely right: https://www.asus.com/support/FAQ/1010006/. AICloud Smart Sync supports Dropbox. Could have saved me some time, but most likely wouldn't have been as educational as this has been so far Nevertheless, good to know and someone else might be able to benefit from it.

 

[thelonelycoder]

A full How-To is in order once you confirm it works.
This might be helpful to others.

 

[MarCoMLXXV]

A full How-To is in order once you confirm it works.
This might be helpful to others.

It's like you're reading my mind ... However, will wait until this build of rclone is stable, as it is a very active project with many open issues on Github.

Ps. Had to shorten my signature as it became longer and longer, but you know who I'm referring to as well

 

[MarCoMLXXV]

Okay, I'm actually quite impressed of how lightning fast rclone managed to check 12.8Gb of files

marco@router:/tmp/mnt/usb/rclone/beta# ./rclone -v sync Dropbox:\/ /tmp/mnt/usb/Dropbox
2017/09/03 06:55:18 INFO  : Local file system at /tmp/mnt/usb/Dropbox: Modify window is 1s
2017/09/03 06:55:59 INFO  : Local file system at /tmp/mnt/usb/Dropbox: Waiting for checks to finish
2017/09/03 06:55:59 INFO  : Local file system at /tmp/mnt/usb/Dropbox: Waiting for transfers to finish
2017/09/03 06:55:59 INFO  : Waiting for deletions to finish
2017/09/03 06:55:59 INFO  : 
Transferred:      0 Bytes (0 Bytes/s)
Errors:                 0
Checks:             15076
Transferred:            0
Elapsed time:       42.5s

marco@router:/tmp/mnt/usb/rclone/beta#

Now, while preparing to bake some fresh bread thinking of to reliably check whether the cloned mirror is really identical to my Dropbox folder...

 

[MarCoMLXXV]

Okay, an update: the bread was delicious and rclone (latest beta) is doing fine besides some minor things I cannot explain yet. Found out today that it's not able to run as a daemon which would mean I'd need crond to have it run every x minutes to keep the local mirror of my Dropbox folder up to date.

While searching for alternative solutions I came across this project, which claims to do exactly what I want it to do and was written for... rclone.
It's a script using inotify to monitor a filesystem for changes and performs an action when it detects something has changed. In this case it can run rclone for a fast sync to make sure the latest updates on the local mirror are being updated.

Having said that, it can't detect any changes on the remote (actual) dropbox folder, so I'll probably have to use cron anyway to make sure changes to Dropbox remotely are being synced to the local mirror periodically as well.

It'll be an educational week, I suppose... Will keep you posted, for those interested!

 

[zyxmon]

not sure how often the Entware-ng packages are updated, but in the past five months I've been running Asuswrt-Merlin, haven't seen a single update of the several packages I have installed,

We try to update Entware every month or two. Sometimes it take a longer time. The changelog is here - https://github.com/Entware-ng/Entware-ng/wiki/Changelog
The last update was in December (binaries were uploaded only for Entware-3x, Entware-ng is comming). The previous update was in September.

 

[JoaoAntonio]

I'm trying to figure out a way to sync the hard disk connect to my router with Onedrive. Is it possible to use Rclone for this? I'm really noob on these kind of things and need some help of the experts here. Thz