BACKUPMON BACKUPMON v1.8.20 -Oct 12, 2024- Backup/Restore your Router: JFFS + NVRAM + External USB Drive! CIFS/SMB/NFS! (Now available in AMTM!)

[Viktor Jaep]

BACKUPMON v1.8.20
Released October 12, 2024

First off -- HUGE thanks to @Jeffrey Young for sharing his original backup script. His script is the main engine of BACKUPMON, and all credit goes to him! BACKUPMON is simply a wrapper around Jeff's backup script functionality, adding easy-to-use menus, more status feedback, and the ability to launch a restore based on your previous backups. Also, big thanks to @Martinski for his many contributions as well as his extremely helpful AMTM email library script, and huge props to @visortgw for contributing to the backup methodologies thread with his scripts and wisdom!

Executive Summary: BACKUPMON is a shell script that provides backup and restore capabilities for your Asus-Merlin firmware router's JFFS, NVRAM and external USB drive environment. By creating a network share off a NAS, server, or other device, BACKUPMON can point to your location of choice, and perform a daily backup to this mounted drive. To perform daily, unattended backups, simply schedule a daily job through BACKUPMON, which will automatically get added to your CRON schedule. During a situation of need, where a backup would need to be restored after a catastrophic event with either your router or attached USB storage, simply copy the backupmon.sh + backupmon.cfg files over to a newly formatted /jffs/scripts folder, ensuring that your external USB storage was formatted with the same exact name (how-to's are all retrievable from the instructions.txt in your backup folder), and perform the restore by running the "backupmon.sh -restore" command, selecting the backup you want to use, and going through the prompts to complete the restoration of both your /jffs, NVRAM and external USB drive environments.

Use-case: BACKUPMON was designed to backup from, and restore to an already configured router from an external network resource, given a situation of a corrupted USB drive, botched Entware environment, or other general corruption issues. It can, however, also restore you back to a previous state if you decide to completely wipe your router or external drive from scratch. You can use it to move from one external USB drive to another... say, upgrading from a flashdrive to an SSD! You could also use it to restore your environment to a similar router if your old one dies, and you pick up the same model + hardware revision + firmware level as a replacement (at your own risk).

Here are a couple of different network/USB backup scenarios that it is able to handle (as of v1.35):

Router USB Drive -> External Network Device/Share (on your local network)
Router USB Drive -> Local Router Network Share (could be mounted to a secondary partition on your USB Drive, or secondary USB Drive)
Router USB Drive -> Secondary Router USB Drive (plugged into the secondary USB port)
Router USB Drive -> Router USB Drive (backing up to itself into a separate folder... of course, not recommended, but possible)
Router USB Drive Partition 1 -> Router USB Drive Partition 2 (kinda like the one above, but gives it a little more separation)

NOTE: If you do go down the path of backing your USB drive to your USB drive, it's possible, but not recommended. If you accept this risk, you want to also make sure you exclude your backup folder name in the exclusion file, so you don't back up your backup folder, which will lead to exponential growth of your backup files. The safest way still is to store backups is far away from the device being backed up... so use at your own risk.

What it should NOT be used for: It is not meant to restore backups from one particular model of router (ex: RT-AC86U), to a different shiny new model (ex: GT-AX6000) that you just picked up. In this case, it's still best to set your new router up from scratch, and selectively import any critical files manually from your .TAR backups to your new router if needed. Also, please do not restore your settings/backups from an older firmware to a newer firmware. Your router's CFG (settings) file is meant for the current firmware you're on, so if you do restore, make sure it's still on the same firmware as before.

Requirements:

• This script will allow you to back up one partition of your choice. You are no longer limited to sda1.
• Your Network Backup Target must be able to speak the NFS or CIFS/SMB protocol, as that is the method currently used to back up your files across the network. CIFS (Common Internet File System) is a dialect of the SMB (Server Message Block) protocol, and is a very broad standard supported by Windows, Linux and Apple devices. NFS is more heavily used with Linux implementations, though supported on Windows as well, and is also supported by BACKUPMON.
• You now have the option to backup from USB1 to USB2, or USB to itself... though not recommended.
• Your External USB should have a valid drive label. Having a blank label may create issues backing up or restoring.

BACKUPMON is free to use under the GNU General Public License version 3 (GPL 3.0).

This project is hosted on GitHub

Latest release notes: [updated here]

Github Change Log [Here]

What's new: Patches & Fixes, Shared Email Library script redundancy, -secondary switch functionality, Firmware 3006 CIFS fixes, Error Log/Error Display Optimizations, NFS+New UI, Restore File Order, Exclusion file guidance, AMTM Email Notifications, tar.gz Integrity Checks, Integration with MerlinAU!, Swap File Exclusion, Multiple SMB Protocol Support, Dedicated Log File, USB Backup Options/BW Mode, Base64 Pwd Encoding & More!, Added to AMTM!, EXT USB optional, Secondary Backups + Connection Tester, Auto Purge, Timer bypass, Forced Reboot/Source-Target Checks, NVRAM backup/restore!, NVRAM Extract, Go Live!, Perpetual Backup Purging, Perpetual Frequency added, Basic vs Advanced Backup Modes, Scheduled backups & Weekly/Monthly/Yearly Frequencies!, Release Candidate!

Screenshots:

In normal backup mode, a backup will launch after 10 seconds, giving you a chance to enter setup mode, or restore mode. Normally, you would be running this command from an automated CRON job, to back up your environment unattended each day.

In a normal backup scenario, you will get STATUS feedback after each major item is completed. TAR status messages are left on to help you with any troubleshooting should something not work right, or be locked.

In a restore scenario (by executing 'backupmon.sh -restore" or hitting the X key within 10 seconds of a backup, you are presented with a list of information to successfully accomplish a restore.

How is this script supposed to run?​

In a normal, daily mode, this script would run from a CRON job, in order to back router completely on a daily basis. Instructions:

1. Download and install directly using your favorite SSH tools, copy & paste this command (or install directly from AMTM):
   

   curl --retry 3 "https://raw.githubusercontent.com/ViktorJp/BACKUPMON/master/backupmon.sh" -o "/jffs/scripts/backupmon.sh" && chmod 755 "/jffs/scripts/backupmon.sh"

2. Configure it using this command:
   

   sh /jffs/scripts/backupmon.sh -setup

3. Run a backup manually in an SSH window with this command:
   

   sh /jffs/scripts/backupmon.sh -backup (or simply just execute 'backupmon')

4. Run a restore manually in an SSH window with this command (or hit the X button with 10 seconds of a backup starting):
   

   sh /jffs/scripts/backupmon.sh -restore

5. To make things easier, you can now just type the script name itself (without the path/extension), like so:
   

   backupmon

 

[Viktor Jaep]

Do I need to configure anything?​

You can enter the setup screen with the command 'backupmon.sh -setup' or by hitting the 's' key within 10 seconds of the backup starting:

Hitting the "sc" option, there are currently 15 items you can configure.

I'm definitely looking for your feedback... what works, what doesn't... what else would you like to see. But all-in-all, as good ideas come up for things to possibly add, very much a WIP (work-in-progress). Lots more stuff to come... I'm looking at adding a way to add some functionality to schedule the backups in CRON... so check back soon!

More Information:

1.) How does it all work? In TLDR; format, when BACKUPMON runs a regular backup, it will create a daily folder under the folder structure you have defined for it to connect to on your network share. So for example today, on September 4, 2023, it would create a /04 folder (if you're on a "Monthly" frequency), where its backups will get dumped into.

2.) Backup Retentions- there are 4 different kinds depending on your preference:

• "Weekly" retention would write "Mon, Tue... Sun" folders.
• "Monthly" would write daily "01, 02, 03... 30, 31" folders for each day of the month,
• "Yearly" would write "001, 002, 003... 364, 365" folders for each day of the year.
• "Perpetual" would write a unique folder based on date/time, such as "20230909-092511".

At this time, "basic mode" (explained below in more detail) will back everything up in 1 week, month or year cycles, and will overwrite the backup files each time it comes across the same day in its cycle, unless you've chosen the "Perpetual" retention, in which case it keeps backups forever under its unique folder structure. In "advanced mode", it does not overwrite backups in your weekly, monthly and yearly folders, and keeps everything forever... Here's a screenshot of what your backup folders on your network share might look like (please note there's a bit of a mix between "Monthly" and "Perpetual" retention backups as I was testing...

3.) Backup copies - During a backup, BACKUPMON will also copy over a copy of 'backupmon.sh', 'backupmon.cfg', any optional TAR exceptions list you have defined, a text dump of your NVRAM, and a copy of restore "instructions.txt". (see above) Within this instructions.txt file, you will find the same instructions you see on screen during a restore, telling you what steps need to be taken for a successful restore. In it, is also your latest external USB drive label, which must be duplicated when you format a new USB drive, and want to restore your backup to it.

4.) The optional TAR exclusion list can be placed in a directory of your choice. PLEASE NOTE: By default, backupmon will back up everything, swap files, log files, included! It is a good idea to exclude these using this exclusion list file. You may encounter issues restoring backups if you try to restore a swap file to an already existing swap file on your router. You can create this file anywhere, but I prefer to keep things in /jffs/addons/backupmon.d and named it "exclusions.txt", just to clear the clutter. The contents of this file is simply a list of files that TAR can safely ignore during a backup. Certain files, like your swap file, or other temporary files, aren't needed when restoring a backup, and get recreated after a restore. As of v1.42, the swap file is being excluded from backups unless you override this option (#9) in the config menu. Huge thanks to @visortgw for sharing his exclusion file... the default exclusion items list looks something like this:

exclusions.txt

myswap.swp
entware/var/log/*
skynet/skynet.log
entware/var/lib/unbound/unbound.log
*.tar
*.gz

Save this file somewhere on your router, and make sure it's properly referenced in BACKUPMON. Eliminating swap and other temp files could really speed up your backups tremendously, and save you from headaches down the road on a restore.

5.) The Basic vs Advanced Mode differences are described below:

BASIC
- Only backs up one backup set per daily folder
- Backup file names have standard names based on jffs and USB drive label names
- Self-prunes the daily backup folders by deleting contents before backing up new set (N/A when 'Perpetual Retention' is enabled)
- Will overwrite daily backups, even if multiple are made on the same day (N/A when 'Perpetual Retention' is enabled)
- Restore more automated, and only required to pick which day to restore from

ADVANCED
- Backs up multiple daily backup sets per daily folder
- Backup file names contain extra unique date and time identifiers
- Keeps all daily backups forever, and no longer self-prunes
- Will not overwrite daily backups, even if multiple are made on the same day
- Restore more tedious, and required to type exact backup file names before restore

6.) Manually Purging Perpetual Backups -- Under the config menu, when selecting "perpetual" backup retention, you will have the option to choose whether you want to enable backup purging for everything older than a certain number of days. Once configured, you can manually access the purge backups functionality under the operations area under the setup menu. This function will step you through multiple prompts before committing to delete any backups, and will show you which backups are affected. PLEASE NOTE: deleting backups is permanent, and advise caution on what you are deleting, and ensure that the list presented to you is what you want deleted. Also, PLEASE NOTE: if there are any backups that you want to save or store for later use, PLEASE move these to a SAFE and otherwise secure location outside of the backup folder structure that BACKUPMON utilizes, else there will be a chance these backups will get deleted.

Automatically Purging Perpetual Backups -- if you have enabled "Perpetual Backups", and "Purge backups", and run BACKUPMON with the -backup switch (ie. "sh backupmon.sh -backup"), then it will auto purge your backups after it finishes your normal backup. This was done to make it simpler when running this under a daily CRON job to not only back up but purge as well with one command.

See screenshot below as an example:

7.) Commandline switches - When running backupmon with a switch, you can utilize other functionality that is normally only available through the UI. Here is an explanation of what each switch does:

sh /jffs/scripts/backupmon.sh -- Running the script with no switches simply runs the primary and secondary backups (no auto purge).
sh /jffs/scripts/backupmon.sh -setup -- This displays the setup + operations menu
sh /jffs/scripts/backupmon.sh -backup -- This runs the normal primary and secondary backups, and runs auto purge
sh /jffs/scripts/backupmon.sh -restore --This initiates the restore procedure, allowing you to pick from primary or secondary backups
sh /jffs/scripts/backupmon.sh -purge -- This runs the auto purge routine against the perpetual backup folders
sh /jffs/scripts/backupmon.sh -secondary -- This runs the secondary backup only, and does not perform auto purge

B&W Mode: You can now optionally run a monochrome color scheme for those who are capturing screen output by using a secondary -bw switch after any of the other primary switches. Example:

sh /jffs/scripts/backupmon.sh -backup -bw -- This runs the normal primary and secondary backups, and runs auto purge in B&W mode!

 

[Viktor Jaep]

How do I restore a backup?

As per the instructions.txt file that is automatically copied to your backup folders, the restore instructions are as follows:

RESTORE INSTRUCTIONS

IMPORTANT:
Asus Router Model: GT-AX6000
Firmware/Build Number: 3004.388.4
EXT USB Drive Label Name: ASUS-SSD

WARNING: Do NOT attempt to restore if your Asus Router Model or Firmware/Build Numbers differ from your backups!

Please ensure your have performed the following before restoring your backups:
1.) Enable SSH in router UI, and connect via an SSH Terminal (like PuTTY).
2.) Run "AMTM" and format a new USB drive on your router - label it exactly the same name as before (see above)! Reboot.
3.) After reboot, SSH back in to AMTM, create your swap file (if required). This action should automatically enable JFFS.
4.) From the UI, verify JFFS scripting enabled in the router OS, if not, enable and perform another reboot.
5.) Restore the backupmon.sh & backupmon.cfg files (located under your backup folder) into your /jffs/scripts folder.
6.) Run "sh backupmon.sh -setup" and ensure that all of the settings are correct before running a restore.
7.) Run "sh backupmon.sh -restore", pick which backup you want to restore, and confirm before proceeding!
8.) After the restore finishes, perform another reboot.  Everything should be restored as normal!

PLEASE NOTE: The USB Drive Label name is included in this file for you to remember what your drive was called, so when you format a new USB drive, you can label it with the same exact name. If you really want to call it something different, try changing the label after you have restored all your files.

Post-restore activities
I don't have an overly complex environment on my test router, and after testing out a restore on my test router using the instructions above, I didn't have any post-restore issues.

The key is, once all files are restored, virtually any complex environment should continue to run, even if you're also running Diversion, YazFi, Unbound, x3mrouting, etc... If you run across any other gotchas, please post them below, and I'll add to the instructions.

Automation
BACKUPMON now has the option to schedule your daily backup routine, and will add a CRON command to kick things off. Please refer to the configuration menu to set the hour and minute. Once enabled, it will automatically schedule your backup job. If you want to get creative and add other jobs, please use the instructions below to create a backup job in CRON... You can look up how to properly format a scheduled event using tools like these: https://crontab.guru/

To run a daily job at 2:30am, BACKUPMON will add this line to your services-start file (located under your /jffs/scripts folder), and it will automatically be scheduled after your next router reboot:

cru a RunBackupMon "30 2 * * * sh /jffs/scripts/backupmon.sh"

 

[Viktor Jaep]

You know what makes me a GRUMPY CAT!? Starting a new thread due to the <grrr> DREADED 6 month age rule </grrr>

 

[Ripshod]

You know what makes me a GRUMPY CAT!? Starting a new thread due to the <grrr> DREADED 6 month age rule </grrr>

Look how far the idea has come in just 6 months. Your baby is walking!

 

[visortgw]

Look how far the idea has come in just 6 months. Your baby is walking!

No, she is running!

 

[ExtremeFiretop]

You know what makes me a GRUMPY CAT!? Starting a new thread due to the <grrr> DREADED 6 month age rule </grrr>

 

[Viktor Jaep]

View attachment 56904

LOL, that's fantastic. Definitely makes me less grumpy.

Look how far the idea has come in just 6 months. Your baby is walking!

No, she is running!

With many thanks to you guys... Appreciate your support with all this...

 

[Viktor Jaep]

Many thanks to both @Martinski and @ExtremeFiretop for their continued development efforts to determine a fool-proof way of determining EXT USB label names... Also a bunch of other new logs/checks added in this version!

What's new?
v1.5.13 - (March 3, 2024)
- PATCH: Continued development efforts from both @Martinski and @ExtremeFiretop to come up with a fool-proof solution to determine EXT USB drive labels. Thanks to both for your contributions to BACKUPMON! With the recent finding that the blkid command is creating some weird kernel error messages in the syslog, some workarounds have been put in place to only call on mounted partitions.
- PATCH: Included more log statements for the EXT USB .tar.gz file integrity check. Now includes a new on-screen/log entry that indicates that the integrity check is starting, and to please stand by. When it comes to larger .tar.gz files, even integrity checks can take some time to complete. This just gives an indicator that it started working on it.
- PATCH: Included an error check on the NVRAM SAVE command completion. If the router isn't able to save it's firmware .cfg file, BACKUPMON will error out with the proper error codes.

Download link (or update directly within AMTM/BACKUPMON):

curl --retry 3 "https://raw.githubusercontent.com/ViktorJp/BACKUPMON/master/backupmon.sh" -o "/jffs/scripts/backupmon.sh" && chmod 755 "/jffs/scripts/backupmon.sh"

Significant screenshots:

Here you can see that an additional entry is made indicating the start of the EXT USB Drive Integrity Check

 

[Viktor Jaep]

I updated the script to 1.5.10 and started seeing this kernel log whenever I open the backupmon config menu:

RT-AX88U-4533 kernel: ubi0 error: ubi_open_volume: cannot open device 0, volume 0, error -16

and found that the new blkidLabels statement (line 1759) is the culprit. It's returning the USB drive label correctly so probably okay to ignore though.

@Viktor Jaep did you notice anything during PR?

xxx@RT-AX88U-4533:/tmp/home/root# blkid
/dev/sda1: LABEL="MERLIN-SSD" UUID="662fe021-75bf-fds1-8ebc-6dec1f4ddedf"
xxx@RT-AX88U-4533:/tmp/home/root# blkid | grep "^/dev/sd.*: LABEL=" | sort -dt ':' -k 1 | awk -F
' ' '{print $2}' | awk -F '"' '{print $2}'
MERLIN-SSD

This has been resolved in 1.5.13... thanks for your feedback on this!

 

[Ripshod]

Updated and test backup completed

 

[monkitrainer]

This has been resolved in 1.5.13... thanks for your feedback on this!

Looks good and works great Thanks as always!

 

[Viktor Jaep]

Looks good and works great Thanks as always!

Many thanks to @Martinski and @ExtremeFiretop for this one!

 

[ExtremeFiretop]

Many thanks to @Martinski and @ExtremeFiretop for this one!

Happy I didn't have to send anyone down the path of Util-Linux never to be seen again. This solution we found is pretty solid I think. Enjoy buddy!

 

[smarthome-enthusiast]

@Viktor Jaep I know we should be patient for NFS support, but (also @everyone) are there any workarounds that are reliable for backing up to network share that doesn't involve SMB?

SMB breaks my file sharing (after ~6hrs file server is unreachable and requires a reboot to reconnec and all devices are non-Windows Systems) and seems a waste of interface bandwidth (separate docker server) and cpu to build a docker container to SMB -> NFS for periodic backups.

 

[Viktor Jaep]

@Viktor Jaep I know we should be patient for NFS support, but (also @everyone) are there any workarounds that are reliable for backing up to network share that doesn't involve SMB?

SMB breaks my file sharing (after ~6hrs file server is unreachable and requires a reboot to reconnec and all devices are non-Windows Systems) and seems a waste of interface bandwidth (separate docker server) and cpu to build a docker container to SMB -> NFS for periodic backups.

I wish I could be of more help here... but I don't have a really good way to test this on my end... coding of which would be like throwing darts blindfolded at a target 50 feet away. Based on some earlier conversations with @Jeffrey Young, it seems there needs to be bunch of boxes checked for this to work from our routers... and may be more trouble that it's worth to get configured. That's really my hesitation... is that whatever/however your DFS server is configured, that the variations out there would not make it conducive to build a "standard" solution for in BACKUPMON.

However, if anyone is able to make the necessary mount statement changes in the 'develop' branch for the backupmon.sh script on github, or provide other suggestions/code, I would be happy to write more code around that to make it easier to work with for everyone else that wants to use NFS. My only request is that whatever is suggested, would work for everyone... something vanilla enough that would eliminate as many obstacles, special cases/configuration issues.

Based on that earlier conversation, here's a starting point... this is one suggested approach that @Jeffrey Young worked out... I found that NFSD needs to be turned on under the servers center on our router for that second statement not to bomb out.

modprobe nfs nfsv3
[ $(pidof mountd) ] || /usr/sbin/mountd -N 2
mount -t nfs 10.0.100.90:/Backups /tmp/mnt/primary -o "nfsvers=3,nolock,_netdev,rsize=8192,wsize=8192"

 

[Jeffrey Young]

I believe BACKUPMON can work now without having to call a mount to an SMB share within the script. If so, mount your NFS share prior to calling BACKUPMON and unmount afterwards.

 

[Jeffrey Young]

Based on that earlier conversation, here's a starting point... this is one suggested approach that @Jeffrey Young worked out... I found that NFSD needs to be turned on under the servers center on our router for that second statement not to bomb out.

Interesting, I found that you did not have to call mountd if you had NFS turned on under the GUI. Otherwise, the call to mountd is required. I will have to check my notes, but I think you need to call mountd as mountd -N 2 &

 

[Viktor Jaep]

Interesting, I found that you did not have to call mountd if you had NFS turned on under the GUI. Otherwise, the call to mountd is required. I will have to check my notes, but I think you need to call mountd as mountd -N 2 &

How can that be? With the mount point being the same as where the NFS mount point would end up?.

 

[visortgw]

How can that be? With the mount point being the same as where the NFS mount point would end up?.

If it works, PFM!