BACKUPMON BACKUPMON v1.5.10 -Mar 1, 2024- Backup/Restore your Router: JFFS + NVRAM + External USB Drive! (**Thread closed due to age**)

[Viktor Jaep]

@Viktor Jaep I believe I'm seeing similar. Occasionally a daily backup will not be done, I only noticed last night when moving data from one raid to another. It's seems like something else may be resetting the cron list, losing the backupmon entry as well as other entries. I'll dig a little deeper for you when I next see it (not) happen.

Very peculiar... my services-start and cru look correct... I've been seeing backups taken each day, but I have a mystery gap between Jan 9 - Jan 15... not sure if I rebooted on the 15th. That's my only oddity. I'll keep an eye on this as well.

 

[Viktor Jaep]

Hi Viktor,

I don't use BACKUPMON - still using the original "one page" script that this all started from.
But I've made a little change you might want to 'borrow'.

I realised after the latest Merlin release from 388.5 to 388.6 that I couldn't tell which release my backups applied to.

So, if I wanted to roll back to a previous release, I have to remember when I 'upgraded' and pick a backup from before that date.

I've made a very crude change (I'm sure you can do better),
I just add $(nvram get webs_state_info) to the beginning of the folder name.

Intead of backing up to
M:\BACKUPS\Routers\SCHED_BU\AX86U\23
it creates the folder
M:\BACKUPS\Routers\SCHED_BU\AX86U\3004_388_6_0-23

I know you love to tinker - Maybe you could so something similar?

Thanks for the suggestion, @alan6854321! That certainly is a great way to filter out the FW version. The way BACKUPMON currently works, is that the user has complete control over the target path name. The last part of the target path is controlled by automation, depending if you backup by week (and inserts Mon, Tue, Wed, etc. folders), or month (and inserts 01, 02, 03, etc. day folders), or year (and inserts 001, 002, 003 day folders). Throwing this variable into the path mix may really mess with automation when the script is trying to go back in time to find the right folder to restore from.

But one thing it does do is keep track of what FW was used to back up the latest backup set. Each backup folder contains a file called "routerfw.txt", and contains this info:

RESTOREMODEL="GT-AX6000"
RESTOREBUILD="3004.388.6_0"

I use this info to compare to the current state of the router and FW when a restore commences, to ensure they match up, preventing someone from restoring an old firmware ontop of a new, or one router model to another. It would take a little searching, probably less than 30sec, but at least it gives people a clue when a FW upgrade happened.

 

[visortgw]

Very peculiar... my services-start and cru look correct... I've been seeing backups taken each day, but I have a mystery gap between Jan 9 - Jan 15... not sure if I rebooted on the 15th. That's my only oddity. I'll keep an eye on this as well.

I've seen the same, but not for just BACKUPMON. It appears that periodically some, but not all, cron jobs disappear for whatever reason. Also, I manually created multiple BACKUPMON cron tasks in post-mount to back up to local NAS, remote NAS, and USB — not that it's necessary, but simply because I could! Reboot restores them.

 

[Viktor Jaep]

I've seen the same, but not for just BACKUPMON. It appears that periodically some, but not all, cron jobs disappear for whatever reason. Also, I manually created multiple BACKUPMON cron tasks in post-mount to back up to local NAS, remote NAS, and USB — not that it's necessary, but simply because I could! Reboot restores them.

That's definitely concerning now that you and others are reporting issues with multiple cron jobs completely disappearing... I'm hoping it was just related to some temporary bug that worked itself out in some other 3rd party script update or FW upgrade as of late... ugh! If we continue noticing this behavior, I think it's time to bring this up under the main Asuswrt-Merlin group.

...and wow, sounds like you've got some triple redundancy going on over there!

 

[visortgw]

...and wow, sounds like you've got some triple redundancy going on over there!

Like I said, because I CAN, not because I need to... Also, it makes it easier to help verify functionality before installing on other rrouters that I manage remotely.

 

[bibikalka]

Very peculiar... my services-start and cru look correct... I've been seeing backups taken each day, but I have a mystery gap between Jan 9 - Jan 15... not sure if I rebooted on the 15th. That's my only oddity. I'll keep an eye on this as well.

Well, I reboot daily, and I have scheduled backups about ~20 min after a reboot. So I don't think /jffs/scripts/services-start cron job is disappearing, it's more like it has not been scheduled in the first place. Probably tasks before this one in services-start never finish?

/jffs/scripts/post-mount cron jobs do get scheduled properly, no issues there.

I do get an occasional issue where scribe starts to misbehave, and does not let some output into messages file. Strangely enough, a soft reboot does not fix this!!! If I reboot it, and then turn off power for a few seconds manually - things get restored. Somehow a soft reboot does not put things back into the pristine state, which is very strange to me. This behavior has been reported on other issues as well, such as misbehaving WiFi.

 

[miazza]

a soft reboot does not fix this!!!

Try with

service reboot

instead of a simple

reboot

.
It's not the same and is the way amtm reboot scheduler works.

 

[Viktor Jaep]

Try with

service reboot

instead of a simple

reboot

.
It's not the same and is the way amtm reboot scheduler works.

This is the more formal way from what I've found:

/sbin/service 'reboot'

 

[bibikalka]

This is the more formal way from what I've found:

/sbin/service 'reboot'

Excellent! I put this in!

Also, I noticed my backup cron job in post-mount was named the same, and was overwriting the original one from services-start, so I changed the line in post-mount to :

cru a RunBackupMon0 "28 4 * * * sh /jffs/scripts/backupmon.sh"

This way I could monitor the non-performance of services-start cron by whatever the timestamp is on the archives. I spaced them in time - this way it'll be easy to see. Here is my post-mount cron job:

cru a RunBackupMon "46 14 * * * sh /jffs/scripts/backupmon.sh"

 

[Martinski]

That's definitely concerning now that you and others are reporting issues with multiple cron jobs completely disappearing... I'm hoping it was just related to some temporary bug that worked itself out in some other 3rd party script update or FW upgrade as of late... ugh! If we continue noticing this behavior, I think it's time to bring this up under the main Asuswrt-Merlin group.

Well, I reboot daily, and I have scheduled backups about ~20 min after a reboot. So I don't think /jffs/scripts/services-start cron job is disappearing, it's more like it has not been scheduled in the first place. Probably tasks before this one in services-start never finish?

Given that stuck "nvram" calls are occurring in @Viktor Jaep's AX-class router (see this post), it's possible that other routers may be experiencing the same, which would mean that any "cru" call could also stuck while trying to add a cron job (as a consequence of the stuck "nvram" cmd), and if this happens during reboot, one or more cron jobs from "/jffs/scripts/services-start" may not get scheduled as expected.

Here's an example of a stuck "cru a" command as shown in the debug output from @Viktor Jaep's router:

2) "nvram get http_username" call while trying to add a cron job (Diversion_UpdateBL) related to Diversion.

 5925  5924 ViktorJp S     4008  0.3   2  0.0 nvram get http_username
 5924     1 ViktorJp S     3328  0.3   0  0.0 {cru} /bin/sh /usr/sbin/cru a Diversion_UpdateBL 00 2 * * Fri /bin/sh /opt/share/diversion/file/update-bl.div reset

 

[Ripshod]

Given that stuck "nvram" calls are occurring in @Viktor Jaep's AX-class router (see this post), it's possible that other routers may be experiencing the same, which would mean that any "cru" call could also stuck while trying to add a cron job (as a consequence of the stuck "nvram" cmd), and if this happens during reboot, one or more cron jobs from "/jffs/scripts/services-start" may not get scheduled as expected.

Here's an example of a stuck "cru a" command as shown in the debug output from @Viktor Jaep's router:

My issue is not that the cron jobs aren't added but the cron jobs getting wiped. I don't regularly reboot but my router can backup several days in a row then stop because the cron job is gone.

 

[Viktor Jaep]

Excellent! I put this in!

Also, I noticed my backup cron job in post-mount was named the same, and was overwriting the original one from services-start, so I changed the line in post-mount to :

cru a RunBackupMon0 "28 4 * * * sh /jffs/scripts/backupmon.sh"

This way I could monitor the non-performance of services-start cron by whatever the timestamp is on the archives. I spaced them in time - this way it'll be easy to see. Here is my post-mount cron job:

cru a RunBackupMon "46 14 * * * sh /jffs/scripts/backupmon.sh"

Good call! Yeah, if named the same, one would overwrite the other.

 

[Martinski]

My issue is not that the cron jobs aren't added but the cron jobs getting wiped. I don't regularly reboot but my router can backup several days in a row then stop because the cron job is gone.

Yeah, I understand. My post was directly referring to the scenario @bibikalka described (which is why I quoted his post just above my response). IOW, I'm not saying that stuck "nvram/cru" commands will explain all cases, only that they might explain cases such as his with daily reboots.

 

[bibikalka]

Yeah, I understand. My post was directly referring to the scenario @bibikalka described (which is why I quoted his post just above my response). IOW, I'm not saying that stuck "nvram/cru" commands will explain all cases, only that they might explain cases such as his with daily reboots.

Interesting idea! I've added this to my post-mount :

cru a ScheduledReboot "5 4 * * * /sbin/service 'reboot'"
#CheckStuckProcCmds.sh
cru a stuck "*/30 * * * * /jffs/scripts/CheckStuckProcCmds.sh 2>&1 | /usr/bin/logger -t Stuck"
cru a stuck "12 4 * * * /jffs/scripts/CheckStuckProcCmds.sh 2>&1 | /usr/bin/logger -t Stuck_boot"

I will specifically run the stuck script ~7 min after a reboot, to have it unstick whatever got stuck during reboot.

 

[Viktor Jaep]

Interesting idea! I've added this to my post-mount :

cru a ScheduledReboot "5 4 * * * /sbin/service 'reboot'"
#CheckStuckProcCmds.sh
cru a stuck "*/30 * * * * /jffs/scripts/CheckStuckProcCmds.sh 2>&1 | /usr/bin/logger -t Stuck"
cru a stuck "12 4 * * * /jffs/scripts/CheckStuckProcCmds.sh 2>&1 | /usr/bin/logger -t Stuck_boot"

I will specifically run the stuck script ~7 min after a reboot, to have it unstick whatever got stuck during reboot.

It's sad we have to worry about this in general. I'm going to have to start including @Martinski's script as a dependency upon install to provide system stability in addition to the timeout command. Ugh.

 

[bibikalka]

It's sad we have to worry about this in general. I'm going to have to start including @Martinski's script as a dependency upon install to provide system stability in addition to the timeout command. Ugh.

It sounds like this issue hits mostly ARMv8 + AES devices for sure, or maybe only the older HND (BCM490x), and not the later ones? If it's a dead branch (HND), the problem is probably never getting addressed ...

 

[Martinski]

cru a ScheduledReboot "5 4 * * * /sbin/service 'reboot'"
#CheckStuckProcCmds.sh
cru a stuck "*/30 * * * * /jffs/scripts/CheckStuckProcCmds.sh 2>&1 | /usr/bin/logger -t Stuck"
cru a stuck "12 4 * * * /jffs/scripts/CheckStuckProcCmds.sh 2>&1 | /usr/bin/logger -t Stuck_boot"

BTW, as you did before with the Backupmon cron jobs, make sure to rename the 2nd "stuck" cron job ID (e.g. "stuck_boot"); otherwise, it will overwrite the 1st one when the script is executed.

 

[bibikalka]

BTW, as you did before with the Backupmon cron jobs, make sure to rename the 2nd "stuck" cron job ID (e.g. "stuck_boot"); otherwise, it will overwrite the 1st one when the script is executed.

Of course you are correct

That was actually an early thought - I now just run the script right before I schedule the cron jobs:

admin@RT-AC86U-9988:/jffs/scripts# cat services-start
#!/bin/sh
/jffs/scripts/scmerlin startup & # scMerlin
/jffs/addons/amtm/shellhistory # Added by amtm
/jffs/scripts/CheckStuckProcCmds.sh 2>&1 | /usr/bin/logger -t Stuck_boot2
cru a root_servers  "12 4 * * * curl -o \/opt\/var\/lib\/unbound\/root\.hints https://www.internic.net/domain/named.cache"      # unbound_manager
cru a RunBackupMon "46 14 * * * sh /jffs/scripts/backupmon.sh"

Of course, I also keep the script scheduled too:

cru a stuck "*/3 * * * * /jffs/scripts/CheckStuckProcCmds.sh 2>&1 | /usr/bin/logger -t Stuck"

Now, since I have you online, here are some more details on running your script - link . Interesting stuff!

 

[TonyK132]

I just started using this script, and also saw occasional cases when the backup did not run. But I must say I also saw this when I was running the older nsru backup script. I really never worried about it since it usually ran fine the next day. But that could be a problem if you only did weekly or monthly backups.

On a separate note, is it possible for additional sub-dirs to get backed-up when this script runs, maybe from a user-controlled file that contains sub-dir names? I run ntpMerlin, YazDHCP, Unbound, vnStat, and of course, BackUpMon, and would like to preserve the settings and historical data for them all when this script runs. I previous ran my own script to save those locations then ran nsru to backup everything else, but it would be much more convenient if you provided a mechanism to do that. Even better, as a wish-list item, in additional to a user file that contains sub-dir names, provide a selector to select amtm scripts and auto back their locations (each one seems to store their settings and data in different locations). But this idea might be too off-the-wall to consider. But the user file of sub-dirs would help to preserve historical data and load settings in case of a router swap.

 

[Viktor Jaep]

I just started using this script, and also saw occasional cases when the backup did not run. But I must say I also saw this when I was running the older nsru backup script. I really never worried about it since it usually ran fine the next day. But that could be a problem if you only did weekly or monthly backups.

On a separate note, is it possible for additional sub-dirs to get backed-up when this script runs, maybe from a user-controlled file that contains sub-dir names? I run ntpMerlin, YazDHCP, Unbound, vnStat, and of course, BackUpMon, and would like to preserve the settings and historical data for them all when this script runs. I previous ran my own script to save those locations then ran nsru to backup everything else, but it would be much more convenient if you provided a mechanism to do that. Even better, as a wish-list item, in additional to a user file that contains sub-dir names, provide a selector to select amtm scripts and auto back their locations (each one seems to store their settings and data in different locations). But this idea might be too off-the-wall to consider. But the user file of sub-dirs would help to preserve historical data and load settings in case of a router swap.

Just for understanding, in addition to the External Drive and JFFS folder structures, you are looking for singular folders for specific scripts, like YazDHCP? So a compressed folder that just contains this script and it's settings folders and whatnot? How did nsru do this? Do you have any examples you might be able to share to understand these requirements a bit more please?