Crontab jobs disappeared.

[alan6854321]

Something strange just happened, all but one cronjobs disappeared from my router.
There's normally six running on my router as follows.

cru l
15 3 * * * /jffs/scripts/regular-email #AGTxEM#
3 4 * * * wl -i eth5 deauthenticate 8C:85:80:54:82:28 #AGTxEU#
*/5 * * * * /jffs/scripts/send-cputemp-mqtt #AGTxMQ#
20 3 * * * /jffs/AGT/BackupToNAS.sh #AGTxBU#
*/2 * * * * /etc/openvpn/server1/vpn-watchdog1.sh #CheckVPNServer1#
12 14 */7 * * service restart_letsencrypt #LetsEncrypt#

All except the "CheckVPNServer1" job disappeared.

I know pretty much when it happened as I have a Raspi plotting data from the send-cputemp-mqtt job and it complains when it misses data. There's nothing obvious in the syslog around that time to suggest anything unusual happened. They all came back fine on a reboot.

Any ideas what might have happened?
Or what to investigate if they disappear again?

 

[Viktor Jaep]

There have been multiple instances lately I can think of where people have noted that their cron jobs have disappeared. Some folks in the backupmon thread noticed as well. I have about a 5 day deadzone where my backup cron jobs weren't firing off, and then they started back up. I don't think anyone's been able to put their finger on this yet. ;(

 

[SomeWhereOverTheRainBow]

There have been multiple instances lately I can think of where people have noted that their cron jobs have disappeared. Some folks in the backupmon thread noticed as well. I have about a 5 day deadzone where my backup cron jobs weren't firing off, and then they started back up. I don't think anyone's been able to put their finger on this yet. ;(

It appears to be more than one scheduled job disappearing in each instance. I don't think this is specifically tied to backupmon.

For any users reporting this same issue, read what I said here. https://www.snbforums.com/threads/b...drive-now-available-in-amtm.86645/post-890241

It may help us understand what is going on. I know the crontab can be abit unstable if a script is bombarding it, but I have not seen any developers scripts doing that lately. But it wouldn't hurt developers here to double check to ensure they are not nuking the crontab with cru commands everytime their script is called. Another possible scenario is that users routers are not completely Rebooting on scheduled reboots. There have been reports lately of routers not "fully" completing the reboot process. This might leave the users router with some cronjobs missing while others are still there. Not every developer removes the cron schedules upon their scripts departure from service where as some developers do. I can provide conjecture all day on this matter, but it wouldn't hurt to create a script that debugs crontab checking it every 15 minutes or so and recording the amount of jobs to a file by appending to it. Obviously, the script should record a time stamp as well. This will give the user some kind of idea exactly "when" the count of jobs in crontab is changing.

 

[SomeWhereOverTheRainBow]

@Viktor Jaep You should read my last edit to the previous post. Hopefully one of you explorative genius are able to debug this. I would but I am out of town for work.

 

[Viktor Jaep]

but it wouldn't hurt to create a script that debugs crontab checking it every 15 minutes or so and recording the amount of jobs to a file by appending to it. Obviously, the script should record a time stamp as well. This will give the user some kind of idea exactly "when" the count of jobs in crontab is changing.

This sounds like something I can work on... would you like to collaborate with me on this, as we did with the Skynet filter list checker? I can get started in the AM! Get ready for...

CRONMON!

 

[ColinTaylor]

cat /var/spool/cron/crontabs/$(nvram get http_username) | wc -l

 

[SomeWhereOverTheRainBow]

cat /var/spool/cron/crontabs/$(nvram get http_username) | wc -l

You can do the same with cru l | wc -l without invoking nvram.

 

[SomeWhereOverTheRainBow]

This sounds like something I can work on... would you like to collaborate with me on this, as we did with the Skynet filter list checker? I can get started in the AM! Get ready for...

CRONMON!

Lol you can use your CRONITOR to keep track of things from the ssh terminal.

 

[dave14305]

I don’t really understand the locking mechanism used by cru. There’s nothing obvious preventing the deletion of the lock file.

asuswrt-merlin.ng/release/src/router/others/cru at master · RMerl/asuswrt-merlin.ng

Third party firmware for Asus routers (newer codebase) - RMerl/asuswrt-merlin.ng

github.com

 

[SomeWhereOverTheRainBow]

I don’t really understand the locking mechanism used by cru. There’s nothing obvious preventing the deletion of the lock file.

asuswrt-merlin.ng/release/src/router/others/cru at master · RMerl/asuswrt-merlin.ng

Third party firmware for Asus routers (newer codebase) - RMerl/asuswrt-merlin.ng

github.com

One thing that is noteworthy is cru hasn't change in 7 years.

 

[dave14305]

All jobs could also disappear if someone botches the cru command and passes an asterisk as the second parameter. Just one example.

cru a * 1 * * * echo hi

 

[Ripshod]

To be honest I'm torn between CRONMON being 'a good idea' or 'overkill'. While I like the idea of an email when there's a change, I'd much rather see a fix.

 

[SomeWhereOverTheRainBow]

To be honest I'm torn between CRONMON being 'a good idea' or 'overkill'. While I like the idea of an email when there's a change, I'd much rather see a fix.

While I would love to think the issue is with crontab, I think it is a bigger issue within the reporting users router environment like I previously said. The cronmon was just an idea to help users on this path to discovering the actual problem, as opposed to being an actual fix. You cannot fix a problem indefinitely if you don't know what the cause of the problem is. For all you know is you may have identified a symptom of a bigger underlying issue.

 

[alan6854321]

I guess I already have something like CRONMON!

The "send-cputemp-mqtt" cronjob I have sends the cpu temp to my MQTT broker every 5 mins.

I have Node-red running on a Raspi which tracks it and, if it misses two messages in a row, it sends me and email. That's how I knew about it pretty much right way.

I looked through syslog but couldn't see anything out of place (I have a remote syslog server, so it gets all messages including info and debug levels).

Could the "cru" command be modified to syslog each time a crontab job is added or deleted? (Or is that a job for @RMerlin )

 

[glens]

Alias cru to call whatever /plus/ logger.

 

[Ripshod]

I've absorbed what's been said/suggested in this and other threads and decided to bite the bullet with a rewind back to 3004.388.5_0 (purely as a test). We'll see what happens.

 

[Martineau]

Could the "cru" command be modified to syslog each time a crontab job is added or deleted? (Or is that a job for @RMerlin )

So back in 2016/2017 I posted a crude script to try and track nominated command executions - although not all commands can be successfully tracked.

Join to Wireless network with CLI

Hello Guys! I'm working on an RT-N66U for a while and try to join to a wireless network which has encryption but could not succeed. If I try to join to an open wifi network it works after the following commands: udhcpc -i eth1 -s /tmp/udhcpc -O33 -O249 wl ap 0 wlconf eth1 up wl -i eth1 join...

www.snbforums.com

So changing line 32 to

cru)

allows tracking cru commands to syslog

e.g.

admin@RT-AX86U_Pro:/jffs/scripts# ./Trace_cmd.sh cru

(Trace_cmd.sh): 5339 Fake '/tmp/cru' wrapper script enabled - will write 'cru' ('/usr/sbin') invocation trace messages to Syslog.....

admin@RT-AX86U_Pro:/jffs/scripts# cru l

**TRACE: cmd request--> cru l

5 0 * * * /opt/sbin/logrotate /opt/etc/logrotate.conf >> /opt/tmp/logrotate.daily 2>&1 #logrotate#
00 2 * * Tue /bin/sh /opt/share/diversion/file/update-bl.div reset #Diversion_UpdateBL#

This may assist in identifying the rogue cru deletion command

 

[ColinTaylor]

You can do the same with cru l | wc -l without invoking nvram.

cru invokes nvram. cru is just a shell script.

 

[Ripshod]

So back in 2016/2017 I posted a crude script to try and track nominated command executions - although not all commands can be successfully tracked.

Join to Wireless network with CLI

Hello Guys! I'm working on an RT-N66U for a while and try to join to a wireless network which has encryption but could not succeed. If I try to join to an open wifi network it works after the following commands: udhcpc -i eth1 -s /tmp/udhcpc -O33 -O249 wl ap 0 wlconf eth1 up wl -i eth1 join...

www.snbforums.com

So changing line 32 to

cru)

allows tracking cru commands to syslog

e.g.

admin@RT-AX86U_Pro:/jffs/scripts# ./Trace_cmd.sh cru

(Trace_cmd.sh): 5339 Fake '/tmp/cru' wrapper script enabled - will write 'cru' ('/usr/sbin') invocation trace messages to Syslog.....

admin@RT-AX86U_Pro:/jffs/scripts# cru l

**TRACE: cmd request--> cru l

5 0 * * * /opt/sbin/logrotate /opt/etc/logrotate.conf >> /opt/tmp/logrotate.daily 2>&1 #logrotate#
00 2 * * Tue /bin/sh /opt/share/diversion/file/update-bl.div reset #Diversion_UpdateBL#

View attachment 56200


This may assist in identifying the rogue cru deletion command

Works sweet

RT-AX88U spoof.logger (cru) 26890 **TRACE: cmd request--> cru l

I'll be sticking with 388.5 for now - may as well explore all angles.

*EDIT* Thinking about it could an amtm email notification be included with this? Anyone?

 

[alan6854321]

Could I just edit the

/usr/sbin/cru

script and put something like

/usr/bin/logger $0: $@

at the start of it. Would that work?