Crontab jobs disappeared.

[ColinTaylor]

It's almost like it's inserting a folder listing in the middle of the cru statement

The "*" are being globbed.

It looks like the script isn't putting the required quotes around "00 2 * * Fri" for Diversion_UpdateBL, for example. @thelonelycoder ?

 

[dave14305]

The "*" are being globbed.

It looks like the scripts isn't putting the required quotes around "* * * * *" for example. @thelonelycoder ?

I think it might be the Martineau script echoing the cru command.

 

[ColinTaylor]

I think it might be the Martineau script echoing the cru command.

Ah, right. I forgot the cru commands were going through a wrapper script. @thelonelycoder, ignore my previous post.

 

[Ripshod]

So, what to try next? Rebuild the sytem without the wrapper script and see what happens? Maybe I'll wait until it all disappears again.

 

[ColinTaylor]

So, what to try next? Rebuild the sytem without the wrapper script and see what happens? Maybe I'll wait until it all disappears again.

Verify the true contents of the crontab with this command:

cat /var/spool/cron/crontabs/$(nvram get http_username)

 

[Ripshod]

Verify the true contents of the crontab with this command:

cat /var/spool/cron/crontabs/$(nvram get http_username)

no change

ASUSWRT-Merlin RT-AX88U 3004.388.6_0 Sat Jan 20 17:35:33 UTC 2024
Gilly@ripshod:/tmp/home/root# cat /var/spool/cron/crontabs/$(nvram get http_username)
45 */6 * * * /jffs/addons/amtm/routerdate cron #amtm_RouterDate#
10 7 * * Sun /bin/sh /jffs/addons/amtm/sc_update.mod -run #amtm_ScriptsUpdateNotification#
15 1 * * * sh /jffs/scripts/backupmon.sh -backup #RunBackupMon#
*/10 * * * * /jffs/scripts/YazFi check #YazFi#
5 0 * * * /opt/sbin/logrotate /opt/etc/logrotate.conf >> /opt/tmp/logrotate.daily 2>&1 #logrotate#
45 19 */7 * * service restart_letsencrypt #LetsEncrypt#
25 3 * * * sh /jffs/scripts/firewall banmalware #Skynet_banmalware#
23 1 * * Mon sh /jffs/scripts/firewall update #Skynet_autoupdate#
0 * * * * sh /jffs/scripts/firewall save #Skynet_save#
*/10 * * * * /jffs/scripts/ntpmerlin generate #ntpMerlin#
12,27,42,57 * * * * /jffs/scripts/spdmerlin generate #spdMerlin#
0 1-23 * * * /jffs/scripts/uiDivStats generate #uiDivStats_generate#
1 0 * * * /jffs/scripts/uiDivStats trimdb #uiDivStats_trim#
* * * * * /jffs/scripts/uiDivStats querylog #uiDivStats_querylog#
4-59/5 * * * * /jffs/scripts/uiDivStats flushtodb #uiDivStats_flushtodb#
00 2 bin bootfs cifs1 cifs2 data debug dev etc home jffs lib lib64 mmc mnt opt proc rom root sbin sys sysroot tmp usr var www bin bootfs cifs1 cifs2 data debug dev etc home jffs lib lib64 mmc mnt opt proc rom root sbin sys sysroot tmp usr var www Fri /bin/sh /opt/share/diversion/file/update-bl.div reset #Diversion_UpdateBL#
20 5 bin bootfs cifs1 cifs2 data debug dev etc home jffs lib lib64 mmc mnt opt proc rom root sbin sys sysroot tmp usr var www bin bootfs cifs1 cifs2 data debug dev etc home jffs lib lib64 mmc mnt opt proc rom root sbin sys sysroot tmp usr var www bin bootfs cifs1 cifs2 data debug dev etc home jffs lib lib64 mmc mnt opt proc rom root sbin sys sysroot tmp usr var www /bin/sh /opt/share/diversion/file/rotate-logs.div #Diversion_RotateLogs#
20 17 bin bootfs cifs1 cifs2 data debug dev etc home jffs lib lib64 mmc mnt opt proc rom root sbin sys sysroot tmp usr var www bin bootfs cifs1 cifs2 data debug dev etc home jffs lib lib64 mmc mnt opt proc rom root sbin sys sysroot tmp usr var www bin bootfs cifs1 cifs2 data debug dev etc home jffs lib lib64 mmc mnt opt proc rom root sbin sys sysroot tmp usr var www diversion count_ads count #Diversion_CountAds#
30 1 bin bootfs cifs1 cifs2 data debug dev etc home jffs lib lib64 mmc mnt opt proc rom root sbin sys sysroot tmp usr var www bin bootfs cifs1 cifs2 data debug dev etc home jffs lib lib64 mmc mnt opt proc rom root sbin sys sysroot tmp usr var www Fri /bin/sh /opt/share/diversion/file/stats.div #Diversion_WeeklyStats#

I'm also seeing this in scMerlin:

I know it's probably obvious to most, but it's only the Diversion cron jobs.

 

[alan6854321]

Can you tell from the wrapper script output what process called the "cru d ... " ?

 

[Ripshod]

Can you tell from the wrapper script output what process called the "cru d ... " ?

Other than what could possibly be a PID (long dead now anyway) not really. I was wondering if the script could be adapted in any way. Any takers?

*EDIT* No reference to PID that I can see in the script.

 

[dave14305]

Uninstall the wrapper. It’s mangling the cru a commands. Do no harm…

 

[Ripshod]

Uninstall the wrapper. It’s mangling the cru a commands. Do no harm…

Done, and fixed -apart from the still missing Skynet_genstats

cat /var/spool/cron/crontabs/$(nvram get http_username)
45 */6 * * * /jffs/addons/amtm/routerdate cron #amtm_RouterDate#
10 7 * * Sun /bin/sh /jffs/addons/amtm/sc_update.mod -run #amtm_ScriptsUpdateNotification#
15 1 * * * sh /jffs/scripts/backupmon.sh -backup #RunBackupMon#
*/10 * * * * /jffs/scripts/YazFi check #YazFi#
5 0 * * * /opt/sbin/logrotate /opt/etc/logrotate.conf >> /opt/tmp/logrotate.daily 2>&1 #logrotate#
*/10 * * * * /jffs/scripts/ntpmerlin generate #ntpMerlin#
25 11 * * * sh /jffs/scripts/firewall banmalware #Skynet_banmalware#
23 1 * * Mon sh /jffs/scripts/firewall update #Skynet_autoupdate#
12,27,42,57 * * * * /jffs/scripts/spdmerlin generate #spdMerlin#
0 * * * * sh /jffs/scripts/firewall save #Skynet_save#
55 1 */7 * * service restart_letsencrypt #LetsEncrypt#
0 1-23 * * * /jffs/scripts/uiDivStats generate #uiDivStats_generate#
1 0 * * * /jffs/scripts/uiDivStats trimdb #uiDivStats_trim#
* * * * * /jffs/scripts/uiDivStats querylog #uiDivStats_querylog#
4-59/5 * * * * /jffs/scripts/uiDivStats flushtodb #uiDivStats_flushtodb#
00 2 * * Fri /bin/sh /opt/share/diversion/file/update-bl.div reset #Diversion_UpdateBL#
20 5 * * * /bin/sh /opt/share/diversion/file/rotate-logs.div #Diversion_RotateLogs#
20 17 * * * diversion count_ads count #Diversion_CountAds#
30 1 * * Fri /bin/sh /opt/share/diversion/file/stats.div #Diversion_WeeklyStats#

I think at this point I should start from scratch. I don't know what state my router's in.

 

[Viktor Jaep]

Other than what could possibly be a PID (long dead now anyway) not really. I was wondering if the script could be adapted in any way. Any takers?

*EDIT* No reference to PID that I can see in the script.

I wonder if @Martinski would like to undertake this with his script? He's a pro when it comes to this stuff! Other than a wrapper... is there any other way to intercept cru commands in order to determine who or what is manipulating them? It would be great to be able to have something that could determine "Script/Program X added/deleted Y cru statement on Z date/time"

 

[thelonelycoder]

I wonder if @Martinski would like to undertake this with his script? He's a pro when it comes to this stuff! Other than a wrapper... is there any other way to intercept cru commands in order to determine who or what is manipulating them? It would be great to be able to have something that could determine "Script/Program X added/deleted Y cru statement on Z date/time"

cru is a Merlin command, somewhere there’s a shell command that simplifies this for us. One could add log function to that.

 

[Ripshod]

Luckily I made a backup just before I started making major changes. Diversion_LocalBackup and Skynet_genstats are both back where they should be.

 

[thelonelycoder]

Luckily I made a backup just before I started making major changes. Diversion_LocalBackup and Skynet_genstats are both back where they should be.

I just checked all cru a and d commands in amtm and Diversion. They all are correctly formatted.

 

[Ripshod]

I just checked all cru a and d commands in amtm and Diversion. They all are correctly formatted.

You called in at the right time. Is Diversion_LocalBackup supposed to be permanent? Asking because it's already disappeared.
Going back to my original list it wasn't there either, so it seems to disappear after only a short time. The wrapper showed this happening so I assume it should be there but something is killing it.

 

[thelonelycoder]

You called in at the right time. Is Diversion_LocalBackup supposed to be permanent? Asking because it's already disappeared.
Going back to my original list it wasn't there either, so it seems to disappear after only a short time. The wrapper showed this happening so I assume it should be there but something is killing it.

This is added permanently if the feature is activated.
All cron jobs in Diversion are either deleted or re-created in one file, post-conf.div. This file gets triggered every time when Dnsmasq is restarted. This has been technically the same since Diversion was first released on August 25, 2018.

I'm sure @dave14305 is on the case to find a hack to add logging in that cru shell script Merlin comes shipped with.

 

[Jeffrey Young]

I'm sure @dave14305 is on the case to find a hack to add logging in that cru shell script Merlin comes shipped with.

Ha! I was going to suggest maybe making some mods to the current cru wrapper (maybe using a bind mount to overwrite the current script) with some logging in order to get to the bottom of things. Assuming we are thinking a bad cru call is doing this.

The Lonelycoder beat me to the idea.

 

[dave14305]

I'm sure @dave14305 is on the case to find a hack to add logging in that cru shell script Merlin comes shipped with.

I would like to work on these improvements to cru:

1. Proper locking to prevent concurrent updates.
2. Call busybox crontab command within cru.
3. Add optional logging if debug mode active.

cru hasn’t been modified in a meaningful way since July 2014. We must tread lightly…

 

[dave14305]

Only for those experiencing the problems:

curl https://raw.githubusercontent.com/RMerl/asuswrt-merlin.ng/master/release/src/router/others/cru -o /jffs/addons/cru
chmod 755 /jffs/addons/cru
mount -o bind /jffs/addons/cru /usr/sbin/cru

After a while, when you're sure jobs are adding and deleting successfully, add the same mount line to /jffs/scripts/init-start.
Here's my own output after a reboot. The command line of the parent process is condensed by Linux, but still readable:

Spoiler

Feb  6 20:58:01 cru[3339]: Action: d, JobID: flexqos_5min, Args: d flexqos_5min
Feb  6 20:58:01 cru[3339]: Calling PID: 3106, CMD: sh/jffs/addons/flexqos/flexqos.sh-start
Feb  6 20:58:01 cru[3339]: Jobs before: 1, after: 1
Feb  6 20:58:03 cru[3444]: Action: d, JobID: WireGuard, Args: d WireGuard
Feb  6 20:58:03 cru[3444]: Calling PID: 3087, CMD: /tmp/udhcpc_wanbound
Feb  6 20:58:03 cru[3444]: Jobs before: 1, after: 1
Feb  6 20:59:09 cru[3687]: Action: d, JobID: routerbackup, Args: d routerbackup
Feb  6 20:59:09 cru[3687]: Calling PID: 3667, CMD: /bin/sh/jffs/scripts/post-mount/tmp/mnt/apps
Feb  6 20:59:09 cru[3687]: Jobs before: 1, after: 1
Feb  6 20:59:09 cru[3712]: Action: a, JobID: routerbackup, Args: a routerbackup 45 3 * * Fri sh /jffs/local/routerbackup.sh
Feb  6 20:59:09 cru[3712]: Calling PID: 3667, CMD: /bin/sh/jffs/scripts/post-mount/tmp/mnt/apps
Feb  6 20:59:09 cru[3712]: Jobs before: 1, after: 2
Feb  6 20:59:10 cru[3908]: Action: d, JobID: Diversion_RotateLogs, Args: d Diversion_RotateLogs
Feb  6 20:59:10 cru[3908]: Calling PID: 3714, CMD: /bin/sh/jffs/scripts/dnsmasq.postconf/etc/dnsmasq.conf
Feb  6 20:59:10 cru[3908]: Jobs before: 2, after: 2
Feb  6 20:59:10 cru[3932]: Action: d, JobID: Diversion_CountAds, Args: d Diversion_CountAds
Feb  6 20:59:10 cru[3932]: Calling PID: 3714, CMD: /bin/sh/jffs/scripts/dnsmasq.postconf/etc/dnsmasq.conf
Feb  6 20:59:10 cru[3932]: Jobs before: 2, after: 2
Feb  6 20:59:10 cru[3956]: Action: d, JobID: Diversion_WeeklyStats, Args: d Diversion_WeeklyStats
Feb  6 20:59:10 cru[3956]: Calling PID: 3714, CMD: /bin/sh/jffs/scripts/dnsmasq.postconf/etc/dnsmasq.conf
Feb  6 20:59:10 cru[3956]: Jobs before: 2, after: 2
Feb  6 20:59:10 cru[3983]: Action: d, JobID: Diversion_LocalBackup, Args: d Diversion_LocalBackup
Feb  6 20:59:10 cru[3983]: Calling PID: 3714, CMD: /bin/sh/jffs/scripts/dnsmasq.postconf/etc/dnsmasq.conf
Feb  6 20:59:10 cru[3983]: Jobs before: 2, after: 2
Feb  6 20:59:10 cru[4011]: Action: a, JobID: Diversion_UpdateBL, Args: a Diversion_UpdateBL 00 4 * * Mon /bin/sh /opt/share/diversion/file/update-bl.div reset
Feb  6 20:59:10 cru[4011]: Calling PID: 3714, CMD: /bin/sh/jffs/scripts/dnsmasq.postconf/etc/dnsmasq.conf
Feb  6 20:59:10 cru[4011]: Jobs before: 2, after: 3
Feb  6 20:59:10 cru[4043]: Action: a, JobID: Diversion_UpdateBLTwo, Args: a Diversion_UpdateBLTwo 00 4 * * Thu /bin/sh /opt/share/diversion/file/update-bl.div
Feb  6 20:59:10 cru[4043]: Calling PID: 3714, CMD: /bin/sh/jffs/scripts/dnsmasq.postconf/etc/dnsmasq.conf
Feb  6 20:59:10 cru[4043]: Jobs before: 3, after: 4
Feb  6 20:59:18 cru[5033]: Action: d, JobID: flexqos_5min, Args: d flexqos_5min
Feb  6 20:59:18 cru[5033]: Calling PID: 4800, CMD: sh/jffs/addons/flexqos/flexqos.sh-start
Feb  6 20:59:18 cru[5033]: Jobs before: 4, after: 4
Feb  6 20:59:19 cru[6098]: Action: d, JobID: Skynet_save, Args: d Skynet_save
Feb  6 20:59:19 cru[6098]: Calling PID: 3107, CMD: sh/jffs/scripts/firewallstartskynetloc=/tmp/mnt/apps/skynet
Feb  6 20:59:19 cru[6098]: Jobs before: 4, after: 4
Feb  6 20:59:19 cru[6150]: Action: d, JobID: Skynet_banmalware, Args: d Skynet_banmalware
Feb  6 20:59:19 cru[6150]: Calling PID: 3107, CMD: sh/jffs/scripts/firewallstartskynetloc=/tmp/mnt/apps/skynet
Feb  6 20:59:19 cru[6150]: Jobs before: 4, after: 4
Feb  6 20:59:19 cru[6200]: Action: d, JobID: Skynet_autoupdate, Args: d Skynet_autoupdate
Feb  6 20:59:19 cru[6200]: Calling PID: 3107, CMD: sh/jffs/scripts/firewallstartskynetloc=/tmp/mnt/apps/skynet
Feb  6 20:59:19 cru[6200]: Jobs before: 4, after: 4
Feb  6 20:59:19 cru[6229]: Action: a, JobID: flexqos_5min, Args: a flexqos_5min 04 21 06 02 Tue /jffs/addons/flexqos/flexqos.sh -check
Feb  6 20:59:19 cru[6229]: Calling PID: 4800, CMD: sh/jffs/addons/flexqos/flexqos.sh-start
Feb  6 20:59:19 cru[6242]: Action: d, JobID: Skynet_checkupdate, Args: d Skynet_checkupdate
Feb  6 20:59:19 cru[6242]: Calling PID: 3107, CMD: sh/jffs/scripts/firewallstartskynetloc=/tmp/mnt/apps/skynet
Feb  6 20:59:19 cru[6229]: Jobs before: 4, after: 5
Feb  6 20:59:19 cru[6242]: Jobs before: 5, after: 5
Feb  6 20:59:19 cru[6286]: Action: d, JobID: Skynet_genstats, Args: d Skynet_genstats
Feb  6 20:59:19 cru[6286]: Calling PID: 3107, CMD: sh/jffs/scripts/firewallstartskynetloc=/tmp/mnt/apps/skynet
Feb  6 20:59:19 cru[6286]: Jobs before: 5, after: 5
Feb  6 20:59:19 cru[6340]: Action: a, JobID: Skynet_banmalware, Args: a Skynet_banmalware 25 7 * * * sh /jffs/scripts/firewall banmalware
Feb  6 20:59:19 cru[6340]: Calling PID: 3107, CMD: sh/jffs/scripts/firewallstartskynetloc=/tmp/mnt/apps/skynet
Feb  6 20:59:19 cru[6340]: Jobs before: 5, after: 6
Feb  6 20:59:19 cru[6372]: Action: a, JobID: Skynet_autoupdate, Args: a Skynet_autoupdate 18 1 * * Mon sh /jffs/scripts/firewall update
Feb  6 20:59:19 cru[6372]: Calling PID: 3107, CMD: sh/jffs/scripts/firewallstartskynetloc=/tmp/mnt/apps/skynet
Feb  6 20:59:19 cru[6372]: Jobs before: 6, after: 7
Feb  6 20:59:19 cru[6447]: Action: a, JobID: Skynet_save, Args: a Skynet_save 0 * * * * sh /jffs/scripts/firewall save
Feb  6 20:59:19 cru[6447]: Calling PID: 3107, CMD: sh/jffs/scripts/firewallstartskynetloc=/tmp/mnt/apps/skynet
Feb  6 20:59:19 cru[6447]: Jobs before: 7, after: 8
Feb  6 20:59:25 cru[7080]: Action: d, JobID: Diversion_RotateLogs, Args: d Diversion_RotateLogs
Feb  6 20:59:25 cru[7080]: Calling PID: 6971, CMD: /bin/sh/jffs/scripts/dnsmasq.postconf/etc/dnsmasq.conf
Feb  6 20:59:25 cru[7080]: Jobs before: 8, after: 8
Feb  6 20:59:25 cru[7154]: Action: d, JobID: Diversion_CountAds, Args: d Diversion_CountAds
Feb  6 20:59:25 cru[7154]: Calling PID: 6971, CMD: /bin/sh/jffs/scripts/dnsmasq.postconf/etc/dnsmasq.conf
Feb  6 20:59:25 cru[7154]: Jobs before: 8, after: 8
Feb  6 20:59:25 cru[7216]: Action: d, JobID: Diversion_WeeklyStats, Args: d Diversion_WeeklyStats
Feb  6 20:59:25 cru[7216]: Calling PID: 6971, CMD: /bin/sh/jffs/scripts/dnsmasq.postconf/etc/dnsmasq.conf
Feb  6 20:59:25 cru[7216]: Jobs before: 8, after: 8
Feb  6 20:59:25 cru[7292]: Action: d, JobID: Diversion_LocalBackup, Args: d Diversion_LocalBackup
Feb  6 20:59:25 cru[7292]: Calling PID: 6971, CMD: /bin/sh/jffs/scripts/dnsmasq.postconf/etc/dnsmasq.conf
Feb  6 20:59:25 cru[7292]: Jobs before: 8, after: 8
Feb  6 20:59:25 cru[7377]: Action: a, JobID: Diversion_UpdateBL, Args: a Diversion_UpdateBL 00 4 * * Mon /bin/sh /opt/share/diversion/file/update-bl.div reset
Feb  6 20:59:25 cru[7377]: Calling PID: 6971, CMD: /bin/sh/jffs/scripts/dnsmasq.postconf/etc/dnsmasq.conf
Feb  6 20:59:25 cru[7377]: Jobs before: 8, after: 8
Feb  6 20:59:25 cru[7467]: Action: a, JobID: Diversion_UpdateBLTwo, Args: a Diversion_UpdateBLTwo 00 4 * * Thu /bin/sh /opt/share/diversion/file/update-bl.div
Feb  6 20:59:25 cru[7467]: Calling PID: 6971, CMD: /bin/sh/jffs/scripts/dnsmasq.postconf/etc/dnsmasq.conf
Feb  6 20:59:25 cru[7467]: Jobs before: 8, after: 8
Feb  6 20:59:34 cru[8871]: Action: d, JobID: Skynet_genstats, Args: d Skynet_genstats
Feb  6 20:59:34 cru[8871]: Calling PID: 3107, CMD: sh/jffs/scripts/firewallstartskynetloc=/tmp/mnt/apps/skynet
Feb  6 20:59:34 cru[8871]: Jobs before: 8, after: 8
Feb  6 20:59:34 cru[8901]: Action: a, JobID: Skynet_genstats, Args: a Skynet_genstats 44 */12 * * * sh /jffs/scripts/firewall debug genstats
Feb  6 20:59:34 cru[8901]: Calling PID: 3107, CMD: sh/jffs/scripts/firewallstartskynetloc=/tmp/mnt/apps/skynet
Feb  6 20:59:34 cru[8901]: Jobs before: 8, after: 9

If it breaks anything, run:

umount /usr/sbin/cru
rm /jffs/addons/cru

and delete the line from init-start if you added it already.

 

[ColinTaylor]

Great work @dave14305. How about putting a logger command near the very beginning? The reason being we saw in some of the output that cru appeared to be hanging on the nvram get http_username command. Putting a logger before that might show cru commands that started and then hung.