Crontab jobs disappeared.

[Viktor Jaep]

Gilly@ripshod:/jffs/asd# ls
blockdnsip2023080803   chknvram2023080803     version
blockfile2023103001    monitorproc2023080401

Somehow methinks asd is innocent.
Yes, I also suffered the asd/SkyNet issue too.

Yeah, without updated signatures, why would it all of a sudden start blocking skynet. Yet we determined it was 100% responsible. Perhaps I'm not understanding how this ASD process works.

 

[Ripshod]

Yeah, without updated signatures, why would it all of a sudden start blocking skynet. Yet we determined it was 100% responsible. Perhaps I'm not understanding how this ASD process works.

I was one who pointed the finger. Is it possible asus have made a change to the asd engine? Would make sense they wouldn't publicise a change for security(?). The skynet script that was deleted was definitely caused by asd - possibly a false positive.

 

[dave14305]

Yeah, without updated signatures, why would it all of a sudden start blocking skynet. Yet we determined it was 100% responsible. Perhaps I'm not understanding how this ASD process works.

My signatures were updated Jan 31 05:01, even though they have the same dated filename.

Skynet could be seen as a risk since it deliberately contains strings associated with previously known malware. What took asd so long?

I’ve never seen my crontab disappear, so everyone that has experienced this ought to be sharing which jobs they have normally and look for the common factor.

 

[dave14305]

I was thinking the same thing as LetsEncrypt is common to both reports. But like you I don't use LetsEncrypt and I know nothing about it.

Is it using the acme.sh script? I note that it has a --cron option which overwrites the crontab directly rather than using cru.

I just see that the LetsEncrypt cru ID only seems to exist in libletsencrypt.so in the repo, and the start_letsencrypt() is also closed source.

I’d feel better if cru did better locking.

 

[aru]

I have never seen my crontab disappear since upgrading to 388.6 14 days ago.

My signatures version as below:

admins@GT-AX6000-0FC8:/jffs/asd# ls
blockdnsip2023080803 chknvram2023080803 version
blockfile2023103001 monitorproc2023080401

Through the script provided by dave14305, it was discovered that crontab activities were not updated in real time, because without a timeline it's unclear whether there were any recent changes.

/bin/grep -rE "(\bcru [ad]\b|crontab)" /jffs /tmp/mnt/*/

 

[Adamm]

My signatures were updated Jan 31 05:01, even though they have the same dated filename.

Skynet could be seen as a risk since it deliberately contains strings associated with previously known malware. What took asd so long?

I’ve never seen my crontab disappear, so everyone that has experienced this ought to be sharing which jobs they have normally and look for the common factor.

That was a fun surprise to wake up to, very not cool to think of how many Skynet installs will have been wiped because of it I personally wasn't affected on my two networks but there must have been some sort of trigger.

On the bright side, Skynet protected users for 949 days before Asus also added mitigations against this IOC, so that's nice to know.

One would assume the engine looks for updates every hour based on our timestamps...

skynet@GT-AXE16000-9D20:/jffs/asd# ls -la
drwxr--r-- 2 skynet root 544 Feb 5 05:40 .
drwxr-xr-x 14 skynet root 2000 Feb 5 19:15 ..
-rw-rw-rw- 1 skynet root 9772 Jan 31 22:01 blockdnsip2023080803
-rw-rw-rw- 1 skynet root 38060 Jan 31 22:01 blockfile2023103001
-rw-rw-rw- 1 skynet root 3456 Jan 31 22:01 chknvram2023080803
-rw-rw-rw- 1 skynet root 1728 Jan 31 22:01 monitorproc2023080401
-rw-rw-rw- 1 skynet root 2392 Jan 31 22:01 version

 

[Ripshod]

That was a fun surprise to wake up to, very not cool to think of how many Skynet installs will have been wiped because of it I personally wasn't affected on my two networks but there must have been some sort of trigger.

On the bright side, Skynet protected users for 949 days before Asus also added mitigations against this IOC, so that's nice to know.

One would assume the engine looks for updates every hour based on our timestamps...

Is that an older result? Using "l's - la" all mine are dated Feb 2nd (17:14)

 

[alan6854321]

Is that an older result? Using "l's - la" all mine are dated Feb 2nd (17:14)

Mine are all dated Feb 1st

Don't know if it's relevant, but at the same timestamp (1st Feb 0452am) there's a 'restart_firewall' event in the syslog.

ls -la
drwxr--r--    2 RT-AC86U root             0 Feb  5 03:59 .
drwxr-xr-x   20 RT-AC86U root             0 Feb  5 10:44 ..
-rw-rw-rw-    1 RT-AC86U root          9772 Feb  1 04:52 blockdnsip2023080803
-rw-rw-rw-    1 RT-AC86U root         38060 Feb  1 04:52 blockfile2023103001
-rw-rw-rw-    1 RT-AC86U root          3456 Feb  1 04:52 chknvram2023080803
-rw-rw-rw-    1 RT-AC86U root          1728 Feb  1 04:52 monitorproc2023080401
-rw-rw-rw-    1 RT-AC86U root          2392 Feb  1 04:52 version

 

[Ripshod]

Don't know if it's relevant, but at the same timestamp (1st Feb 0452am) there's a 'restart_firewall' event in the syslog.

Good spot. Mine shows the same firewall behaviour at the same time as the update. May just be circumstantial but it was that evening I noticed my cron jobs went AWOL.

 

[XIII]

My asd files are even older:

➜ la
drwxr--r-- admin root 544B 05-Feb-2024 09:56  .
drwxr-xr-x admin root 2.3K 05-Feb-2024 15:30  ..
.rw-rw-rw- admin root 9.5K 31-Jan-2024 21:24  blockdnsip2023080803
.rw-rw-rw- admin root  37K 31-Jan-2024 21:24  blockfile2023103001
.rw-rw-rw- admin root 3.4K 31-Jan-2024 21:24  chknvram2023080803
.rw-rw-rw- admin root 1.7K 31-Jan-2024 21:24  monitorproc2023080401
.rw-rw-rw- admin root 2.3K 31-Jan-2024 21:24  version

I still have all cron jobs. Hope I don't lose them when these files get updated...

 

[bibikalka]

OK, a data point here. Could be not very helpful but anyways. I scheduled backupmon in 2 places, in post-mount, and services-start. First goes at 4:28, second at 14:46. The router reboots itself daily at 4:05. By the backup file time I can see if the 2nd job fired off. It seems to have about 50% success rate. The 1st job from post-mount always works.

admin@RT-AC86U-9988:/# cru l |grep back
28 4 * * * sh /jffs/scripts/backupmon.sh #RunBackupMon0#
46 14 * * * sh /jffs/scripts/backupmon.sh #RunBackupMon#

admin@RT-AC86U-9988:/# ls -lta /tmp/mnt/spare/router/AC86U-Backup/*/jffs*
-rw-rw-rw-    1 admin    root       4508443 Feb  5 04:28 /tmp/mnt/spare/router/AC86U-Backup/05/jffs.tar.gz
-rw-rw-rw-    1 admin    root       4483054 Feb  4 04:28 /tmp/mnt/spare/router/AC86U-Backup/04/jffs.tar.gz
-rw-rw-rw-    1 admin    root       4540482 Feb  3 04:28 /tmp/mnt/spare/router/AC86U-Backup/03/jffs.tar.gz
-rw-rw-rw-    1 admin    root       4555296 Feb  2 04:28 /tmp/mnt/spare/router/AC86U-Backup/02/jffs.tar.gz
-rw-rw-rw-    1 admin    root       4640523 Feb  1 14:46 /tmp/mnt/spare/router/AC86U-Backup/01/jffs.tar.gz
-rw-rw-rw-    1 admin    root       4506364 Jan 31 04:28 /tmp/mnt/spare/router/AC86U-Backup/31/jffs.tar.gz
-rw-rw-rw-    1 admin    root       4516870 Jan 30 14:46 /tmp/mnt/spare/router/AC86U-Backup/30/jffs.tar.gz
-rw-rw-rw-    1 admin    root       4505106 Jan 29 14:46 /tmp/mnt/spare/router/AC86U-Backup/29/jffs.tar.gz
-rw-rw-rw-    1 admin    root       4542287 Jan 28 14:46 /tmp/mnt/spare/router/AC86U-Backup/28/jffs.tar.gz
-rw-rw-rw-    1 admin    root       4390342 Jan 27 04:28 /tmp/mnt/spare/router/AC86U-Backup/27/jffs.tar.gz
-rw-rw-rw-    1 admin    root       4477974 Jan 26 14:46 /tmp/mnt/spare/router/AC86U-Backup/26/jffs.tar.gz
-rw-rw-rw-    1 admin    root       4300909 Jan 25 04:28 /tmp/mnt/spare/router/AC86U-Backup/25/jffs.tar.gz
-rw-rw-rw-    1 admin    root       4175532 Jan 24 04:28 /tmp/mnt/spare/router/AC86U-Backup/24/jffs.tar.gz

 

[Viktor Jaep]

OK, a data point here. Could be not very helpful but anyways. I scheduled backupmon in 2 places, in post-mount, and services-start. First goes at 4:28, second at 14:46. The router reboots itself daily at 4:05. By the backup file time I can see if the 2nd job fired off. It seems to have about 50% success rate. The 1st job from post-mount always works.

admin@RT-AC86U-9988:/# cru l |grep back
28 4 * * * sh /jffs/scripts/backupmon.sh #RunBackupMon0#
46 14 * * * sh /jffs/scripts/backupmon.sh #RunBackupMon#

admin@RT-AC86U-9988:/# ls -lta /tmp/mnt/spare/router/AC86U-Backup/*/jffs*
-rw-rw-rw-    1 admin    root       4508443 Feb  5 04:28 /tmp/mnt/spare/router/AC86U-Backup/05/jffs.tar.gz
-rw-rw-rw-    1 admin    root       4483054 Feb  4 04:28 /tmp/mnt/spare/router/AC86U-Backup/04/jffs.tar.gz
-rw-rw-rw-    1 admin    root       4540482 Feb  3 04:28 /tmp/mnt/spare/router/AC86U-Backup/03/jffs.tar.gz
-rw-rw-rw-    1 admin    root       4555296 Feb  2 04:28 /tmp/mnt/spare/router/AC86U-Backup/02/jffs.tar.gz
-rw-rw-rw-    1 admin    root       4640523 Feb  1 14:46 /tmp/mnt/spare/router/AC86U-Backup/01/jffs.tar.gz
-rw-rw-rw-    1 admin    root       4506364 Jan 31 04:28 /tmp/mnt/spare/router/AC86U-Backup/31/jffs.tar.gz
-rw-rw-rw-    1 admin    root       4516870 Jan 30 14:46 /tmp/mnt/spare/router/AC86U-Backup/30/jffs.tar.gz
-rw-rw-rw-    1 admin    root       4505106 Jan 29 14:46 /tmp/mnt/spare/router/AC86U-Backup/29/jffs.tar.gz
-rw-rw-rw-    1 admin    root       4542287 Jan 28 14:46 /tmp/mnt/spare/router/AC86U-Backup/28/jffs.tar.gz
-rw-rw-rw-    1 admin    root       4390342 Jan 27 04:28 /tmp/mnt/spare/router/AC86U-Backup/27/jffs.tar.gz
-rw-rw-rw-    1 admin    root       4477974 Jan 26 14:46 /tmp/mnt/spare/router/AC86U-Backup/26/jffs.tar.gz
-rw-rw-rw-    1 admin    root       4300909 Jan 25 04:28 /tmp/mnt/spare/router/AC86U-Backup/25/jffs.tar.gz
-rw-rw-rw-    1 admin    root       4175532 Jan 24 04:28 /tmp/mnt/spare/router/AC86U-Backup/24/jffs.tar.gz

Based on this cron job schedule, shouldn't you be seeing 2 backups for each day? Almost like either one or the other works for each day, but not both. Oh never mind... it's set to monthly/basic backup. So the 14:46 backup would overwrite the 04:28 backup. These should all say 14:46 in other words.

That is an excellent data point! In fact, @bibikalka, it would be interesting to take a snapshot running "cru l" and seeing what is in your list on both days it works only at 04:28, and days with 14:46.

 

[Ripshod]

I have to ask because it's bugging me. I've gone back to 388.6 and the Trace_cmd.sh script is spamming my system log. Is it normal to see a cru l in the log every time a cron job runs?
Yes, it was a flash, reset and rebuild.

 

[bibikalka]

That is an excellent data point! In fact, @bibikalka, it would be interesting to take a snapshot running "cru l" and seeing what is in your list on both days it works only at 04:28, and days with 14:46.

Yes indeed! On some days the cron job from services-start never gets scheduled in the first place, and does not show up in "cru l". Or maybe it disappears before I have a chance to see it in the morning!

On the bright side the other issue below has gone away after I scheduled reboots using [cru a ScheduledReboot "5 4 * * * /sbin/service 'reboot'"], from here:

Scribe - /usr/bin/logger messages don't make it to the log file - intermittent issue

I am running Scribe with Entware. Sometimes my cron job messages don't make it to the "messages" file for stretches of time, then it's all fine again for a while. Here is an example of a cron job that I want to output to messages every 15 min: cru a temp "*/15 * * * * /jffs/scripts/temp_check...

www.snbforums.com

 

[Martinski]

Yes indeed! On some days the cron job from services-start never gets scheduled in the first place, and does not show up in "cru l". Or maybe it disappears before I have a chance to see it in the morning!

I'm very curious to see when the problem with "disappearing cron jobs" happens so I wrote a script to take a snapshot of the cron job list at a given frequency, specified in minutes. The script is available on GitHub if you want to try it:

mkdir -m 755 -p /jffs/scripts
curl -kLSs --retry 3 --retry-delay 5 --retry-connrefused  https://raw.githubusercontent.com/Martinski4GitHub/CustomMiscUtils/master/Cron/MonitorCronJobList.sh  -o /jffs/scripts/MonitorCronJobList.sh
chmod 755 /jffs/scripts/MonitorCronJobList.sh

After downloading the script, go to the "/jffs/scripts" directory to verify that it works for you by typing the following command:

./MonitorCronJobList.sh start 2 &

Note the ampersand character at the end of the command to run it in the background.

To stop the script, type the following command

./MonitorCronJobList.sh stop

If you have Entware installed on the USB-attached drive, the default LOG file is found here:

/opt/var/log/MonitorCronJobList.LOG

If you don't have Entware installed, the LOG file is found here:

/jffs/scripts/logs/MonitorCronJobList.LOG

You can change the defaults by modifying the directory path variables defined in the
"CUSTOMIZABLE PARAMETERS SECTION" near the top of the script file:

defltLogDirPath="/opt/var/log"
altn1LogDirPath="/jffs/scripts/logs"

After you've verified that the script works on your router, you can launch it during bootup by typing the following command in the "services-start" script:

[ -x /jffs/scripts/MonitorCronJobList.sh ] && /jffs/scripts/MonitorCronJobList.sh start MIN &

Where "MIN" is the frequency given in minutes, meaning that every "MIN" minutes the script will take a snapshot of the cron job list along with a count & a timestamp.

HTH

 

[Ripshod]

One cron job removed by something today. No sign of it being added previously or since.

Feb 06 03:25:25 RT-AX88U spoof.logger (cru) 15420 **TRACE: cmd request--> cru d Diversion_LocalBackup

I've also lost another cron job but no trace in the log - Skynet_genstats


*EDIT* All the other Diversion jobs have also been messed up. If this is a corruption of the crontab does the firmware ever check the crontab and try to repair?

45 */6 * * * /jffs/addons/amtm/routerdate cron #amtm_RouterDate#
10 7 * * Sun /bin/sh /jffs/addons/amtm/sc_update.mod -run #amtm_ScriptsUpdateNotification#
15 1 * * * sh /jffs/scripts/backupmon.sh -backup #RunBackupMon#
*/10 * * * * /jffs/scripts/YazFi check #YazFi#
5 0 * * * /opt/sbin/logrotate /opt/etc/logrotate.conf >> /opt/tmp/logrotate.daily 2>&1 #logrotate#
45 19 */7 * * service restart_letsencrypt #LetsEncrypt#
25 3 * * * sh /jffs/scripts/firewall banmalware #Skynet_banmalware#
23 1 * * Mon sh /jffs/scripts/firewall update #Skynet_autoupdate#
0 * * * * sh /jffs/scripts/firewall save #Skynet_save#
*/10 * * * * /jffs/scripts/ntpmerlin generate #ntpMerlin#
12,27,42,57 * * * * /jffs/scripts/spdmerlin generate #spdMerlin#
0 1-23 * * * /jffs/scripts/uiDivStats generate #uiDivStats_generate#
1 0 * * * /jffs/scripts/uiDivStats trimdb #uiDivStats_trim#
* * * * * /jffs/scripts/uiDivStats querylog #uiDivStats_querylog#
4-59/5 * * * * /jffs/scripts/uiDivStats flushtodb #uiDivStats_flushtodb#
00 2 bin bootfs cifs1 cifs2 data debug dev etc home jffs lib lib64 mmc mnt opt proc rom root sbin sys sysroot tmp usr var www bin bootfs cifs1 cifs2 data debug dev etc home jffs lib lib64 mmc mnt opt proc rom root sbin sys sysroot tmp usr var www Fri /bin/sh /opt/share/diversion/file/update-bl.div reset #Diversion_UpdateBL#
20 5 bin bootfs cifs1 cifs2 data debug dev etc home jffs lib lib64 mmc mnt opt proc rom root sbin sys sysroot tmp usr var www bin bootfs cifs1 cifs2 data debug dev etc home jffs lib lib64 mmc mnt opt proc rom root sbin sys sysroot tmp usr var www bin bootfs cifs1 cifs2 data debug dev etc home jffs lib lib64 mmc mnt opt proc rom root sbin sys sysroot tmp usr var www /bin/sh /opt/share/diversion/file/rotate-logs.div #Diversion_RotateLogs#
20 17 bin bootfs cifs1 cifs2 data debug dev etc home jffs lib lib64 mmc mnt opt proc rom root sbin sys sysroot tmp usr var www bin bootfs cifs1 cifs2 data debug dev etc home jffs lib lib64 mmc mnt opt proc rom root sbin sys sysroot tmp usr var www bin bootfs cifs1 cifs2 data debug dev etc home jffs lib lib64 mmc mnt opt proc rom root sbin sys sysroot tmp usr var www diversion count_ads count #Diversion_CountAds#
30 1 bin bootfs cifs1 cifs2 data debug dev etc home jffs lib lib64 mmc mnt opt proc rom root sbin sys sysroot tmp usr var www bin bootfs cifs1 cifs2 data debug dev etc home jffs lib lib64 mmc mnt opt proc rom root sbin sys sysroot tmp usr var www Fri /bin/sh /opt/share/diversion/file/stats.div #Diversion_WeeklyStats#

I have the uneasy feeling most of these will be gone by the morning!!

 

[alan6854321]

One cron job removed by something today

I've also lost another cron job but no trace in the log - Skynet_genstats

So, the file isn't getting corrupted then,
Something actually called "cru d Diversion_LocalBackup",
Have I got that correct?

 

[Ripshod]

So, the file isn't getting corrupted then,
Something actually called "cru d Diversion_LocalBackup",
Have I got that correct?

Yes, correct.

 

[Viktor Jaep]

One cron job removed by something today. No sign of it being added previously or since.


I've also lost another cron job but no trace in the log - Skynet_genstats


*EDIT* All the Diversion jobs have also been messed up. If this is a corruption of the crontab does the firmware ever check the crontab ant try to repair?

It's almost like it's inserting a folder listing in the middle of the cru statement

 

[dave14305]

It's almost like it's doing a folder listing in the middle of the cru statement

There’s a tab being interpreted at the command line that is enumerating everything in the PATH. Press tab twice at the command line to see the same effect.

No, maybe I’m wrong. But it’s definitely odd.

The * is being echoed and listing the content of the current directory.