RT-AX89X suddenly started using a lot more CPU and memory leak leads to lockup every ~10mins

[ColinTaylor]

So far there's nothing to suggest this is an exploit. So far everything points to a bad overnight update to asd.

 

[Fiberton]

So far there's nothing to suggest this is an exploit. So far everything points to a bad overnight update to asd.

My router did not have an update. A lot of people did not get an update.

 

[ColinTaylor]

My router did not have an update. A lot of people did not get an update.

Really, I didn't see anybody discussing that (I'm not talking about a firmware update).

 

[jstrellner]

I have been fighting this all day with my RT-AX89X, and only a few minutes ago did it occur to me to check the internet for others that may be having the same issues. Same exact symptoms as OP, reboot and 10-15 minutes later it crashes again. Did a full factory restore checking the box to wipe everything. Downloaded latest firmware and uploaded it and restored again... always the same thing. I also have a RP-AX56 that similarly stops functioning.

I enabled SSH so can see via TOP it happening every time... there's a huge CPU spike before everything stops working. The ssh connection drops, wifi drops, and all ethernet drops/stops working. Oddly, the lights continue to function like everything is working normally.

I'm going to try the beta firmware now.

 

[bav5]

I had the same problem.
asd keeps writing [chknvram_action] Invalid string into /jffs/asd.log

My temporary solution:
rm -v /jffs/asd/* ; killall asd

The files in /jffs/asd will still be downloaded back after a reboot.

 

[jstrellner]

I had the same problem.
asd keeps writing [chknvram_action] Invalid string into /jffs/asd.log

My temporary solution:
rm -v /jffs/asd/* ; killall asd

The files in /jffs/asd will still be downloaded back after a reboot.

That makes sense. I noticed that /jffs was 100% full. Now after installing the beta firmware, it's sitting at 1% and seemingly not increasing.

It's still up in the air that the beta fully fixed it, but so far it seems to be the solution.

 

[ColinTaylor]

I had the same problem.
asd keeps writing [chknvram_action] Invalid string into /jffs/asd.log

My temporary solution:
rm -v /jffs/asd/* ; killall asd

The files in /jffs/asd will still be downloaded back after a reboot.

If it's the size of the log file that's the problem then the solution would be to delete that instead:

killall asd; rm /jffs/asd.log

Or do you think it's the contents of the /jffs/asd directory that's bad and need to be replaced to fix the problem?

 

[bav5]

If it's the size of the log file that's the problem then the solution would be to delete that instead:

killall asd; rm /jffs/asd.log

Or do you think it's the contents of the /jffs/asd directory that's bad and need to be replaced to fix the problem?

There are two log files /jffs/asd.log and /jffs/asd.log.1, they have a maximum log size, so won't grow unlimited.
I noticed some files in /jffs/asd/ are recently created, after when my router start memory leaks.

 

[ColinTaylor]

There are two log files /jffs/asd.log and /jffs/asd.log.1, they have a maximum log size, so won't grow unlimited.

Yes, I see the same on my router. They're only 100kB.

I noticed some files in /jffs/asd/ are recently created, after when my router start memory leaks.

Those files are also quite small on my router (running a completely different firmware than you):

-rw-rw-rw-    1 admin    root        2.1K May 12 02:25 blockfile20230510
-rw-rw-rw-    1 admin    root         448 Jan  2  2022 chknvram20201015
-rw-rw-rw-    1 admin    root         384 May 12 02:25 version

Again, perhaps because my firmware is running a very old version of asd, when I kill asd it immediately restarts itself (as can be seen in the log file):

1684371928[register_feature] blockfile registered
1684371928[register_feature] chknvram registered
1684371928[register_feature] misc registered
1684371928[init] Load existing signature files.
1684371928[update_signature_in_feature] Update sig in feature(blockfile)
1684371928[update_signature_in_feature] Update sig in feature(chknvram)

 

[bav5]

Those files are also quite small on my router (running a completely different firmware than you):

-rw-rw-rw- 1 admin root 2.1K May 12 02:25 blockfile20230510
-rw-rw-rw- 1 admin root 448 Jan 2 2022 chknvram20201015
-rw-rw-rw- 1 admin root 384 May 12 02:25 version

I have chknvram20230516 instead of chknvram20201015 in my router. I think that's the file cause the problem.

Again, perhaps because my firmware is running a very old version of asd, when I kill asd it immediately restarts itself (as can be seen in the log file):
Code:

Yes, asd will restart itself after I kill it.

 

[ColinTaylor]

I have chknvram20230516 instead of chknvram20201015 in my router. I think that's the file cause the problem.

Seems likely. It would also explain why everyone started to have this problem around the same time.

 

[ENGER]

I managed to get through to ASUS tech support. Here is a cut/paste of what they said is the problem:
"Yes Enger it was due to signature version/security update that has been provided automatically to the routers which i believe is due to FCC problem that was accused by different competitor, Apparently all router has been affected by this issue."
Their instruction to me was to downgrade to an older version of firmware. The link he gave me resulted in the following file name being download. I have not installed it yet. (just got off chat, had to wait more than an hour to get through!)

 

[smartbits]

I managed to get through to ASUS tech support. Here is a cut/paste of what they said is the problem:
"Yes Enger it was due to signature version/security update that has been provided automatically to the routers which i believe is due to FCC problem that was accused by different competitor, Apparently all router has been affected by this issue."
Their instruction to me was to downgrade to an older version of firmware. The link he gave me resulted in the following file name being download. I have not installed it yet. (just got off chat, had to wait more than an hour to get through!)

Which firmware version number have you downgraded to?

 

[Mistro]

Long time lurker and registered just to comment on this issue.

I've been using Asus networking products as long as I can remember, and the fact that they can remotely push an update to my router, even thought I declined all the possible update options and info sharing options, just doesn't sit right with me at all.

I'm seriously considering switching products now.

Yes, asd will restart itself after I kill it.

Instead of hard killing it, try adding flag -SIGSTOP to the kill command. That will just pause it vs killing it, so it shouldn't restart until your router reboots or similar.

 

[ENGER]

Which firmware version number have you downgraded to?

Sorry. They had me downgrade to 3.0.0.4.386_47191-g9ddcc31
That DOES NOT fix the problem, as best as I can tell.
Still saturating a single-core of CPU (load moves from core to core, but only seems to hit a single core at any instance). ASD is still depleting memory, and being auto-killed by some memory management routine.

 

[CBme]

That makes sense. I noticed that /jffs was 100% full. Now after installing the beta firmware, it's sitting at 1% and seemingly not increasing.

It's still up in the air that the beta fully fixed it, but so far it seems to be the solution.

View attachment 50195

Been back to normal all day (with additional feature or two) for all of us that did it earlier. CPU usage is sitting in single digits and ram usage is steady at 44%

 

[Mistro]

Sorry. They had me downgrade to 3.0.0.4.386_47191-g9ddcc31
That DOES NOT fix the problem, as best as I can tell.
Still saturating a single-core of CPU (load moves from core to core, but only seems to hit a single core at any instance). ASD is still depleting memory, and being auto-killed by some memory management routine.

Give this one a shot. It worked for me: https://www.snbforums.com/threads/asus-rt-ax89x_9-0-0-4_388_31185-beta.82276/page-6#post-841736

It's the beta though, so some things may not work - Dual WAN setup screen being one of them. Make sure you're happy with your config before flashing.
Hopefully it's just a stopgap flash for a few days, and ASUS gets their act together on this.

Part of me is worried that tonight, the router will again "phone home" and pull a bad asd file in again.

 

[CBme]

"Yes Enger it was due to signature version/security update that has been provided automatically to the routers which i believe is due to FCC problem that was accused by different competitor, Apparently all router has been affected by this issue."

That sounds a lot like a first line person who heard something from someone who heard something from someone who made something up based on partial information they heard... ;-) Can't find any "fcc problem accused by different competitor" (as opposed to 'the same' competitor?) about Asus within the past 7 years. Weird that they would push out some undefined, unannounced security update, at the same time to everyone. This would also be the first I'm hearing of them pushing anything that is not part of a firmware update. Wasn't aware they have a mechanism to do that.

 

[L&LD]

Is any of this possibly linked to this (below)?

Malware turns home routers into proxies for Chinese state-sponsored hackers

Following in the footsteps of VPNFilter, new firmware obscures hackers’ endpoints.

arstechnica.com

 

[mpsharp]

FWIW: It hit both my AC5300 and my AC68U running in AP mode. asd sucking up all memory and crashing the system. Discovered Merlin tonight and reflashed with the latest release builds. Looking good for now.

BTW: What is asd supposed to do?