RT-AX89X suddenly started using a lot more CPU and memory leak leads to lockup every ~10mins

[Broken Hope]

This is really not ASUS's month. It looks like they pushed a bad update to their asd router security daemon yesterday. It's causing ASUS routers (mine included) to peg a CPU core while writing out a "chknvram" error message to a JFFS log as quickly as possible

— Ryan Smith (@RyanSmithAT) May 18, 2023

 

[Jbennett360]

This is really not ASUS's month. It looks like they pushed a bad update to their asd router security daemon yesterday. It's causing ASUS routers (mine included) to peg a CPU core while writing out a "chknvram" error message to a JFFS log as quickly as possible

— Ryan Smith (@RyanSmithAT) May 18, 2023

It only seems to be affecting routers with .386 firmware though.

Can't say I've seen anyone with 388 have this issue.

 

[ViRGE]

Another long-time lurker checking in to thank the SNB crew for the info needed to make my RT-AC87U behave again.

I really wish Asus documented asd better. While I get the idea that there's a degree of security through obscurity, the fact that it doesn't show up in any official (or even semi-official) literature is problematic. Fittingly, the best source on asd is SNBForums, and the fact that the answer is there is practically happenstance.

If asd was properly documented, then it would have saved us all a lot of grief in trying to determine what was wrong, and how to fix it. As well as avoiding any speculation that this was malware. Though the irony that the interim fix is turning off the router's security daemon is not lost on me...

That sounds a lot like a first line person who heard something from someone who heard something from someone who made something up based on partial information they heard... ;-) Can't find any "fcc problem accused by different competitor" (as opposed to 'the same' competitor?) about Asus within the past 7 years. Weird that they would push out some undefined, unannounced security update, at the same time to everyone. This would also be the first I'm hearing of them pushing anything that is not part of a firmware update. Wasn't aware they have a mechanism to do that.

Agreed. That is a case of where someone on the support staff knew just enough to get themselves in trouble - or at least make wild claims.

Based on all of the evidence thus far, everything points to Asus's release of a definitions/signature file (chknvram20230516) for use by asd across all of their routers released in the last several years. Something about chknvram20230516 is malformed/corrupt, and as asd is incapable of properly handling the malformed file, it goes nuttier than an Almond Joy bar. Eventually, it consumes all of a router's available RAM, which is what finally makes the router unusable.

I'm a bit flummoxed by the fact that it has taken Asus so long to fix the issue. Based on asd.log and the [chknvram_action] Invalid string error, in retrospect this should have been obvious for their higher tier support staff an engineers. Especially as Asus wouldn't even be the first vendor to distribute a bad definitions update for their security software - Microsoft has done it a couple of times.

Perhaps the most interesting thing is the fact that using 388 beta firmwares offers a layer of mitigation against the issue. The root cause of the problem isn't the firmware itself (which is why downgrading doesn't help), but it seems that the version of asd included with 388 is capable of more gracefully handling malformed definitions? I do worry that it serves to obfuscate the root cause though; updating a router's firmware isn't truly fixing the problem. That'll come once Asus stops distributing the bad chknvram file.

 

[ENGER]

Maybe fixed?
Tech Support had me downgrade FW earlier. That did not help.
I deleted the chknvram2020516 file, then ran FW update. It found 3.0.0.4.386_47468-g73fe1fe and rebooted.
System has been up for one hour and 30 minutes. RAM seems stable (419MB used). CPU display shows cores sitting at 1% to 3% most of the time.
A new chknvram file is now in the asd directory: chknvram20230518

 

[MakeItEasy]

Hello Experts

Third reboot since this morning for my RT-AC86U using FW 3.0.0.4.386_48260 ...

I guess Asus will push a fix ASAP because I'm not expert enough to understand all the posts on this topic

 

[ColinTaylor]

I managed to get through to ASUS tech support. Here is a cut/paste of what they said is the problem:
"Yes Enger it was due to signature version/security update that has been provided automatically to the routers which i believe is due to FCC problem that was accused by different competitor, Apparently all router has been affected by this issue."
Their instruction to me was to downgrade to an older version of firmware. The link he gave me resulted in the following file name being download. I have not installed it yet. (just got off chat, had to wait more than an hour to get through!)

They're probably referring to the security issues they had a few years ago which resulted in the FTC forcing Asus to keep their customer's routers security up to date. So for all those people whining complaining about forced updates, this isn't a forced firmware update it's the equivalent of Microsoft pushing a Windows Defender update each day.

ASUS Settles FTC Charges That Insecure Home Routers and “Cloud” Services Put Consumers’ Privacy At Risk

Taiwan-based computer hardware maker ASUSTeK Computer, Inc. has agreed to settle Federal Trade Commission charges that critical secur

www.ftc.gov

There was an unrelated issue where Netgear complained to the FCC about Asus' test procedures. It sounds like tech support were conflating these two issues.

 

[guetech]

So for all those people whining about forced updates, this isn't a forced firmware update it's the equivalent of Microsoft pushing a Windows Defender update each day.

You could really do without the snark. I (and apparently many others) struggled all day to stay online because of an update that asus pushed to our devices. Does it matter if the update is a list of strings or a full firmware? Not at all, they broke our devices remotely. We're "whining" that asd can't be disabled. Yes I saw your opinion about it in other threads, we get it. Asus is our savior and will protect us from botnets.

I guess making our devices crash so bad they can't stay online is one way to do it.

 

[Jbennett360]

You could really do without the snark. I (and apparently many others) struggled all day to stay online because of an update that asus pushed to our devices. Does it matter if the update is a list of strings or a full firmware? Not at all, they broke our devices remotely. We're "whining" that asd can't be disabled. Yes I saw your opinion about it in other threads, we get it. Asus is our savior and will protect us from botnets.

I guess making our devices crash so bad they can't stay online is one way to do it.

But it's a classic case of heads you don't win, tails you lose.

Lets say they give the option to disable it, then something happens to all those that have it disabled. They'll complain
You have it enabled and can't disable it, this scenario happens. They'll complain.

It only seems to have effected certain routers on certain firmware's (.386 by the looks of it), I don't believe all ASUS routers where effected.

 

[ENGER]

Up for 3 hours 43 minutes. Still stable. No growth in memory-used. CPU use still low.

 

[ColinTaylor]

But it's a classic case of heads you don't win, tails you lose.

Lets say they give the option to disable it, then something happens to all those that have it disabled. They'll complain
You have it enabled and can't disable it, this scenario happens. They'll complain.

It only seems to have effected certain routers on certain firmware's (.386 by the looks of it), I don't believe all ASUS routers where effected.

Being able to disable asd would render the process pointless as the malware it's designed to prevent would also just disable it. Again the Windows equivalent would be malware being able to disable your anti-virus software, you may as well not have any.

I'm not defending Asus on this, just explaining why they've been put in this position by governments insisting that they (and other manufacturers) "do something" to stop routers being infected. That said, there's still no excuse for pushing out bad code that so badly effects the router, especially so recently after the last problem with asd.

 

[feardc]

Same issue on RT-AX55 this night. Resolved with upgrade to 3.0.0.4.386_51598-ge383e0a, officially released on 2023/04/26.

 

[phoenix_pl]

I have same issue. Router DSL-AC68U

I found in /jffs/asd/ a chknvram20230516
Also asd.log is filled with "1684415553[chknvram_action] Invalid string"
I tried to suspend asd by kill -STOP PIDofASD but this doesn't work.
So far I do not have any workaround.
Hope Asus will release a fixed file.
Someone should be fired at Asus at least by 2 reasons:
1. Pushing some crappy updates even when AutoUpdate is OFF
2. Releasing anything which is causing so much trouble to so many different routers

My firmware:3.0.0.4.386_50117

 

[somax]

I have same issue. Router DSL-AC68U

I found in /jffs/asd/ a chknvram20230516
Also asd.log is filled with "1684415553[chknvram_action] Invalid string"
I tried to suspend asd by kill -STOP PIDofASD but this doesn't work.
So far I do not have any workaround.
Hope Asus will release a fixed file.
Someone should be fired at Asus at least by 2 reasons:
1. Pushing some crappy updates even when AutoUpdate is OFF
2. Releasing anything which is causing so much trouble to so many different routers

My firmware:3.0.0.4.386_50117

I have the same router and firmware version as you.

rm -f /jffs/asd/*
killall asd
restart router
upgrade firmware with the same version 3.0.0.4.386_50117-geaaff54
power off for 10 minutes without power plug
power on

now the file in /jffs/asd wasn't

chknvram20230516

but there is chknvram20230518
and i don't see more errors in asd.log

this work for me.

bye

 

[phoenix_pl]

I have the same router and firmware version as you.

rm -f /jffs/asd/*
killall asd
restart router
upgrade firmware with the same version 3.0.0.4.386_50117-geaaff54
power off for 10 minutes without power plug
power on

now the file in /jffs/asd wasn't


but there is chknvram20230518
and i don't see more errors in asd.log

this work for me.

bye

As I found out there is a simpler procedure:
rm /jffs/asd/chknvram20230516
restart router from WebUI

asd folder will be emptied and new files will be downloaded including chknvram20230518.

 

[danceedance]

What *i think* works for me was to disable aiprotection and reflash 386.
CPU i am seeing around <10% jumping around all 4 cores with random peaks up to 30% on a single core
Ram stays around 450-480mb after an hour.
CPU seem more spiky than i remembered...

 

[NXIL]

So for all those people whining about forced updates, this isn't a forced firmware update it's the equivalent of Microsoft pushing a Windows Defender update each day.

Hi Colin,

my firmware got a forced update, from

• Firmware version 3.0.0.4.388.22525 to

• Firmware version 3.0.0.4.388.23012

three days ago at 2AM.

Note that auto update is off. (And yes, this router on stock firmware instead of Merlin, which is dumb of me, because main user has been finishing busy college semester with lots of on line classes, on line study groups, research projects, etc, it has been stable, and I didn't want to mess with it while it was being used heavily. But Asus did a potentially disruptive update without permission a few days ago, which is pretty bogus I think.)

Anyway, semester ended yesterday, about to do factory reset and flash Merlin--

 

[ColinTaylor]

Hi Colin,

my firmware got a forced update, from

• Firmware version 3.0.0.4.388.22525 to

• Firmware version 3.0.0.4.388.23012

Yes I agree that does sound bad. My comment was only referring to the posts (in this and other threads) regarding the asd signature update on 2023-05-16. Sorry for the confusion.

 

[CBme]

For those of you not wanting to try that unofficial beta software, or if you have a different router, some folks are reporting having success with just deleting the file "chknvram20230516", as Somax and Pheonix mention above, which then updates to "chknvram2023518" and then they aren't experiencing the leak. Here is how to do it if you don't know how to run the command to delete the file:

Re: RT-AX89X going out of memory every 10 minutes (asd process?)​

You need to have telnet or ssh enabled. I used ssh. So if not enabled, between 2 crashes of the router, enable SSH (in Administration / System). Choose LAN only, choose a port, let's say 2215, and choose yes to "Allow Password Login". CONNECT --- MacOS or linux: connect by doing this in a...

rog-forum.asus.com
Tonio007
Level 1
In response to MrSusa
Options

5 hours ago
You need to have telnet or ssh enabled. I used ssh.
So if not enabled, between 2 crashes of the router, enable SSH (in Administration / System). Choose LAN only, choose a port, let's say 2215, and choose yes to "Allow Password Login".

CONNECT
---
MacOS or linux:
connect by doing this in a terminal:
ssh admin@<your router IP> -p 2215
and type the admin account password when invited.

Windows
Download and use PuTTY sofware to connect to your router, specifying the IP and port, user (admin) and password.

DELETE THE FILE
---
Once connected, from the prompt: simply do:
rm /jffs/asd/chknvram20230516
and then type
exit
to close

Then, you can use the router's UI to reboot it. Or using the switch.

 

[NXIL]

Thank you Colin!

I was quite surprised by this (the firmware change from an external source), and of course if Asus can do it, others can as well. (I know Merlin is against remote or automatic firmware updates).

Anyway, router is now updated with latest Merlin. Tried to do a factory reset via the web interface, but it did not clear out a lot of old data, so going to do a hard reset later and set it up again by hand (not hard, I keep it simple.)

Thanks Colin and the rest of the crew here, and of course Merlin!

 

[ColinTaylor]

Tried to do a factory reset via the web interface, but it did not clear out a lot of old data, so going to do a hard reset later and set it up again by hand (not hard, I keep it simple.)

You need to tick the "Initialize all the settings..." box to clear down data like web history, etc. Or just do the hard reset which achieves the same thing.

P.S. My guess is that the forced firmware update you experienced is not some kind of clandestine activity from Asus but more likely a bug in this particular firmware whereby it wasn't checking the auto-update setting correctly. But that's just a guess as this doesn't seem to have happened before.