RT-AX89X suddenly started using a lot more CPU and memory leak leads to lockup every ~10mins

[Chumara]

I registered to say thanks for all the work in this thread!

I have an ASUS RT-AC5300, so not the same model referenced in this thread, and also not the exact same symptoms, but close enough that this thread seemed like the right place to be.
In my instance, I was getting the constant spikes between my two CPU cores, as well as the wild increase in RAM usage(>70%). The reason I even found this out is because the actual problem I was having is that after a reboot I'd have about 5-6 devices connected, but then the 7th or so device that I'd connect just couldn't get an IP address assigned from the router.

The system log would show:
failed to write /var/lib/misc/dnsmasq.leases: No space left on device
without stopping until I rebooted the router.

I've had this router for about 5 years now with more or less the same number of devices and I've never had an issue until yesterday.
I followed the instructions that CBme outlined above and it seems to have settled things down, my CPU usage on both cores is <10%, and my RAM usage is down to ~33% after deleting the file and rebooting. I haven't had a device fail to grab an IP yet, so hopefully all is well, I'll keep an eye on resource usage and see.

As part of my troubleshooting yesterday I did install the latest firmware (didn't help), and I was going to do a factory reset (would've SUCKED) had I not stumbled upon this thread, so thanks again to everyone!

 

[MakeItEasy]

Upgrading FW from 3.0.0.4.386_48260 to 3.0.0.4.386_51529 solves the issue for my RT-AC86U (done two hours ago).

 

[CBme]

Thank you Colin!

I was quite surprised by this (the firmware change from an external source), and of course if Asus can do it, others can as well. (I know Merlin is against remote or automatic firmware updates).

Anyway, router is now updated with latest Merlin. Tried to do a factory reset via the web interface, but it did not clear out a lot of old data, so going to do a hard reset later and set it up again by hand (not hard, I keep it simple.)

Thanks Colin and the rest of the crew here, and of course Merlin!

Just to be clear for anyone who found this thread because they have the issue with their AX89X, Merlin is NOT an option for the AX89X as it is Qualcomm based and not Broadcom. Merlin is Broadcom based Asus only. Models it is an option for: https://www.asuswrt-merlin.net/

 

[smartbits]

I deleted the file that caused the memory leak, rebooted and stopped the process. I just want to make sure Asus releases a fix or confirms that future versions of that bad file will not cause the same problem. I also think they should have made an official announcement as soon as they discovered there was an issue.

Imagine how many people there are who had the same problem and do not frequent this forum or the Asus support forum. They probably still have no idea what caused the problem and either called their ISP because they think their internet service is not working properly or they think the router went bad and either threw it out if it's no longer under warranty or just bought a new one because they need one right away.

Asus handled this very poorly.

 

[2B\/~2B <- qn]

Networking novice here having the same problem with my home AC3100, also starting yesterday morning.

Starting first thing yesterday, the syslog has hundreds of the following errors in the log:

dnsmasq[393]: failed to allocate 124 bytes
dnsmasq-dhcp[393]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)

In addition to this it's been assigning different IP addresses to client machines, applying internet access schedules wrong, randomly disallowing access to my 2.4 WiFi and generally being flaky. My current firmware version is the latest, 3.0.0.4.386_48260-gd4c241c. Rebooting fixes the problems and memory errors temporarily, but they soon come back.

Before finding this forum I contacted ASUS support and they said to restore the router to factory settings. I would rather not go to all that trouble if I don't have to, and if it's a firmware bug that won't fix it anyway (?). Sounds like this is a known & very recent issue but the support guy did not mention that. I asked if re-applying the current firmware version would help: he said it might. I downloaded and applied it, but the "no space left" error came back in about half an hour. I also saw the following in the log after uploading the firmware version:

May 18 13:40:14 WATCHDOG: [FAUPGRADE][auto_firmware_check:(7669)]do webs_update
May 18 13:40:18 WATCHDOG: [FAUPGRADE][auto_firmware_check:(7687)]retrieve firmware information
May 18 13:40:18 WATCHDOG: [FAUPGRADE][auto_firmware_check:(7702)]firmware update check first time
May 18 13:40:18 WATCHDOG: [FAUPGRADE][auto_firmware_check:(7733)]no need to upgrade firmware

Looks like it didn't actually reapply the current version ( ? ).

So the current advice from ASUS is to drop back to the previous firmware version for your router models; do you think I should try the same for my AC3100 (back to 3.0.0.4.386.46065)?

Thanks for your help!

 

[CBme]

Networking novice here having the same problem with my home AC3100, also starting yesterday morning.

So the current advice from ASUS is to drop back to the previous firmware version for your router models; do you think I should try the same for my AC3100 (back to 3.0.0.4.386.46065)?

Thanks for your help!

No, the first thing you should do is see if deleting the problematic file resolves the issue: https://www.snbforums.com/threads/r...eads-to-lockup-every-10mins.85018/post-842138

 

[Chumara]

Networking novice here having the same problem with my home AC3100, also starting yesterday morning.

Starting first thing yesterday, the syslog has hundreds of the following errors in the log:

dnsmasq[393]: failed to allocate 124 bytes
dnsmasq-dhcp[393]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)

In addition to this it's been assigning different IP addresses to client machines, applying internet access schedules wrong, randomly disallowing access to my 2.4 WiFi and generally being flaky. My current firmware version is the latest, 3.0.0.4.386_48260-gd4c241c. Rebooting fixes the problems and memory errors temporarily, but they soon come back.

Before finding this forum I contacted ASUS support and they said to restore the router to factory settings. I would rather not go to all that trouble if I don't have to, and if it's a firmware bug that won't fix it anyway (?). Sounds like this is a known & very recent issue but the support guy did not mention that. I asked if re-applying the current firmware version would help: he said it might. I downloaded and applied it, but the "no space left" error came back in about half an hour. I also saw the following in the log after uploading the firmware version:

May 18 13:40:14 WATCHDOG: [FAUPGRADE][auto_firmware_check:(7669)]do webs_update
May 18 13:40:18 WATCHDOG: [FAUPGRADE][auto_firmware_check:(7687)]retrieve firmware information
May 18 13:40:18 WATCHDOG: [FAUPGRADE][auto_firmware_check:(7702)]firmware update check first time
May 18 13:40:18 WATCHDOG: [FAUPGRADE][auto_firmware_check:(7733)]no need to upgrade firmware

Looks like it didn't actually reapply the current version ( ? ).

So the current advice from ASUS is to drop back to the previous firmware version for your router models; do you think I should try the same for my AC3100 (back to 3.0.0.4.386.46065)?

Thanks for your help!

Yeah I was having these exact same issues on my model (RT-AC5300) and I tried CBme's fix earlier today, everything still looking good so far so I think that was the cause.

 

[2B\/~2B <- qn]

ASUS just put out an announcement on the subject: https://www.asus.com/ca-en/support/

 

[jstrellner]

ASUS just put out an announcement on the subject: https://www.asus.com/ca-en/support/

Wow, look at that timing... 10 mins ago I decided to move forward with switching to Ubiquiti Unifi specifically because they still had not publicly addressed that they had this issue.

I lost an entire day of work (I work from home) because of this issue. Never again.

 

[2B\/~2B <- qn]

Hopefully you'll have better luck with your new router


So the ASUS solution, from that support page, is to [1] reboot, and then [2] do a factory restore. I just tried the reboot and I'm monitoring the CPU and RAM ... I have 2 cores and 512MB in my AC3100. The cores are taking turns spiking up to 90-100% every 2-3 seconds -- I don't know if that's normal, never really paid attention to it before -- and the memory usage seems to have levelled off at around 72% / 360MB.

Fingers crossed. Don't want to have to do the hard reset.

 

[drinkingbird]

Wow, look at that timing... 10 mins ago I decided to move forward with switching to Ubiquiti Unifi specifically because they still had not publicly addressed that they had this issue.

I lost an entire day of work (I work from home) because of this issue. Never again.

View attachment 50208

I'm actually shying away from Unifi because it requires you to use a new open sourced version of Java now which does not receive automatic patches and none of the variants seem to be thoroughly tested or supported. Given that there is a new Java vulnerability every month or two, that's not an option for me. For now I'm running the last version of their controller that supports official Java, which for my older APs is fine.

 

[CBme]

Hopefully you'll have better luck with your new router


So the ASUS solution, from that support page, is to [1] reboot, and then [2] do a factory restore. I just tried the reboot and I'm monitoring the CPU and RAM ... I have 2 cores and 512MB in my AC3100. The cores are taking turns spiking up to 90-100% every 2-3 seconds -- I don't know if that's normal, never really paid attention to it before -- and the memory usage seems to have levelled off at around 72% / 360MB.

Fingers crossed. Don't want to have to do the hard reset.

Aside from an update firmware that prevents the issue, deleting that file is the only thing that has been reported to consistantly work. Doing a factory reset, not a hard reset, was one of the first things many people tried, and it didn't work for us as the bad file remained.

 

[CBme]

Wow, look at that timing... 10 mins ago I decided to move forward with switching to Ubiquiti Unifi specifically because they still had not publicly addressed that they had this issue.

I lost an entire day of work (I work from home) because of this issue. Never again.

I'd get the Asus working and keep it as a backup. I had been thinking about getting a backup internet service should one go out, and since the AX89X can do failover WAN, but this is a great reminder that having a backup router from a different manufacturer is just as important if you need that level of uptime.

 

[jz238]

For those of you not wanting to try that unofficial beta software, or if you have a different router, some folks are reporting having success with just deleting the file "chknvram20230516", as Somax and Pheonix mention above, which then updates to "chknvram2023518" and then they aren't experiencing the leak. Here is how to do it if you don't know how to run the command to delete the file:

Re: RT-AX89X going out of memory every 10 minutes (asd process?)​

You need to have telnet or ssh enabled. I used ssh. So if not enabled, between 2 crashes of the router, enable SSH (in Administration / System). Choose LAN only, choose a port, let's say 2215, and choose yes to "Allow Password Login". CONNECT --- MacOS or linux: connect by doing this in a...

rog-forum.asus.com
Tonio007
Level 1
In response to MrSusa
Options

5 hours ago
You need to have telnet or ssh enabled. I used ssh.
So if not enabled, between 2 crashes of the router, enable SSH (in Administration / System). Choose LAN only, choose a port, let's say 2215, and choose yes to "Allow Password Login".

CONNECT
---
MacOS or linux:
connect by doing this in a terminal:
ssh admin@<your router IP> -p 2215
and type the admin account password when invited.

Windows
Download and use PuTTY sofware to connect to your router, specifying the IP and port, user (admin) and password.

DELETE THE FILE
---
Once connected, from the prompt: simply do:
rm /jffs/asd/chknvram20230516
and then type
exit
to close

Then, you can use the router's UI to reboot it. Or using the switch.

My router is an RT-AC88U and the /jffs/asd/chknvram20230516 did not exist. Following the directions from CBme, I was able to see I had a different file in /jffs/asd/. Follow CDme directions above. From Putty change directory by typing.

cd /jffs/asd

List directory contents by typing.

ls
There was 1 file listed /jffs/asd/blockfile20230510 Your router may have a different file? Type the following command to remove the file.

rm /jffs/asd/blockfile20230510
from Putty type

exit
to close and then reboot the router.

 

[Matthew Patrick]

My router is an RT-AC88U and the /jffs/asd/chknvram20230516 did not exist. Following the directions from CBme, I was able to see I had a different file in /jffs/asd/. Follow CDme directions above. From Putty change directory by typing.

cd /jffs/asd

List directory contents by typing.

ls
There was 1 file listed /jffs/asd/blockfile20230510 Your router may have a different file? Type the following command to remove the file.

rm /jffs/asd/blockfile20230510
from Putty type

exit
to close and then reboot the router.

View attachment 50217

How is it? Is it fixed?

 

[jz238]

How is it? Is it fixed?

Yes. The processor and RAM usage screenshot is after 9 hours of stable operation.

 

[Matthew Patrick]

Yes. The processor and RAM usage screenshot is after 9 hours of stable operation.

I see. Can you check what the current naming of the file in /jffs/asd ? So i know which one is confirmed safe? Thank you

 

[jz238]

The router is a RT-AC88U. The blockfile20230510 has the same name that I deleted. That was the only listing in ASD last night. The chknvram20230518 was not in the folder last night.

 

[Matthew Patrick]

View attachment 50227

The router is a RT-AC88U. The blockfile20230510 has the same name that I deleted. That was the only listing in ASD last night. The chknvram20230518 was not in the folder last night.

Okay so both are fine now. Thank you!!

 

[Ameenhuus]

Last night at 7:40 PM my router rebooted when I was not home. It has now worked fine for 24 hours. Checked the file in Putty. This one was not present but it was the one that should be there. So it can go well by itself.

admin@RT-AX89X-C8A0:/tmp/home/root# rm /jffs/asd/chknvram20230516
rm: can't remove '/jffs/asd/chknvram20230516': No such file or directory
admin@RT-AX89X-C8A0:/tmp/home/root# ls /jffs/asd/
blockfile20230510 chknvram20230518 version versionbk
admin@RT-AX89X-C8A0:/tmp/home/root#