RT-AX89X suddenly started using a lot more CPU and memory leak leads to lockup every ~10mins

[CBme]

EDIT: As of May 19th, the issue should be fixed for everyone.

It took 48 hours, but the mystery of the mass Asus router outage is solved

Asus finally responds after being castigated by users.

arstechnica.com

At around 2am the router locked up. Rebooting and troubleshooting I found that CPU is much higher with different cores randomly hitting 100% momentarily but the main issue is that a very fast memory leak issue seems to be happening and that is what is causing it to lock up. Starts in the 40's and steadily climbs until it hits 100% and locks. Power cycle allows it to start working again until the memory gets consumed and locks. I've never seen the memory usage above the 40's nor any of the cores much above idle.

Haven't had any issue with the router for over a year and half. Haven't don't any recent changes to anything on it. Firmware isn't new 3.0.0.4.386_47468 . Currently only two systems on the lan using it. WAN is plugged into the 10g port (10g link -5g service).

- There was nothing in the logs that showed any errors.
- I did a Restore to Factory settings and it seemed to fix it initially but after another restart it resumed the strange cpu usage and memory consumption. I did not change any settings after the restore/did not turn on any features like QoS or anything else nor where they on before the restore.
- I disabled wifi and there was no change.
- I unplugged the DAC from the SFP+ port and put the computers on the 1gbit ports with no change.
- After reset - cleared the logs and started to get see some entries indicating it sees that it is running out of memory and a few times now has successfully killed some processes to stop it from locking up. However it goes right back to eating up the memory. I pasted the log in the attached text file in case any of you can tell what it is doing from that information.

Anyone have any idea what could be going on and/or how to troubleshoot it further? This is a life or death situation as my partner works from home and is looking at me with murderous intent.

 

[Zlid]

Same problem tonight as well, I noticed it also on the other thread. ASD starts leaking very fast.

Install the beta firmware from this page:

Beta - Asus RT-AX89X_9.0.0.4_388_31185 Beta

I've a feeling the RT-AX89X may be reaching EOL soon....I've been getting random reboots and Wifi disconnections for the past 2-3 months. Probably time for an upgrade to Wifi 7. What to replace the RT-AX89X with though? I use both the 10G base-T and 10G SFP+ ports.

www.snbforums.com

That fixed it for me. Registered to save you

 

[CBme]

Same problem tonight as well, I noticed it also on the other thread. ASD starts leaking very fast.

Install the beta firmware from this page:

Beta - Asus RT-AX89X_9.0.0.4_388_31185 Beta

I've a feeling the RT-AX89X may be reaching EOL soon....I've been getting random reboots and Wifi disconnections for the past 2-3 months. Probably time for an upgrade to Wifi 7. What to replace the RT-AX89X with though? I use both the 10G base-T and 10G SFP+ ports.

www.snbforums.com

That fixed it for me. Registered to save you

Thank you! That thread did come up in my search but I didn't look at it as I am not running that beta firmware so I assumed whatever issue was going on there was related to that beta firmware. I will get it read to apply as soon as she is off meetings and has a break and has no weapons within reach.

That is very strange that you, Townshend and I (and I assume others) had this issue out of the blue starting last night without a firmware update. Wondering what in the world could have triggered it.

 

[Townshend]

Just saw this thread too, something must've happened with the firmware (epoch related, too odd to all occur at the same time) or something in the Asus ecosystem kicked it off that we aren't privy too. Fortunately those here are fairly tech savvy, scary to think how many folks out there with this router are SOL.

 

[Zlid]

Yeah I was thinking the same, a lot of folks will start returning the router. Hopefully this thread will save some.

 

[Fiberton]

I believe it is a remote exploit that is using the router to mine something. If you reset the Router Then flash it.. Do not connect the wan to the net. Use another router to connect to the net and use its lan to connect the lan of your main router.Then it does not do it. Mine is working fine with another router taking care of the wan , and dhcp. But this is bad. You will need to make sure you set ips etc right but this is not good for ASUS. the logs will show a buffer allocation or something I forget. I caught it after about a 1/2 out screwing around to figure it out. Basically means its either the usuage of the wan port all of sudden some timed bug OR a remote exploit to the wan port. .. My bet is on the exploit.

 

[CBme]

Yeah I was thinking the same, a lot of folks will start returning the router. Hopefully this thread will save some.

Looks like the have a bug (or attack) that is taking out different models of routers

https://www.reddit.com/r/ASUS/comments/13k1kr1

 

[CBme]

I believe it is a remote exploit that is using the router to mine something. If you reset the Router Then flash it.. Do not connect the wan to the net. Use another router to connect to the net and use its lan to connect the lan if your main router.Then it does not do it. Mine is working fine with another router taking care of the wan , and dhcp. But this is bad. You will need to make sure you set ips etc right but this is not good for ASUS. the logs will show a buffer allocation or something I forget. I caught it after about a 1/2 out screwing around to figure it out.

That would be super concerning massive exploit from just being connected to the internet but certainly would explain the weird CPU usage and the memory leak and different models being affected.
But for this model, this beta seems to be resolving the issue so if it is an exploit, vs some other firmware bug, this is apparently stopping it from working: https://www.snbforums.com/threads/asus-rt-ax89x_9-0-0-4_388_31185-beta.82276/post-841736

 

[Volt]

I have the same issue with RT-AX56U! Tried to reset it multiple times, nothing helps - as soon as I restart the router, it starts using the CPU and cannot be stopped until it runs out of memory. I even ended up ordering another router tonight, because I though that it is broken. What is interesting is that installing Merlin fixes the issue. What a strange situation, indeed!

 

[Fiberton]

That would be super concerning massive exploit from just being connected to the internet but certainly would explain but the weird CPU usage and the memory leak. But this beta seems to be resolving the issue so if it is an exploit, vs some other firmware bug, this is apparently stopping it from working: https://www.snbforums.com/threads/asus-rt-ax89x_9-0-0-4_388_31185-beta.82276/post-841736

A massive remote exploit would explain why it is happening on so many routers of different types. The same exploit would mess up many across the board since they run on basically the same platform.

 

[Fiberton]

I have the same issue with RT-AX56U! Tried to reset it multiple times, nothing helps - as soon as I restart the router, it starts using the CPU and cannot be stopped until it runs out of memory. I even ended up ordering another router tonight, because I though that it is broken. What is interesting is that installing Merlin fixes the issue. What a strange situation, indeed!

I put a router in front of my AX89X and let it take care of the wan and once I reset the firmware on my router the cheap router is taking care of dhcp etc . Connect my AX89 via lan to the lan of the other one.. Sucks but I did this really quickly. No cpu / ram issue so it is a wan issue . More than likely an exploit which there was a write up last year about a remote HTTP exploit that allows you to take the router over. Read that this morning as I was trying to figure out wtf is going on.

 

[Fiberton]

Right now ASUS is more than likely scrambling to figure out how they can put a patch out to fix this. Hope they fix this soon.

 

[Fiberton]

That would be super concerning massive exploit from just being connected to the internet but certainly would explain the weird CPU usage and the memory leak and different models being affected.
But for this model, this beta seems to be resolving the issue so if it is an exploit, vs some other firmware bug, this is apparently stopping it from working: https://www.snbforums.com/threads/asus-rt-ax89x_9-0-0-4_388_31185-beta.82276/post-841736

Thanks I updated to that firmware and everything is fine now.

 

[smartbits]

I started having the same issue this morning (possibly last night) with my RT-AX82U. Is there any firmware for this router that can fix the issue?

 

[smartbits]

I believe it is a remote exploit that is using the router to mine something. If you reset the Router Then flash it.. Do not connect the wan to the net. Use another router to connect to the net and use its lan to connect the lan of your main router.Then it does not do it. Mine is working fine with another router taking care of the wan , and dhcp. But this is bad. You will need to make sure you set ips etc right but this is not good for ASUS. the logs will show a buffer allocation or something I forget. I caught it after about a 1/2 out screwing around to figure it out. Basically means its either the usuage of the wan port all of sudden some timed bug OR a remote exploit to the wan port. .. My bet is on the exploit.

I'd like to use this method until ASUS releases a fix. Would you be able to give me a brief walk through on how to set this up.

I will use my old router to connect to the internet and connect an ethernet cable from my old router's LAN to my ASUS router's WAN port. Do I then disable DHCP on my ASUS router? Which IPs will I need to set?

 

[BurningBronze]

I started having the same issue this morning (possibly last night) with my RT-AX82U. Is there any firmware for this router that can fix the issue?

Same problem here on my RT-AX92U! I have a Motorola Cable Modem that connects to the internet but seeing the same out of memory issues and keep losing the WAN link to the internet.

 

[Fiberton]

I'd like to use this method until ASUS releases a fix. Would you be able to give me a brief walk through on how to set this up.

I will use my old router to connect to the internet and connect an ethernet cable from my old router's LAN to my ASUS router's WAN port. Do I then disable DHCP on my ASUS router? Which IPs will I need to set?

Basically the thing to do is first you need todo a factory reset. Make sure to have the wan disconnected. Second router just do DHCP, and DNS. Connect both by the lan connector. Your gateway will be the ip of the first router. WAN-Router1 192.168.1.1 - lan-Router 2 192.168.1.2. You connect the wan to the first old router. Whatever exploit is happening it is not effecting other old stuff.

 

[smartbits]

Basically the thing to do is first you need todo a factory reset. Make sure to have the wan disconnected. Second router just do DHCP, and DNS. Connect both by the lan connector. Your gateway will be the ip of the first router. WAN-Router1 192.168.1.1 - lan-Router 2 192.168.1.2. You connect the wan to the first old router. Whatever exploit is happening it is not effecting other old stuff.

Thanks. By factory reset, do you mean pressing and holding the reset button or do you mean the hard reset method? I previously did a reset by holding the reset button on the router. I set it up again. I guess I would need to do another reset?
I found a link that explains how to set everyone up when using 2 routers. Does this seem accurate to you?

https://support.bluos.net/hc/en-us/articles/360000220927-Can-I-connect-two-routers-to-the-same-network-with-the-same-network-name-

 

[Fiberton]

Thanks. By factory reset, do you mean pressing and holding the reset button or do you mean the hard reset method? I previously did a reset by holding the reset button on the router. I set it up again. I guess I would need to do another reset?
I found a link that explains how to set everyone up when using 2 routers. Does this seem accurate to you?

https://support.bluos.net/hc/en-us/articles/360000220927-Can-I-connect-two-routers-to-the-same-network-with-the-same-network-name-

Yes that will clear it out. And whatever exploit gets purged.

 

[Fiberton]

Thanks. By factory reset, do you mean pressing and holding the reset button or do you mean the hard reset method? I previously did a reset by holding the reset button on the router. I set it up again. I guess I would need to do another reset?
I found a link that explains how to set everyone up when using 2 routers. Does this seem accurate to you?

https://support.bluos.net/hc/en-us/articles/360000220927-Can-I-connect-two-routers-to-the-same-network-with-the-same-network-name-

Seems correct.