RT-AX88U maxing out a core and regularly showing 60+ MB/s upload

[jd24]

If you have the "sshd" process running, under normal circumstances that would be the OpenSSH server daemon (i.e. /opt/sbin/sshd) which can be installed via Entware (it's *not* built-in as part of the F/W).

View attachment 61858

View attachment 61859

However, if you have *not* explicitly installed the Entware package yourself, then someone or something else may have installed it, likely for nefarious purposes. I'd suggest you remove the OpenSSH package immediately. It may even be a "compromised" version of the OpenSSH server that has been installed by some malware.

Try the following commands to remove the package (if it exists):

{
   opkg list-installed | grep openssh ; echo
   opkg remove --force-removal-of-dependent-packages openssh-server
   opkg list-installed | grep openssh ; echo
}

My 2 cents.

Sorry how do I run this if opkg is not found?

I'm experiencing the same issues again after some hours

 

[ColinTaylor]

Sorry how do I run this if opkg is not found?

If opkg is not found then you don't have Entware installed, and therefore you also don't have the Entware openssh package.

 

[jd24]

I've been thinking of dropping in a dedicated vpn box like the tplink R605 omada or TL-ER7206
If this issue continues may throw this in between the bridge and my asus. (or initially just remove the asus and add the R605 as vpn and dhcp.)

 

[kknishev]

now {sshd} come back on 388.7

it's annoying that the router can work well with 0-2% CPU load for a long time, and then these surges

 

[Ripshod]

now {sshd} come back on 388.7
View attachment 61937

it's annoying that the router can work well with 0-2% CPU load for a long time, and then these surges

You need to find out what device on your network has been compromised, or this will drive you mad. You have reset the router and set up from scratch with new passwords after firmware changes of course(?).
Do you use a USB drive? This will need formatting too.
An afterthought: Are you using the built-in amtm or installing it from the repository (old, superceded method)?

 

[kknishev]

You need to find out what device on your network has been compromised, or this will drive you mad. You have reset the router and set up from scratch with new passwords after firmware changes of course(?).

Am I right that you think it's not a router issue but the someone of clients? It's my home network and I know each device...
I use mesh(eth) ax86u+ax56u with loading Merlin -> factory reset -> restore config (without change passwords and so on). Also I unplug usb-flash with mnt disk for ftp.

 

[ColinTaylor]

now {sshd} come back on 388.7

Please run these commands and post all the output (copy and paste all the text, do not post screenshots):

top -bn 1
netstat -nlp

 

[Ripshod]

Entware lives on the usb drive. If there's a problem there it'll likely still be a problem after a reset.
You're restoring config from a router that has a problem. You could be re introducing that problem. If an external actor has access, restoring the config and not changing passwords leaves the door wide open again.
You really should be completely resetting the router, formatting the usb to ntfs with windows (not an app), and then setting it up manually (not restoring) from scratch preferably in stages with testing at each stage, with fresh passwords.
Any scripts you install should be from developers on these forums. Installing just anything from github etc is asking for trouble.

 

[firecracker]

Hi all. I just registered on this forum to chime in with my experiences. I also have an RT-AX88U experiencing the same problem. After digging online and in this thread specifically I have observed that the offending process is listed as {sshd} when I login and run the "top" command. While the issue is occurring, one core is pegged at %100 and the WAN upload reports approximately a full gigabit of traffic. All clients on the network lose connection to the internet until the spike dissipates. I am running stock Asus firmware without Entware or any other modifications. The traffic seems to be generated in the router as there is no corresponding traffic from any device on the LAN. The following screenshot from the traffic monitor illustrates this perfectly. My upstream bandwidth to the ISP can only hit about 25Mbps. Each of those blue upload spikes FAR exceeds that. This has been driving my entire household crazy as it impacts work-from-home, gaming, streaming, etc. I have tried downgrading to the second most recent firmware, disconnecting the USB drive, ensured that the VPN server, DDNS, port forwarding/triggering, web access from WAN are disabled. I am wondering if there is some kind of new zero day being exploited or something. Any ideas? Thanks

 

[kknishev]

Please run these commands and post all the output (copy and paste all the text, do not post screenshots):

top -bn 1
netstat -nlp

attached in files, thanks a lot for support

 

[kknishev]

While the issue is occurring, one core is pegged at %100 and the WAN upload reports approximately a full gigabit of traffic. All clients on the network lose connection to the internet until the spike dissipates.

Absolutely right - you describe my situation exactly.

 

[firecracker]

Please run these commands and post all the output (copy and paste all the text, do not post screenshots):

top -bn 1
netstat -nlp

The output was too long for the post. I have attached the output in a text file.

 

[jonathonballen]

Hi all. I just registered on this forum to chime in with my experiences. I also have an RT-AX88U experiencing the same problem. After digging online and in this thread specifically I have observed that the offending process is listed as {sshd} when I login and run the "top" command. While the issue is occurring, one core is pegged at %100 and the WAN upload reports approximately a full gigabit of traffic. All clients on the network lose connection to the internet until the spike dissipates. I am running stock Asus firmware without Entware or any other modifications. The traffic seems to be generated in the router as there is no corresponding traffic from any device on the LAN. The following screenshot from the traffic monitor illustrates this perfectly. My upstream bandwidth to the ISP can only hit about 25Mbps. Each of those blue upload spikes FAR exceeds that. This has been driving my entire household crazy as it impacts work-from-home, gaming, streaming, etc. I have tried downgrading to the second most recent firmware, disconnecting the USB drive, ensured that the VPN server, DDNS, port forwarding/triggering, web access from WAN are disabled. I am wondering if there is some kind of new zero day being exploited or something. Any ideas? Thanks

View attachment 61940

Hi firecracker,

I had exactly the same issues as you and tried all of the same remedies. The thing that worked in the end was to do a factory reset to the last-but-one FW and NOT restore from my backup. It’s a PITA having to redo all the various router settings, but at least it gave me a stable connection. I didn’t have time (much pressure from those trying to WFH) to work out whether the issue was due to something originating from one of the router clients or in the latest FW.

 

[firecracker]

Hi firecracker,

I had exactly the same issues as you and tried all of the same remedies. The thing that worked in the end was to do a factory reset to the last-but-one FW and NOT restore from my backup. It’s a PITA having to redo all the various router settings, but at least it gave me a stable connection. I didn’t have time (much pressure from those trying to WFH) to work out whether the issue was due to something originating from one of the router clients or in the latest FW.

Hi Jon,
Good to hear! How long have you gone without the issue reoccurring?

 

[jonathonballen]

Hi Jon,
Good to hear! How long have you gone without the issue reoccurring?

When I first restored using latest FW and backup, it initially seemed OK for about 12 hours, then the issue came back every 15 minutes or so. I then restored without backup and everything was fine for say 36 hours, before I left on holiday. It’ll be interesting to see how long everything’s OK when I get back. I only reconnected a couple of clients necessary for WFH, so if the issue recurs I can start to eliminate rogue clients.

 

[ColinTaylor]

@kknishev @firecracker can you run those commands again while the CPU problem is actually occurring. Thanks.

 

[jd24]

When I first restored using latest FW and backup, it initially seemed OK for about 12 hours, then the issue came back every 15 minutes or so. I then restored without backup and everything was fine for say 36 hours, before I left on holiday. It’ll be interesting to see how long everything’s OK when I get back. I only reconnected a couple of clients necessary for WFH, so if the issue recurs I can start to eliminate rogue clients.

darn that's a pain. Is there any way of exporting the dhcp static addresses? I have about 50 assigned! Otherwise would need to redo all those manually if I reset and don't use a backup to re-import.

I'm not seeing unsual activity this morning so far. I had switched off the ddns, instantguard and openvpn last night. Also changed my password log in.
Also turned off one of my twingate agents on proxmox.

 

[jd24]

Hi all. I just registered on this forum to chime in with my experiences. I also have an RT-AX88U experiencing the same problem. After digging online and in this thread specifically I have observed that the offending process is listed as {sshd} when I login and run the "top" command. While the issue is occurring, one core is pegged at %100 and the WAN upload reports approximately a full gigabit of traffic. All clients on the network lose connection to the internet until the spike dissipates. I am running stock Asus firmware without Entware or any other modifications. The traffic seems to be generated in the router as there is no corresponding traffic from any device on the LAN. The following screenshot from the traffic monitor illustrates this perfectly. My upstream bandwidth to the ISP can only hit about 25Mbps. Each of those blue upload spikes FAR exceeds that. This has been driving my entire household crazy as it impacts work-from-home, gaming, streaming, etc. I have tried downgrading to the second most recent firmware, disconnecting the USB drive, ensured that the VPN server, DDNS, port forwarding/triggering, web access from WAN are disabled. I am wondering if there is some kind of new zero day being exploited or something. Any ideas? Thanks

View attachment 61940

Good to hear that this is not just a one-off event from 1 person.
All weekend my family was complaining about the internet dropping off. Thought it was my pihole containers so switched over to google. Rebooted, issues kept coming back.
Then had a stroke when I viewed my GB usage on my ISP provider app portal!! Began monitoring and saw these upload spikes consuming my data.

Do we suspect a hacker has targeted our asus gateways?

Touch wood I have had about 17 hours of smooth service. I'm working from my office at home with the traffic analyser running on 1 screen viewing that.

Changed password, ddns and openvpn off. I'll wait another 24hours before I switch on the openvpn/ddns again.

Could have been as simple as just changing the password for the router gui login page!

 

[bennor]

Is there any way of exporting the dhcp static addresses?

Yes there are a variety of ways to export or save the DHCP manual IP reservations (under Asus-Merlin) and reimport them later. One popular method is to use the YazDHCP addon script. Search this subforum to find other discussions and methods of exporting the manual reservations.

YazDHCP - YazDHCP - feature expansion of DHCP assignments (increasing limit on the number of DHCP reservations)

v1.0.6 Updated 2023-11-18 Feature expansion of DHCP assignments using AsusWRT-Merlin's Addons API to read and write DHCP assignments, increasing the limit on the number of reservations. YazDHCP is free to use under the GNU General Public License version 3 (GPL 3.0). This project is hosted on...

www.snbforums.com

GitHub - jackyaz/YazDHCP: Feature expansion of DHCP assignments using AsusWRT-Merlin's Addons API to read and write DHCP assignments, increasing the limit on the number of reservations.

Feature expansion of DHCP assignments using AsusWRT-Merlin's Addons API to read and write DHCP assignments, increasing the limit on the number of reservations. - jackyaz/YazDHCP

github.com

Edit to add: One of the other methods is suggested by RMerlin in this post:

If the goal is to preserve the DHCP reservations, you can copy the /jffs/nvram/dhcp_staticlist file between routers.

If however you mean the networkmap client list, I believe there are also files you can copy, but I don't know which off hand - other people listed them in the past in other posts.

 

[kknishev]

can you run those commands again while the CPU problem is actually occurring