RT-AX88U maxing out a core and regularly showing 60+ MB/s upload

[dave14305]

Can either of you run this command to see what is actually running?

for i in $(pidof sshd); do ls -l /proc/$i/exe; done

This malware is trying very hard to cover its tracks,

 

[doczenith1]

I think I'm experiencing a similar issue. Been going on for a few months now. I may try turning off the OpenVPN server and see if that changes anything.

AiCloud - OFF
AiProtection - OFF
SSH Port Forwarding - OFF
OpenVPN server - ON (non-standard port but do see connection attempts)
WireGuard server - ON

I don't see any sustained data transfer nor have seen any high cpu usage like the OP.

Daily

Last 24 hours showing some ridiculous values.

 

[firecracker]

Can either of you run this command to see what is actually running?

for i in $(pidof sshd); do ls -l /proc/$i/exe; done

This malware is trying very hard to cover its tracks,

I did this and nothing was returned. Simply moved to the next line with another prompt.

 

[firecracker]

Thanks.


It doesn't seem to be the cause, but the option I was referring to was Administration - System > Allow SSH Port Forwarding.

EDIT: Reboot the router after making any changes to get rid of the currently running {sshd} processes.

My "System" page does not have that option. Running stock firmware, not merlin if that helps.

 

[dave14305]

I did this and nothing was returned. Simply moved to the next line with another prompt.

Were the {sshd} processes visible in top at the time you ran it? If not, wait for the next occurrence and try again, please.

 

[zk1sps]

My "System" page does not have that option. Running stock firmware, not merlin if that helps.

I think he means this section of the administration / system configuration. This is the ASUS firmware. I am still testing if this helps to stop the outgoing traffic. ATM for the last 12 hours no traffic flooding after hardware reset and install newest version of ASUS firmware, (not Merlin), again factory reset and new very basic, rudimentary configuration from scratch.

 

[ColinTaylor]

My "System" page does not have that option. Running stock firmware, not merlin if that helps.

Yes, that option isn't present in stock firmware. I assumed you were using Merlin's firmware as this is the Merlin forum.

 

[kknishev]

crossed fingers.. from last night looks like fixed for me
currently on RT-AX86U_3004_388.8_2:
AiCloud 2.0 - totally OFF
System - Authentication Method - HTTP (was BOTH)

 

[Depthcommaoutof]

  PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
19405  5315 bald      R      228  0.0   0 24.8 {sshd} 

bald@rustyb:/tmp/home/root# ps | grep "[s]sh"
  932 bald        224 S    {sshd} 
 1695 bald        224 S    {sshd} 
 4050 bald        224 S    {sshd} 
...
27340 bald        224 S    {sshd} 
27438 bald        224 S    {sshd} 
28062 bald        224 S    {sshd} 
bald@rustyb:/tmp/home/root# 

bald@rustyb:/tmp/home/root# for i in `ps | grep -i "[s]shd" | awk '{print $1}'` ; do ec
ho ${i} ; ls -l /proc/${i}/exe ; echo ; done
932
lrwxrwxrwx    1 bald      root             0 Oct 19 03:18 /proc/932/exe -> /tmp/hklp (deleted)

1695
lrwxrwxrwx    1 bald      root             0 Oct 19 03:18 /proc/1695/exe -> /tmp/hklp (deleted)

4050
lrwxrwxrwx    1 bald      root             0 Oct 19 03:18 /proc/4050/exe -> /tmp/hklp (deleted)

...

27340
lrwxrwxrwx    1 bald      root             0 Oct 19 03:18 /proc/27340/exe -> /tmp/hklp (deleted)

27438
lrwxrwxrwx    1 bald      root             0 Oct 19 03:18 /proc/27438/exe -> /tmp/hklp (deleted)

28062
lrwxrwxrwx    1 bald      root             0 Oct 19 03:18 /proc/28062/exe -> /tmp/hklp (deleted)

bald@rustyb:/tmp/home/root# ls -l /tmp/hklp
-rwxrwxrwx    1 bald      root         70928 Oct 19 02:18 /tmp/hklp
bald@rustyb:/tmp/home/root#

PPID 224 does not exist. I tried to just kill them off and two show up again immediately.

 

[ColinTaylor]

PPID 224 does not exist. I tried to just kill them off and two show up again immediately.

Thanks, that's very interesting.

"224" isn't a PPID it's the memory size. The router's ps doesn't show the PPID unfortunately which is why we were using top in batch mode.

Can you try and make a copy of /tmp/hklp and store it somewhere for later analysis? Might need to send it to Asus or @RMerlin.

 

[dave14305]

PPID 224 does not exist. I tried to just kill them off and two show up again immediately.

Is there anything else in /tmp that looks unusual with a similar timestamp as hklp?

ls -ltr /tmp

 

[Depthcommaoutof]

Thanks, that's very interesting.

"224" isn't a PPID it's the memory size. The router's ps doesn't show the PPID unfortunately which is why we were using top in batch mode.

Can you try and make a copy of /tmp/hklp and store it somewhere for later analysis? Might need to send it to Asus or @RMerlin.

Thank you for the ps info.

I saved a copy of /tmp/hklp. clamscan doesn't alert on it.

# sha256sum /tmp/hklp

7d05acb08bed2d5ac3d0c5e8affdca477a78e19799f2672d6811292f954ea955 /tmp/hklp

I rm'ed /tmp/hklp.

bald@rustyb:/tmp/home/root# ps | grep -i "[s]shd"
27513 bald 224 S {sshd}
bald@rustyb:/tmp/home/root# kill -9 27513
bald@rustyb:/tmp/home/root# ps | grep -i "[s]shd"
28301 bald 224 S {sshd}
28302 bald 232 S {sshd}
bald@rustyb:/tmp/home/root# for i in `ps | grep -i "[s]shd" | awk '{print $1}'` ; do echo ${i} ; ls -l /proc/${i}/exe ; echo ; done
28301
lrwxrwxrwx 1 bald root 0 Oct 19 05:25 /proc/28301/exe -> /tmp/hklp (deleted)
28302
lrwxrwxrwx 1 bald root 0 Oct 19 05:25 /proc/28302/exe -> /tmp/hklp (deleted)
bald@rustyb:/tmp/home/root# ls -lah /tmp | grep hk
bald@rustyb:/tmp/home/root#

--reboot--

bald@rustyb:/tmp/home/root# ps | grep -i "[s]shd"
bald@rustyb:/tmp/home/root# ls -lah /tmp | grep hk
bald@rustyb:/tmp/home/root#


I did start using amtm and entware for the first time and install almost everything...

To get rid of it I have tried: restarting with factory refresh and importing configuration file and reinstalling the firmware image. But the issue always resurfaced. That was all before I rm'ed /tmp/hklp.

How can I dump the configuration in text like form?

How should I properly "nuke this box from orbit?" Hold WPS, etc?

Thank you for your assistance and your work on this. It is nifty.

 

[ColinTaylor]

I saved a copy of /tmp/hklp. clamscan doesn't alert on it.

If you have the time I'd be interested in getting a copy of that file. You can PM me if you're willing, although I'd understand if you don't.

If the problem reoccurs please run the command in post #111 exactly as written.

How should I properly "nuke this box from orbit?" Hold WPS, etc?

Do a hard factory reset. You haven't said what router you're using so you'll have to look up your model on this page:

[Wireless Router] When Standard Reset Isn’t Working: Hard Factory Reset - Models list | Official Support | ASUS Global

 

[Depthcommaoutof]

Is there anything else in /tmp that looks unusual with a similar timestamp as hklp?

ls -ltr /tmp

bald@rustyb:/tmp/home/root# ls -ltr /tmp/
drwxr-xr-x 4 bald root 80 Dec 31 1969 var
drwxr-xr-x 2 bald root 40 Dec 31 1969 share
-rw-r--r-- 1 bald root 0 Dec 31 1969 settings
drwxr-xr-x 3 bald root 60 Dec 31 1969 notify
drwxr-xr-x 2 bald root 40 Dec 31 1969 inadyn.cache
drwxr-xr-x 3 bald root 60 Dec 31 1969 home
drwxr-xr-x 3 bald root 60 Dec 31 1969 confmtd
-rw-r--r-- 1 bald root 17 Dec 31 1969 misc.json
lrwxrwxrwx 1 bald root 18 May 4 2018 syslog.log-1 -> /jffs/syslog.l1
lrwxrwxrwx 1 bald root 16 May 4 2018 syslog.log -> /jffs/syslog.log
------x--T 1 bald root 0 Oct 19 05:27 ebtables.lock
-rw-rw-rw- 1 bald root 2238 Oct 19 05:27 wl0_hapd.conf
drwxrwxrwx 2 bald root 40 Oct 19 05:27 netool
drwxrwxrwx 2 bald root 80 Oct 19 05:27 asdfile
-rw-rw-rw- 1 bald root 0 Oct 19 05:27 asd.init
-rw-rw-rw- 1 bald root 2240 Oct 19 05:27 wl1_hapd.conf
srwxrwxrwx 1 bald root 0 Oct 19 05:27 wpa_ctrl_2509-2
srwxrwxrwx 1 bald root 0 Oct 19 05:27 wpa_ctrl_2509-1
-rw-rw-rw- 1 bald root 0 Oct 19 05:27 awsiot_log
drwxrwxrwx 3 bald root 80 Oct 19 05:27 avahi
-rw-rw-rw- 1 bald root 359 Oct 19 05:27 run_lldpd.sh
-rw-rw-rw- 1 bald root 0 Oct 19 05:27 mastiff_log
-rw-rw-rw- 1 bald root 5 Oct 19 05:27 mastiff.pid
-rw-rw-rw- 1 bald root 64 Oct 19 05:27 lldpd_bind_ifnames
-rw-rw-rw- 1 bald root 64 Oct 19 05:27 lld2d.conf
-rw-rw-rw- 1 bald root 4302 Oct 19 05:27 lighttpd.conf
drwxrwxrwx 2 bald root 40 Oct 19 05:27 asusfbsvcs
drwxr-xr-x 2 bald root 40 Oct 19 05:27 cfg_mnt
lrwxrwxrwx 1 bald root 8 Oct 19 05:27 zcip -> /sbin/rc
lrwxrwxrwx 1 bald root 8 Oct 19 05:27 wpa_cli -> /sbin/rc
lrwxrwxrwx 1 bald root 8 Oct 19 05:27 udhcpc_wan -> /sbin/rc
drwxrwxrwx 3 bald root 260 Oct 19 05:27 ppp
lrwxrwxrwx 1 bald root 8 Oct 19 05:27 dhcp6c -> /sbin/rc
-rw-r--r-- 1 bald root 263 Oct 19 05:27 wchannel.json
-rw-rw-rw- 1 bald root 411 Oct 19 05:27 filter_ipv6.default
-rw-rw-rw- 1 bald root 1200 Oct 19 05:27 filter.default
-rw-r--r-- 1 bald root 398 Oct 19 05:27 chanspec_private.json
-rw-rw-rw- 1 bald root 192 Oct 19 05:27 chanspec_avbl.txt
-rw-r--r-- 1 bald root 248 Oct 19 05:27 chanspec_avbl.json
-rw-r--r-- 1 bald root 446 Oct 19 05:27 chanspec_all.json
drwxrwxrwx 2 bald root 60 Oct 19 05:27 asusdebuglog
-rw-r--r-- 1 bald root 72 Oct 19 05:27 E8:9C:25:D4:B6:B0.cap
-rw-r--r-- 1 bald root 31 Oct 19 05:27 E8:9C:25:D4:B6:B0.bi
-rw-r--r-- 1 bald root 2 Oct 19 05:27 relist.json
-rw-rw-rw- 1 bald root 92 Oct 19 05:27 obvsie
-rw-rw-rw- 1 bald root 1 Oct 19 05:27 obstatus
-rw-rw-rw- 1 bald root 92 Oct 19 05:27 guest_vsie
-rw-r--r-- 1 bald root 93 Oct 19 05:27 aplist.json
drwxrwxrwx 2 bald root 100 Oct 19 05:27 dm
-rw-rw-rw- 1 bald root 19 Oct 19 05:27 hw_auth_clm
drw-rw-rw- 2 bald root 60 Oct 19 05:27 bwdpi
-rw-rw-rw- 1 bald root 432 Oct 19 05:28 wan0_bound.env
-rw-rw-rw- 1 bald root 10 Oct 19 05:28 udhcpc0.expires
-rw-rw-rw- 1 bald root 663 Oct 19 05:28 redirect_rules
-rw-r--r-- 1 bald root 153 Oct 19 05:28 nbr_list.json
-rw-rw-rw- 1 bald root 496 Oct 19 05:28 nat_rules_eth0_eth0
lrwxrwxrwx 1 bald root 24 Oct 19 05:28 nat_rules -> /tmp/nat_rules_et0
-rw-rw-rw- 1 bald root 5818 Oct 19 05:28 filter_rules
srwxrwxrwx 1 bald root 0 Oct 19 05:28 mcpd_mcpctl_addr
-rw-rw-rw- 1 bald root 1 Oct 19 05:28 ping_76.133.210.1_79.done
-rw-rw-rw- 1 bald root 146 Oct 19 05:28 ping_76.133.210.1_79
-rw-r--r-- 1 bald root 120 Oct 19 05:29 resolv.dnsmasq
-rw-r--r-- 1 bald root 46 Oct 19 05:29 resolv.conf
drwxrwxrwx 5 bald root 240 Oct 19 05:29 lighttpd
-rw-rw-rw- 1 bald root 1 Oct 19 05:29 ping_76.133.210.1_80.done
-rw-rw-rw- 1 bald root 146 Oct 19 05:29 ping_76.133.210.1_80
drwxrwxrwx 6 bald root 140 Oct 19 05:29 mnt
-rw-rw-rw- 1 bald root 102 Oct 19 05:29 usb_err
drwxrwxrwx 3 bald root 60 Oct 19 05:29 skynet
drwxrwxrwx 2 bald root 180 Oct 19 05:29 nc
-rw-rw-r-- 1 bald root 26242 Oct 19 05:29 menuTree.js
-rw-rw-rw- 1 bald root 2931 Oct 19 05:29 diag_port_status.json
-rw-rw-rw- 1 bald root 9332 Oct 19 05:34 ce0.log
-rw-rw-rw- 1 bald root 216 Oct 19 05:38 usb.log
-rw-rw-rw- 1 bald root 94 Oct 19 05:46 webs_upgrade.log
-rw-r--r-- 1 bald root 162 Oct 19 05:58 allwclientlist.json
-rw-r--r-- 1 bald root 137 Oct 19 05:59 wiredclientlist.json
-rw-r--r-- 1 bald root 61 Oct 19 05:59 current_wired_client_list.json
-rw-rw-rw- 1 bald root 5832 Oct 19 05:59 dev
drwxr-xr-x 9 bald root 1760 Oct 19 05:59 etc
-rw-r--r-- 1 bald root 304 Oct 19 05:59 clientlist.json
-rw-r--r-- 1 bald root 1450 Oct 19 05:59 nmp_cache.js
-rw-rw-rw- 1 bald root 0 Oct 19 05:59 watchdog_heartbeat
bald@rustyb:/tmp/home/root# date
Sat Oct 19 06:00:47 PDT 2024
bald@rustyb:/tmp/home/root#

It doesn't seem to have left anything else behind that I can see at this time. I remember it also had a new timestamp every couple of hours which could have been every reboot.

 

[Depthcommaoutof]

If you have the time I'd be interested in getting a copy of that file. You can PM me if you're willing, although I'd understand if you don't.

If the problem reoccurs please run the command in post #111 exactly as written.



Do a hard factory reset. You haven't said what router you're using so you'll have to look up your model on this page:

[Wireless Router] When Standard Reset Isn’t Working: Hard Factory Reset - Models list | Official Support | ASUS Global

Sorry, it is a RT-AX88U Pro.

 

[ZebMcKayhan]

I saved a copy of /tmp/hklp. clamscan doesn't alert on it.

If you wish to share the file, it's convenient to use paste.c-net, like:

wget --quiet -O- --post-file='/tmp/hklp' 'https://paste.c-net.org/'

You will get a web link as return on the command which you can share with people you choose. Works directly on the router.

 

[Depthcommaoutof]

If you wish to share the file, it's convenient to use paste.c-net, like:

wget --quiet -O- --post-file='/tmp/hklp' 'https://paste.c-net.org/'

You will get a web link as return on the command which you can share with people you choose. Works directly on the router.

Oh that is super cool. Thank you.

https://paste.c-net.org/NutsoForging

 

[ColinTaylor]

Oh that is super cool. Thank you.

https://paste.c-net.org/NutsoForging

The plot thickens.

Unfortunately, this file was deemed malicious by automated scans.
All files are automatically scanned for malicious content, in effort to avoid serving malware to unsuspecting users.
If you are the original uploader of this file, and believe the detection is a false positive, please get in touch.

As a quick workaround, for passing a malware sample to a colleague, encrypting your content would probably bypass the scan.
See the "Privacy / Encrypted usage" section at https://paste.c-net.org/ for details.

 

[ColinTaylor]

@Depthcommaoutof Might need to zip the file up with a password to avoid the detection.

 

[BikeHelmet]

I think I'm experiencing a similar issue. Been going on for a few months now. I may try turning off the OpenVPN server and see if that changes anything.

AiCloud - OFF
AiProtection - OFF
SSH Port Forwarding - OFF
OpenVPN server - ON (non-standard port but do see connection attempts)
WireGuard server - ON

I don't see any sustained data transfer nor have seen any high cpu usage like the OP.

Daily
View attachment 61983

Last 24 hours showing some ridiculous values.
View attachment 61984

My RT-AX56U used to do the weird bandwidth spikes to super high numbers before I turned off flowcache. The internet would also get crappy for a while when flowcache and QOS were mixed. I ended up keeping QOS (FlexQOS) and disabling flowcache, and then everything was groovy. Wasn't malware for me, just a core router feature misbehaving. I think it misplaces packets, causing other services to get stuck. If any of you have it enabled, you might want to try turning it off temporarily with fc disable over ssh, to see if the behaviour changes. It was pretty immediate for me. If it doesn't fix it, then back to the malware theory.