Suspicious Outgoing traffic on RT-AC86U

[arturk]

Hello everyone, I am new here and looking for some help with my RT-AC86U.

Over the last few days I started experiencing frequent internet connectivity interruptions: video audio streams interrupted, unable to refresh web page.
I setup continuous ping -t google.com and this quickly revealed every few minutes connection was being dropped for few seconds.
Next I logged into router UI to see if I could find out more.
When I noticed that at the same time ping packets dropped, CPU utilization on router would go crazy (from few percent to 80%+).

Switching to "Traffic Analyzer" revealed huge bursts of outbound traffic which collated exactly with high CPU and dropped ping packages.

This chart shows history for last 10 minutes. You can see over 2GB of data being sent during this time, it is absolutely crazy.

I started looking for source of this traffic within my networks. I have close to 40 devices (Windows Laptops and Servers, NAS, Security Cameras, IOT, ....).
I could not find any device ending data using router build in tools like "QoS Bandwidth Monitor" or Wireshark.

Meanwhile gigabytes of data were being sent.

You can see over 62GB reported in around 4 hours span.

I decided to disconnect everything from the router.
With only 1 cellphone connected to the router GUI to be able to see Traffic Monitor nothing changed, outbound traffic still present and unchanged.
In few hour of troubleshooting I could find nothing.

None of this outbound traffic is registered but router's Bandwidth Monitor.
In QoS I limited "Upload Bandwidth" to 5Mb/s and while this restricted bandwidth on the phone it did nothing to slow down suspicious bursts of outbound traffic from the router. When I initiate traffic from cellphone (or other device) traffic is properly indicated by QoS Bandwidth Monitor.
At this point I came to conclusion that whatever it is, router itself must be doing it. I know it sounds crazy, how can router generate this much of outgoing traffic?

After few ours outbound traffic stopped on its own. It was fine for about two days and this morning it returned. it is not as severe as before, down to average 600MB per 10 minutes from over 2GB but still bad.
I do not think it is happening constantly 24 hours a day but at least for few hours and I do not see any obvious pattern.

I am completely lost and looking for some ideas.
Sorry about the long post but I was trying to be as descriptive as possible. If I missed something please let me know.

Here is some more details about router and settings.
2021 ASUS RT-AC86U with appropriate latest Merlin Firmware 386.14.
I have all Port Forwarding disabled.
Web Access from WAN is also disabled.

 

[dave14305]

Sounds like malware. See this thread.

RT-AX88U maxing out a core and regularly showing 60+ MB/s upload

I had Diversion and Skynet running but have since disabled them, suspecting that may be the culprit. Essentially, I can't see any device connected to my network that is sending large amounts of data, but when I look at the traffic monitor (Analyzer) it shows this: I only have ~30Mbps of...

www.snbforums.com

 

[arturk]

Sounds like malware. See this thread.

RT-AX88U maxing out a core and regularly showing 60+ MB/s upload

I had Diversion and Skynet running but have since disabled them, suspecting that may be the culprit. Essentially, I can't see any device connected to my network that is sending large amounts of data, but when I look at the traffic monitor (Analyzer) it shows this: I only have ~30Mbps of...

www.snbforums.com

Hi Dave, thanks a lot for such quick response!
It definitely looks like my problem.
I started reading the thread and going through recommended troubleshooting steps.

So far I identified {sshd} is involved by looking at the TOP while outbound traffic was taking place.

But "find / -name sshd" does not return anything.

I will keep looking...

 

[ColinTaylor]

Please run ls -altr /tmp/ while the problem is happening and post all the output.

Can you also confirm whether you have AiCloud 2.0 enabled.

 

[vaboro]

@arturk do you have AiCloud and/or WAN access to router's UI interface enabled?
386.14 includes lighttpd 1.4.39 that is CVE-2024-3094 vulnerable.

 

[Justinh]

But "find / -name sshd" does not return anything.

Is that because of the braces around the name in the top output? (need to use "{sshd}"?)

What do the braces indicate?

 

[dave14305]

What do the braces indicate?

The program has changed the process name to be different than the binary name. I don’t know all the details of how it works, but shows how the malware is trying to hide its tracks.

Most of the sleuthing was done in the other thread. The sshd processes are links to an unauthorized binary in /tmp.

 

[ColinTaylor]

Is that because of the braces around the name in the top output? (need to use "{sshd}"?)

No.

What do the braces indicate?

It indicates that it's a thread rather than a process and from the other information we can see the original process has terminated leaving the first one of these as an orphan. Periodically it will spawn two more copies of itself, one of which starts listening on a random UDP port.

386.14 includes lighttpd 1.4.39 that is CVE-2024-3094 vulnerable.

I don't think that is applicable. They are saying don't compile lighttpd using the compromised xz 5.6.0 source code (CVE-2024-3094). The router does doesn't use that version.

 

[vaboro]

I don't think that is applicable. They are saying don't compile lighttpd using the compromised xz 5.6.0 source code (CVE-2024-3094). The router does use that version.

Maybe it is not. However, I started having the very same problem some time ago and CVE-2024-3094 timing seemed relevant to me.

Spoiler: lighttpd crashed regularly and then an intruder opened the backdoor

Sep 23 20:08:19 GT-AC2900-1 kernel: lighttpd[1507]: unhandled level 3 translation fault (11) at 0x00000000, esr 0x92000007
Sep 23 20:08:19 GT-AC2900-1 kernel: pgd = ffffffc03196d000
Sep 23 20:08:19 GT-AC2900-1 kernel: [00000000] *pgd=000000003192b003, *pud=000000003192b003, *pmd=00000000318c6003, *pte=0000000000000000
Sep 23 20:08:19 GT-AC2900-1 kernel: CPU: 1 PID: 1507 Comm: lighttpd Tainted: P           O    4.1.27 #2
Sep 23 20:08:19 GT-AC2900-1 kernel: Hardware name: Broadcom-v8A (DT)
Sep 23 20:08:19 GT-AC2900-1 kernel: task: ffffffc03416f540 ti: ffffffc031960000 task.ti: ffffffc031960000
Sep 23 20:08:19 GT-AC2900-1 kernel: PC is at 0xf66ebf4c
Sep 23 20:08:19 GT-AC2900-1 kernel: LR is at 0xf66ebf44
Sep 23 20:08:19 GT-AC2900-1 kernel: pc : [<00000000f66ebf4c>] lr : [<00000000f66ebf44>] pstate: 600f0010
Sep 23 20:08:19 GT-AC2900-1 kernel: sp : 00000000ffa46cf8
Sep 23 20:08:19 GT-AC2900-1 kernel: x12: 00000000f6703354
Sep 23 20:08:19 GT-AC2900-1 kernel: x11: 000000000001cc98 x10: 00000000007eee78
Sep 23 20:08:19 GT-AC2900-1 kernel: x9 : 00000000ffffffff x8 : 0000000000000041
Sep 23 20:08:19 GT-AC2900-1 kernel: x7 : 0000000000000000 x6 : 00000000f66ef5bc
Sep 23 20:08:19 GT-AC2900-1 kernel: x5 : 0000000000878208 x4 : 00000000008e45f0
Sep 23 20:08:19 GT-AC2900-1 kernel: x3 : 00000000008d60d8 x2 : 000000000089ffe8
Sep 23 20:08:19 GT-AC2900-1 kernel: x1 : 000000000000000a x0 : 0000000000000000
Sep 23 20:08:19 GT-AC2900-1 kernel: potentially unexpected fatal signal 11.
Sep 23 20:08:19 GT-AC2900-1 kernel: CPU: 1 PID: 1507 Comm: lighttpd Tainted: P           O    4.1.27 #2
Sep 23 20:08:19 GT-AC2900-1 kernel: Hardware name: Broadcom-v8A (DT)
Sep 23 20:08:19 GT-AC2900-1 kernel: task: ffffffc03416f540 ti: ffffffc031960000 task.ti: ffffffc031960000
Sep 23 20:08:19 GT-AC2900-1 kernel: PC is at 0xf66ebf4c
Sep 23 20:08:19 GT-AC2900-1 kernel: LR is at 0xf66ebf44
Sep 23 20:08:19 GT-AC2900-1 kernel: pc : [<00000000f66ebf4c>] lr : [<00000000f66ebf44>] pstate: 600f0010
Sep 23 20:08:19 GT-AC2900-1 kernel: sp : 00000000ffa46cf8
Sep 23 20:08:19 GT-AC2900-1 kernel: x12: 00000000f6703354
Sep 23 20:08:19 GT-AC2900-1 kernel: x11: 000000000001cc98 x10: 00000000007eee78
Sep 23 20:08:19 GT-AC2900-1 kernel: x9 : 00000000ffffffff x8 : 0000000000000041
Sep 23 20:08:19 GT-AC2900-1 kernel: x7 : 0000000000000000 x6 : 00000000f66ef5bc
Sep 23 20:08:19 GT-AC2900-1 kernel: x5 : 0000000000878208 x4 : 00000000008e45f0
Sep 23 20:08:19 GT-AC2900-1 kernel: x3 : 00000000008d60d8 x2 : 000000000089ffe8
Sep 23 20:08:19 GT-AC2900-1 kernel: x1 : 000000000000000a x0 : 0000000000000000

I upgraded to 384.14 386.14 from 384.12 386.12 but the problem persisted.
As soon as I closed all lighttpd external ports the problem stopped.

 

[ColinTaylor]

Maybe it is not. However, I started having the very same problem some time ago and CVE-2024-3094 timing seemed relevant to me.

I upgraded to 384.14 from 384.12 but the problem persisted.
As soon as I closed all lighttpd external ports the problem stopped.

I'm also suspicious of lighttpd. But it wouldn't be because of this particular CVE as 384.12 predates it by nearly five years.

 

[vaboro]

But it wouldn't be because of this particular CVE as 384.12 predates it by nearly five years.

Yeah, but this particular problem never happened to me before this summer. And CVE-2024-3094 was published on March 29, 2024. It might have taken some time for some enthusiast crackers to explore the vulnerability and write some exploits.

 

[ColinTaylor]

Yeah, but this particular problem never happened to me before this summer. And CVE-2024-3094 was published on March 29, 2024. It might have taken some time for some enthusiast crackers to explore the vulnerability and write some exploits.

I think you're missing the point. The code for that exploit didn't exist before 2014 2024, therefore it couldn't have been included in firmware 384.12.

 

[Tech9]

Yeah, but this particular problem never happened to me before this summer.

But you had Web Access from WAN enabled before the issue happened, no?

 

[vaboro]

I think you're missing the point. The code for that exploit didn't exist before 2014, therefore it couldn't have been included in firmware 384.12.

Am I? 384.12 386.12 was released on Sep 4, 2023. And the same problem persisted in 384.14 386.14 released on Jul 20, 2024. CVE was released on Mar 29, 2024.
Both firmwares include dated versions of lighttpd.

 

[vaboro]

But you had Web Access from WAN enabled before the issue happened, no?

No WAN access but I used AiCloud. As soon as I switched off the AiCloud the problem stopped.

 

[ColinTaylor]

Am I? 384.12 was released on Sep 4, 2023. And the same problem persisted in 384.14 released on Jul 20, 2024. CVE was released on Mar 29, 2024.
Both firmwares include dated versions of lighttpd.

Typo in my previous post: I meant 2024 not 2014.

You're saying 384.12 but pointing to 386.14. In any case the xz source code used by that firmware is seven years old.

But as I said, I'm also suspicious it's lighttpd.

 

[vaboro]

You're saying 384.12 but pointing to 386.14.

You're right, sorry for confusion. My mistake.
Although I had it right in my original post.

 

[arturk]

Folks, I just got back home after few hours and see overwhelming number of tips.
Thanks everyone!!!

Please run ls -altr /tmp/ while the problem is happening and post all the output.

Can you also confirm whether you have AiCloud 2.0 enabled.

It is currently not happening but I will be ready to run it next time it does. Out of curiosity, are we looking for specific suspicious file to show in /tmp folder? There over 60 files there already.

@arturk do you have AiCloud and/or WAN access to router's UI interface enabled?
386.14 includes lighttpd 1.4.39 that is CVE-2024-3094 vulnerable.

Good point, I did not remember I had AiCloud. I checked and "Cloud Disk" and Smart Access" was on. I don't really use it so I can turn it on. I think I wait until it starts uploading again and the will turn it off.

Is that because of the braces around the name in the top output? (need to use "{sshd}"?)

What do the braces indicate?

Yes, I did try with braces and got nothing.

I think you're missing the point. The code for that exploit didn't exist before 2014 2024, therefore it couldn't have been included in firmware 384.12.

I forgot to mention at the time I noticed the problem I was on 386.13_2. Next day I upgraded to 386_14 and problem is still here.

But you had Web Access from WAN enabled before the issue happened, no?

Yes, I did have WAN enabled before, it is setup for HTTPS with Let's Encrypt certificate. I am not ruling out someone get in but I am kind of careful and I never had or knew of any breach before.

 

[vaboro]

@arturk WAN access to web ui and everything related to AiCloud (except ddns service) should be disabled to stop all this madness that is happening to you.
You can continue using ddns safely though. It's on inadyn which seems to be safe for now.

 

[arturk]

@arturk WAN access to web ui and everything related to AiCloud (except ddns service) should be disabled to stop all this madness that is happening to you.
You can continue using ddns safely though. It's on inadyn which seems to be safe for now.

Vaboro, thanks for the tip. Definitely makes sense.
I wanted to have WAN access when I travel for extended period of time but honestly I never really needed it. It is probably not worth the risk.
AiCloud stuff is useless for me so I have no problem disabling it.
I do need DDNS though, so I am glad to hear it is relatively safe.