GT-AX6000 - Memory Leak? Hacked?

[elbubi]

Hi!

Please, don't jump all over me, I know that unused ram is wasted ram, and that is totally normal behaviour to see +90% RAM used as it will be eventually get freed flushing caches and buffers once another process needs it.

I'm using a GT-AX6000 running 3004.388.7_0 (fresh install, no dirty upgrade)
Here's whats happened to me 4/5 times in the last 2/3 months: I see used memory increasing day after day (totally normal) but sometimes it does flush and free itself and sometimes I guess its not because first GUI becomes unresponsive, then can't even connect through SSH, followed by wi fi clients being dropped and last I also lose internet on wired device. Only solution is to power cicle the unit.

The only simpton I can see right before it happens is that used memory is very high, it never happened otherwise, that's why I suspect it might have something to do with it.

Running TOP command there are 2 processes consuming +100% VSZ when memory usage is at top and router is still responsive:

/tmp/socks5go
/tmp/linux

These SS where taken a few hours before the last crash (this morning)

Spoiler: GUI

Spoiler: GUI

I was able to SSH (after 5 minutes of waiting to validate login and other 5 to run TOP), when GUI and wifi/wired clients were already down and this was the output:

Spoiler

Both socks5go and linux procceses weren't there, I don't recall seeing those "nobody" user under normal circumstances but I can't tell if its normal o not, or any other clue that screenshot may pose.

Hope someone can shed a light here and help me troubleshoot this annoying issue,
Regards and thanks in advance!

 

[Yota]

/tmp/socks5go /tmp/linux

This is not normal, these two processes are not from the firmware, your router may have been hacked and is acting as a proxy server.

Factory reset, do not import the previous config, completely manual configuration may help to remove the malware.

But you still need to know how the malware hacked into your router, maybe you turned on WAN access?

 

[elbubi]

Dear god! Not again!!!

I have WAN access disabled (I use VPN server to access it remotely only from my phone and my office pc)

 

[ColinTaylor]

Dear god! Not again!!!

I was about to say the same thing.

Having one router being hacked is unfortunate. Having a second, completely different router hacked might suggest some sort of bad practice on the LAN side.

Errant teenagers?

 

[Yota]

Dear god! Not again!!!

If you have experienced this twice, maybe the real problem is with your LAN devices.

You mentioned that you couldn't get into SSH, twice, it's possible that one of your LAN devices was brute-forcing your SSH at the time, causing the traffic to be blocked. You may want to change the SSH port and password after the next factory reset.

But you still need to find out the cause of the problem, otherwise it will happen again.

 

[elbubi]

I have two little kids (9 and 3), they both use only tablets and are directed to Cleanbrowsing Family using DNS Director.
The little one only uses YT Kids every once in a while, the older mostly YT and gaming (99% Brawls Stars), so I think the are not to blame here...

The wife, she could be easily guilty for anything bad happening in my life hahaha. But I don't think she is to blame neither.

Anything I can do to to dig a little deeper before hard reseting and starting from scratch?
I really can't believe it, the first time I had WAN access enabled but this time I felt almost invencible.

Thanks both for your answers.

 

[Yota]

Maybe putting your kids' and your wife's devices on a guest network?

Anything I can do to to dig a little deeper before hard reseting and starting from scratch?

Dumping nvram and backing up jffs would help aid the investigation.
for dumping nvram

nvram show > /jffs/nvram_backup

But you should not share these dumps with anyone because they contain sensitive information like passwords, DDNS, IP address, etc.

 

[elbubi]

Maybe putting your kids' and your wife's devices on a guest network?

I was waiting for 3006 Merlin's in order to do just that, isolate kids (own and visitors) on a dedicated VLAN with proper dns director, and other one for my wife's devices.
Today I have guest networks enabled but only for visitors.

Dumping nvram and backing up jffs would help aid the investigation.

Ok, I'll do that when I get home and see if I can discover anything weird.

After hard reset I will change SSH port + password and also will change DDNS name (just to be 110% sure I don't repeat anything from past)

Thanks once again for the help

 

[shabbs]

Dear god! Not again!!!

I have WAN access disabled (I use VPN server to access it remotely only from my phone and my office pc)

Did you setup the same WIFI config as before? Same SSID/password? Wonder if your details have been compromised and shared. As noted by others - hard reset/wipe/start fresh - manual setup and go with different WIFI setup entirely. I know that can be a pain if you have a lot of IOT devices etc... but something is up for sure.

 

[elbubi]

Did you setup the same WIFI config as before? Same SSID/password? Wonder if your details have been compromised and shared. As noted by others - hard reset/wipe/start fresh - manual setup and go with different WIFI setup entirely. I know that can be a pain if you have a lot of IOT devices etc... but something is up for sure.

No, different SSID's and passwords, also different router password aswell. Same SSH port and DDNS though.

 

[elbubi]

Well, I did an nvram dump and jffs backup.
From jffs backup didn't find anything weird as from my previous hack when there was and opvn script file.
Only "strange" thing I found was "asd" but I undestand thats a security daemon from asus so nothing wrong there.
From nvram dump did not catch anything neither, but my undestanding is very limited.

What I also check is the syslog, this is from the last 2 hours before it crashed this morning:

Sep 12 08:42:07 wlceventd: wlceventd_proc_event(685): eth7: Auth AC:80:FB:EB:02:C1, status: Successful (0), rssi:-62
Sep 12 08:42:07 wlceventd: wlceventd_proc_event(722): eth7: Assoc AC:80:FB:EB:02:C1, status: Successful (0), rssi:-62
Sep 12 08:42:16 wlceventd: wlceventd_proc_event(645): eth7: Deauth_ind AC:80:FB:EB:02:C1, status: 0, reason: Deauthenticated because sending station is leaving (or has left) IBSS or ESS (3), rssi:0
Sep 12 08:42:16 wlceventd: wlceventd_proc_event(662): eth7: Disassoc AC:80:FB:EB:02:C1, status: 0, reason: Disassociated because sending station is leaving (or has left) BSS (8), rssi:0
Sep 12 08:44:58 wlceventd: wlceventd_proc_event(685): eth7: Auth AC:80:FB:EB:02:C1, status: Successful (0), rssi:0
Sep 12 08:44:58 wlceventd: wlceventd_proc_event(722): eth7: Assoc AC:80:FB:EB:02:C1, status: Successful (0), rssi:-75
Sep 12 08:45:09 wlceventd: wlceventd_proc_event(645): eth7: Deauth_ind AC:80:FB:EB:02:C1, status: 0, reason: Deauthenticated because sending station is leaving (or has left) IBSS or ESS (3), rssi:0
Sep 12 08:45:09 wlceventd: wlceventd_proc_event(662): eth7: Disassoc AC:80:FB:EB:02:C1, status: 0, reason: Disassociated because sending station is leaving (or has left) BSS (8), rssi:0
Sep 12 08:47:51 wlceventd: wlceventd_proc_event(685): eth7: Auth AC:80:FB:EB:02:C1, status: Successful (0), rssi:-62
Sep 12 08:47:51 wlceventd: wlceventd_proc_event(722): eth7: Assoc AC:80:FB:EB:02:C1, status: Successful (0), rssi:-62
Sep 12 08:48:02 wlceventd: wlceventd_proc_event(645): eth7: Deauth_ind AC:80:FB:EB:02:C1, status: 0, reason: Unspecified reason (1), rssi:-68
Sep 12 08:48:03 wlceventd: wlceventd_proc_event(645): eth7: Deauth_ind AC:80:FB:EB:02:C1, status: 0, reason: Unspecified reason (1), rssi:-68
Sep 12 08:52:57 wlceventd: wlceventd_proc_event(645): eth7: Deauth_ind CC:F7:35:E4:EF:FB, status: 0, reason: Disassociated due to inactivity (4), rssi:-68
Sep 12 08:54:22 wlceventd: wlceventd_proc_event(685): eth7: Auth AC:80:FB:EB:02:C1, status: Successful (0), rssi:-72
Sep 12 08:54:22 wlceventd: wlceventd_proc_event(722): eth7: Assoc AC:80:FB:EB:02:C1, status: Successful (0), rssi:-72
Sep 12 08:54:34 wlceventd: wlceventd_proc_event(645): eth7: Deauth_ind AC:80:FB:EB:02:C1, status: 0, reason: Unspecified reason (1), rssi:-72
Sep 12 08:54:34 wlceventd: wlceventd_proc_event(645): eth7: Deauth_ind AC:80:FB:EB:02:C1, status: 0, reason: Unspecified reason (1), rssi:-72
Sep 12 08:55:58 wlceventd: wlceventd_proc_event(685): eth7: Auth AC:80:FB:EB:02:C1, status: Successful (0), rssi:0
Sep 12 08:55:58 wlceventd: wlceventd_proc_event(722): eth7: Assoc AC:80:FB:EB:02:C1, status: Successful (0), rssi:-78
Sep 12 08:56:09 wlceventd: wlceventd_proc_event(645): eth7: Deauth_ind AC:80:FB:EB:02:C1, status: 0, reason: Unspecified reason (1), rssi:-74
Sep 12 08:56:11 wlceventd: wlceventd_proc_event(645): eth7: Deauth_ind AC:80:FB:EB:02:C1, status: 0, reason: Unspecified reason (1), rssi:-74
Sep 12 08:57:47 wlceventd: wlceventd_proc_event(645): eth7: Deauth_ind 64:5A:36:DB:27:F8, status: 0, reason: Disassociated due to inactivity (4), rssi:-74
Sep 12 08:57:50 wlceventd: wlceventd_proc_event(685): eth7: Auth 74:13:EA:ED:E8:FB, status: Successful (0), rssi:-58
Sep 12 08:57:50 wlceventd: wlceventd_proc_event(695): eth7: ReAssoc 74:13:EA:ED:E8:FB, status: Successful (0), rssi:-58
Sep 12 08:57:55 wlceventd: wlceventd_proc_event(645): eth7: Deauth_ind 74:13:EA:ED:E8:FB, status: 0, reason: Unspecified reason (1), rssi:0
Sep 12 08:57:55 wlceventd: wlceventd_proc_event(662): eth7: Disassoc 74:13:EA:ED:E8:FB, status: 0, reason: Disassociated because sending station is leaving (or has left) BSS (8), rssi:0
Sep 12 08:57:55 wlceventd: wlceventd_proc_event(645): eth7: Deauth_ind F4:E8:C7:77:E0:A2, status: 0, reason: Disassociated due to inactivity (4), rssi:-75

That repeats every endleslly every 5 seconds (shorten it cause 10k characters post limit)
Those endless Auth=>Assoc=>Deauth_ind=>Disassoc don't look normal, but I guess they are more a sympton than a cause.

Sep 12 08:58:34 dnsmasq[30660]: possible DNS-rebind attack detected: localhost.sensic.net
Sep 12 08:58:35 dnsmasq[30660]: possible DNS-rebind attack detected: localhost.sensic.net
Sep 12 08:58:35 dnsmasq[30660]: possible DNS-rebind attack detected: localhost.sensic.net
Sep 12 08:58:35 dnsmasq[30660]: possible DNS-rebind attack detected: localhost.sensic.net

Sep 12 09:01:35 kernel: eth5 (Int switch port: 6) (Logical Port: 6) (phyId: 13) Link DOWN.
Sep 12 09:01:39 kernel: eth5 (Int switch port: 6) (Logical Port: 6) (phyId: 13) Link Up at 100 mbps full duplex

Sep 12 09:02:58 kernel: eth5 (Int switch port: 6) (Logical Port: 6) (phyId: 13) Link DOWN.
Sep 12 09:03:07 kernel: eth5 (Int switch port: 6) (Logical Port: 6) (phyId: 13) Link Up at 2500 mbps full duplex

Sep 12 09:05:40 kernel: GET STA INFO failed, -21

Sep 12 09:39:41 check_watchdog: [check_watchdog] restart watchdog for no heartbeat
Sep 12 09:39:49 wlceventd: wlceventd_proc_event(645): eth7: Deauth_ind B4:0B:1D:14:8A:19, status: 0, reason: Deauthenticated because sending station is leaving (or has left) IBSS or ESS (3), rssi:0
Sep 12 09:39:57 wlceventd: wlceventd_proc_event(662): eth7: Disassoc B4:0B:1D:14:8A:19, status: 0, reason: Disassociated because sending station is leaving (or has left) BSS (8), rssi:0
Sep 12 09:41:01 HTTPD: waitting 10 minitues and restart
Sep 12 09:42:19 rc_service: check_watchdog 3306:notify_rc restart_watchdog

I also have the post reboot log, but its too long, didn't find anything strange neither, but...


What dazzles me is that those 2 suspicious processes (socks5go and linux) are not present now, I wonder what triggers them...
This a TOP output right now, I don't recall seeing those "nobody" user in the past but I may be wrong.

If anyone spots anything abnormal, glad to fiddle with it.

Thanks!!!!!!!!

 

[francisport]

I would recommend to reinforce your WAN/LAN/WIFI setup, but I'm guessing you already have done... but below some of my recommendations:

Maybe just have UPnP (specially wan side) and automatic trigger ports enabled? Please, disable it and enable just when you need.

REINFORCING WIFI
First, create a guest wifi for you IPTV/cam/iot devices if you have blocking LAN access

You might create another one for your kids, same blocking LAN access, but you can decide when to give them LAN access.

Set AES+TPKI WPA2 as minimum for wifi and disable UPnP as much as possible but specially at WAN side.

Use different wifi password for each wifi sid ... playing with uppercase and lowercase might help to reuse those >16chars avoid reinventing the wheel.

Set a password not less than 16 chars (repeat X times a simple word, the more X the best)

REINFORCE ADMIN ROUTER
Router username differently than just "admin" and set a password not less than 16 chars for it and also for wifi (repeat X times a simple word, the more X the best), remember that the more complex chars you use, the more difficult to type in some devices


Enable GUI/SSH access remotely might compromise for sure your security, but I think it's important to have control your home infra, this is the easiest way but with some security concerns, I know they are another ways, setup a VPN (with lan access) from your phone or ZeroNet might help to block GUI/SSH access from outside.... but those settings usually are not suitable for everyone.

Maybe you already have done these settings, but if not, let them a try.


I bet the problem might be in those apparently unoffensive "kids apps" to play with beautiful sounds or maybe some of your neighbors are more clever than you think.

 

[adrenalize]

Proton have a good password generator too: https://proton.me/pass/password-generator

The memorable word option is handier than the random ones. Longer length is better than complexity.

I'd also check any computers for unknown/not installed by you remote access software, Anydesk, logmein, splashtop etc etc which is a common phishing payload, not detected by security suites as it is genuine. They could then compromise the router from inside the lan, especially if passwords saved in the browser. You'd probably find a bad actor would have compromised anything they could get like banks, PayPal, email if that was the case though.

 

[elbubi]

Thanks both for your suggestions. I already did 99% of whats told a few months ago after 1st hack, so an "insider" from my LAN is very, but very unlikely (but not impossible)
I do not discard that some device as my wife's laptop or kids tablets might got infected and somehow that malware reached the router, but I don't have save passwords anywhere and only access the router from my pc and my phone, so only way to get password (is strong) is trough some keylogger.

I still can't believe it, hoping Eric stands up saying: "Wait a minute, \tmp\socks5go and \tmp\linux" are mine to blame, they are harmless!"

 

[sfx2000]

download manager running?

socks5 might be part of that, and there, it's known to suck up ram and process resources..

as a security note - one shouldn't be running ssh on a regular basis these days - if malware gets inside your LAN, it's trivial for a bot to get in to the router as credentials are already discovered.

 

[elbubi]

download manager running?

Thanks for your input SFX.

Now that you mention it, I do very rarely use Download Master to add some torrent and save downloads to a samba shared usd drive when I'm not at home and my desktop is off.
Nevertheless, I have DM WAN connection disabled. I open DM through VPN server connection from my office's desktop.
I downloaded a 4GB torrent file as soon as I read your comment, just to check, but socks5go did not show up.

I believed SSH was pretty safe, I use it mostly to check TOP (for the abnormal behaviour on the last months and previous hacks), and to check trifles like AMTM, added cronjobs, ARP table, etc.
I do also often use winSCP to check for something abnormal on scripts or ovpn (given my last hack experience).
Should I do something other way?

What still dazzles me, is that those 2 suspicious processes haven´t come up (yet) since yesterdays reboot, so the alegged malware is not persistent (what would be rare) or it has not been triggered yet.
But, if its not persistent, it would mean that I have been violated many times in the last 2/3 months, as those memory overflow hangs happened several times.
I still don't want to hard reset or change anything yet cause I would like to caugh it in the act (I know its riskier this way but I'm intrigued and I want to learn as much as possible from this event)

Thanks once again for your time (and everyone else's too)

 

[Viktor Jaep]

I would also recommend start browsing through your /jffs folder structures for items that should not be there based on what addons you have installed. Also, running command cru l to see what if anything might be automatedly running at times.

Maybe also browse to that /tmp folder, and see if you can find any files under there like that socks5go file. Download it and upload it to virustotal.com for analysis.

 

[elbubi]

Thanks Victor for your advice.

One of the reasons I frecuently SSH the router is to do precisely those tasks, check for anything unusual on /jffs folder (scripts/ovpn/addons folders specially) and cru l (or trough AMTM) to check for any unwanted running cronjob.

I dug and dug and dug for hours through every visible folder and file but haven´t been able to spot anything that looks suspicious (to me).

 

[AntonK]

Are you running AiProtection?

 

[elbubi]

Are you running AiProtection?

No, AiProtection 100% off.

If I recall correctly I read it can slow down connections (messes with Flow Cache and/or Runner).
Privacy wise I'm not concerned, I'am tracked 24/7 by my phone, apps, browser, etc, so 1 more tracking wouldn't hurt that much, but I don't want any hog slowing down my speeds (600/600).

Do you advice otherwise?