High memory usage on GT-AX6000 - how to diagnose ?

[SR-G]

Hello,

I recently bought a GT-AX6000 ASUS ROG router. Asus Merlin (latest 388.1 version) is installed inside it + Diversion (amtm) + custom DNSMasq.
I'm now encountering some very high memory usage (99% or even 100%), leading to ... the web UI not being sometime anymore accessible, or even the SSH access ...
I have no clue what's going on !
Is it a configuration issue ? An ASUS issue ? A Diversion issue ?

My main question is "how can i troubleshoop that with Asus Merlin" - i just have "top" but it's far from being enough (i don't see so many processes eating up memory, while using top + SHIFT-M to sort per mem usage)
I've just killed the Pixelserv-tls stuff (was top at ~2% but not sure it's the culprit) : i just recovered a few megabytes free (and access to UI), but i don't think it was the culprit

Mem: 980748K used, 38100K free, 322072K shrd, 67664K buff, 420908K cached
CPU: 0.4% usr 1.0% sys 0.0% nic 98.5% idle 0.0% io 0.0% irq 0.0% sirq
Load average: 0.24 0.13 0.11 1/194 19626
PID PPID USER STAT VSZ %VSZ CPU %CPU COMMAND
13872 1 admin S 13288 1.3 1 0.5 watchdog
18990 13872 admin S 14552 1.4 0 0.1 networkmap
8769 1 admin S 18164 1.7 0 0.0 amas_portstatus
26752 1 admin S 15492 1.5 3 0.0 cfg_server
19001 1 admin S 11844 1.1 0 0.0 httpd -i br0
7643 1 admin S 4952 0.4 3 0.0 /usr/sbin/wlc_monitor
8644 1 admin S 20064 1.9 0 0.0 aaews --sdk_log_dir=/tmp
18178 1 nobody S 19388 1.8 3 0.0 dnsmasq --log-async
2387 1 admin S 13944 1.3 0 0.0 nt_monitor
2592 1 admin S 13036 1.2 3 0.0 sw_devled
2589 1 admin S 13036 1.2 1 0.0 alt_watchdog
2590 1 admin S 4468 0.4 3 0.0 rstats
409 2 admin SW 0 0.0 0 0.0 [bcmsw_rx]
401 2 admin SW 0 0.0 2 0.0 [fc_timer]
13931 2 admin IW 0 0.0 0 0.0 [kworker/0:0-eve]
2369 1 admin S 25268 2.4 2 0.0 asd
7785 1 admin S 21184 2.0 3 0.0 amas_lib
7781 1 admin S 19432 1.9 1 0.0 conn_diag

CPU seems OK

In syslog (attached) there are a lot of memory errors, but it's just one process encountering the "'no memory anymore available" symptom so probably not the culprit)

admin@GT-AX6000-98B0:/tmp/home/root# ps |grep 26752
20340 admin 5780 S grep 26752
26752 admin 15492 S cfg_server

Thanks in advance for any ideas / help and/or directions about which tools are available to diagnose the issues on Asus routers (with Merlin activated).

 

[ColDen]

Hello,

I recently bought a GT-AX6000 ASUS ROG router. Asus Merlin (latest 388.1 version) is installed inside it + Diversion (amtm) + custom DNSMasq.
I'm now encountering some very high memory usage (99% or even 100%), leading to ... the web UI not being sometime anymore accessible, or even the SSH access ...
I have no clue what's going on !
Is it a configuration issue ? An ASUS issue ? A Diversion issue ?

My main question is "how can i troubleshoop that with Asus Merlin" - i just have "top" but it's far from being enough (i don't see so many processes eating up memory, while using top + SHIFT-M to sort per mem usage)
I've just killed the Pixelserv-tls stuff (was top at ~2% but not sure it's the culprit) : i just recovered a few megabytes free (and access to UI), but i don't think it was the culprit

CPU seems OK

In syslog (attached) there are a lot of memory errors, but it's just one process encountering the "'no memory anymore available" symptom so probably not the culprit)



Thanks in advance for any ideas / help and/or directions about which tools are available to diagnose the issues on Asus routers (with Merlin activated).

There is a known memory issue with the ASUS 388 code affecting several routers and it have been successfully suggested (by dave14305) to run the following command, after having reboot, from Putty:
service stop_conn_diag
This is what I have done in my case.

 

[SR-G]

Good to know, i'll try that, thanks for the tip (memory was leaking probably over the weeks, so will be hard to say if it definitely solves my issue, but let's see).

 

[bluepoint]

Hello,

I recently bought a GT-AX6000 ASUS ROG router. Asus Merlin (latest 388.1 version) is installed inside it + Diversion (amtm) + custom DNSMasq.
I'm now encountering some very high memory usage (99% or even 100%), leading to ... the web UI not being sometime anymore accessible, or even the SSH access ...
I have no clue what's going on !
Is it a configuration issue ? An ASUS issue ? A Diversion issue ?

My main question is "how can i troubleshoop that with Asus Merlin" - i just have "top" but it's far from being enough (i don't see so many processes eating up memory, while using top + SHIFT-M to sort per mem usage)
I've just killed the Pixelserv-tls stuff (was top at ~2% but not sure it's the culprit) : i just recovered a few megabytes free (and access to UI), but i don't think it was the culprit

CPU seems OK

In syslog (attached) there are a lot of memory errors, but it's just one process encountering the "'no memory anymore available" symptom so probably not the culprit)



Thanks in advance for any ideas / help and/or directions about which tools are available to diagnose the issues on Asus routers (with Merlin activated).

It's normal for asus newer codes to exhaust the memory up to 97% then it wil start to clear/manage older caches when more is needed. My ax88u works no problem as intended. For now it looks like your ax6000 isn't managing the memory right, use diversion's drop caches during blocking list update(b,3,2) option to clear caches, set it twice a week as a work around until asus fix the bug.

 

[SR-G]

Yeah i don't really care about the 97% of memory being taken, but i care more about the web UI being not accessible + SSH not accessible.

Ok, i also reconfigured diversion per your recommendation, let's see

 

[jtp10181]

Out of Memory Errors 388.1 / No space left on device [SOLUTIONS]

Update 3/23/2023 - FW 388.2 B1 So far it looks like 388.2 B1 has fixed the issue with networkmap. I have turned off my cron job and its going on 2 days now, the mem use from networkmap is stable. Highlights of tips from this tread: - Script to restart networkmap and sysstate every night. I...

www.snbforums.com

 

[null_]

While my installed apps are different, I see the same memory usage on my GT-AX6000 as you're seeing (and am also running 388.1). I'm not sure the memory utilization is the issue - it's always been this way for mine and I've never seen any major problems since I got everything finally configured. I specifically remember dropping Diversion for AdGuard Home... though I can't remember what exact issue I was running into. I might start with a clean 388.1 install, then add things one at a time until the issue crops up again? Good luck!

 

[Tech9]

I have no clue what's going on !

About 420MB of your RAM is taken by cache. Do you have USB share (Samba) enabled by chance?

 

[naznaznaz]

The larger the size of the memory inserted into the usb port in Asus routers, the higher the ram usage. If you don't need it, I recommend reducing the capacity of the memory you plug into the USB 3.0 port. Also Aiprotection app uses at least 10% ram. Set your device to turn off and on automatically every day to free up memory. If you don't have a device using the 2.4 wifi network, you can keep the network off all the time. The RAM usage of the AX6000 in stable condition varies between 58% and 64%.

 

[Tech9]

The larger the size of the memory inserted into the usb port in Asus routers, the higher the ram usage.

This is not true. The router will use all available RAM as cache with 8GB USB stick and 2TB HDD drive and not immediately, but when requested on file transfers. Over time this cache will be reduced when USB drive is idle or used for something else. This is completely normal and expected behavior.

 

[naznaznaz]

While the web interface is open, you can check whether it is correct by inserting a memory into the usb 3.0 port and then removing it. When I insert a 500 gb ssd, the ram usage instantly increases to around 70%, and when I remove the memory, it drops to 60%. You can try. You can try the same situation by turning Aiprotection on and off. I tried many times.

 

[naznaznaz]

When I install 8gb capacity memory, ram usage increases by 1 or 2%. It falls off when you take it out. Version 386.7_2

 

[Tech9]

I have checked it many times on many different models and firmware and the moment you start file transfer the router will use all available RAM as cache up to about 97% no matter how large the USB drive is. Just plugging it in and removing it doesn't show you what happens when you start using it.

 

[ColinTaylor]

Over time this cache will be reduced when USB drive is idle or used for something else.

This is incorrect. The disk cache will only reduce in size if an object being cached no longer exists (e.g. it's deleted) or the cache is deliberately flushed.

 

[Tech9]

You're right - my description was incorrect. Not USB drive used for something else, but RAM needed for something else.

 

[naznaznaz]

The fact that the memory is plugged into the usb slot increases the ram usage according to the size of the memory, even if the memory is not used. Of course, when you start using the memory, the ram usage goes up to 97% and 99%. In this normal, non-normal stable state, the ram usage increases and remains there only because the memory is inserted into the usb slot. Even after using any device memory on the network, it is not immediately possible for the device to stabilize. Ram usage stays around 97% and 99% for a long time. An unused memory installed in the usb slot increases ram usage even though it is not used in the network.

 

[Tech9]

I don't know if it works on GT-AX6000, but if this:

free && sync && echo 3 > /proc/sys/vm/drop_caches && free

...recovers the free RAM for @SR-G - this router has no issues.

 

[ColDen]

The larger the size of the memory inserted into the usb port in Asus routers, the higher the ram usage. If you don't need it, I recommend reducing the capacity of the memory you plug into the USB 3.0 port. Also Aiprotection app uses at least 10% ram. Set your device to turn off and on automatically every day to free up memory. If you don't have a device using the 2.4 wifi network, you can keep the network off all the time. The RAM usage of the AX6000 in stable condition varies between 58% and 64%.

 

[naznaznaz]

 

[naznaznaz]