Out of Memory Errors 388.1 / No space left on device [SOLUTIONS]

[jtp10181]

Update 3/23/2023 - FW 388.2 B1
So far it looks like 388.2 B1 has fixed the issue with networkmap. I have turned off my cron job and its going on 2 days now, the mem use from networkmap is stable.

---

Highlights of tips from this tread:
- Script to restart networkmap and sysstate every night. I found sysstate is not usually running and seems to be much less of an issue so I am only killing networkmap now.
https://www.snbforums.com/threads/o...8-1-no-space-left-on-device.82406/post-814204

- Cron job you can run to log networkmap memory use once an hour, and the post right after for how to log /tmp size every hour:
https://www.snbforums.com/threads/o...8-1-no-space-left-on-device.82406/post-818366

- Script to clean up the /tmp/.diag files and log the results (only needed if you find your /tmp is filling up, not everyone is affected by this)
Full Instructions: https://www.snbforums.com/threads/o...8-1-no-space-left-on-device.82406/post-820347
Updated to include code to also shrink the files when done: https://www.snbforums.com/threads/o...ce-left-on-device-solutions.82406/post-823395
Further customized version and info: https://www.snbforums.com/threads/o...ce-left-on-device-solutions.82406/post-823400

- How to monitor RAM/CPU if you have an local server that can host a database:
https://www.snbforums.com/threads/o...8-1-no-space-left-on-device.82406/post-820843

---

ORIGINAL POST BELOW:

RT-AX68U

So I noticed my router kept using swap, which I had never seen it use before. So I removed the swap file.
Here is my other post: https://www.snbforums.com/threads/388-1-sysstate-is-chewing-up-memory.82328/

Now I am getting out of memory errors after running for a few days. Had to reboot and then it was back to normal for now.

Any issues in downgrading back to the previous firmware?
Anything else I should look at?

Dec 18 12:42:10 RT-AX68U-0DB8 conn_diag: Error locking /var/lock/sys_detect.lock: 28 No space left on device
Dec 18 12:42:10 RT-AX68U-0DB8 conn_diag: Error locking /var/lock/wifi_detect.lock: 28 No space left on device
Dec 18 12:42:10 RT-AX68U-0DB8 conn_diag: Error locking /var/lock/wifi_detect.lock: 28 No space left on device
Dec 18 12:42:10 RT-AX68U-0DB8 conn_diag: Error locking /var/lock/net_detect.lock: 28 No space left on device
Dec 18 12:42:10 RT-AX68U-0DB8 conn_diag: Error locking /var/lock/eth_detect.lock: 28 No space left on device
Dec 18 12:42:10 RT-AX68U-0DB8 conn_diag: Error locking /var/lock/eth_detect.lock: 28 No space left on device
Dec 18 12:42:10 RT-AX68U-0DB8 conn_diag: Error locking /var/lock/port_status_usb_change.lock: 28 No space left on device
Dec 18 12:42:11 RT-AX68U-0DB8 dnsmasq-dhcp[9132]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 8 s)
Dec 18 12:42:17 RT-AX68U-0DB8 hostapd: wl0.2: STA 2c:aa:8e:b9:75:1a IEEE 802.11: associated
Dec 18 12:42:17 RT-AX68U-0DB8 hostapd: wl0.2: STA 2c:aa:8e:b9:75:1a RADIUS: starting accounting session 2A3CD10ED856AD2D
Dec 18 12:42:17 RT-AX68U-0DB8 hostapd: wl0.2: STA 2c:aa:8e:b9:75:1a WPA: pairwise key handshake completed (RSN)
Dec 18 12:42:19 RT-AX68U-0DB8 dnsmasq-dhcp[9132]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 9 s)
Dec 18 12:42:20 RT-AX68U-0DB8 out of memory [1165]
Dec 18 12:42:20 RT-AX68U-0DB8 out of memory [1165]
Dec 18 12:42:28 RT-AX68U-0DB8 dnsmasq-dhcp[9132]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 8 s)

I am not running much for scripts. Diversion is just loaded for update checks and some settings, it is turned off for ad blocking

 

[Tech9]

I would remove nsrum (obsolete) and spdMerlin (needs RAM to run and stresses the ISP line for no reason) and try again.

 

[jtp10181]

Odd thing is I had this same setup on an AC56U and it worked fine, and also on this AX68U on prior firmware, it never used the swap file before.
I enabled the swap file again just so it can keep working if it needs it, removed those two scripts you suggested.

Memory looks about the same with those two removed and rebooting but maybe not running the speed test will help. I did have the speed test only running once a day, I think the default was much more often. I will keep an eye on it and see if it starts swapping again.

Also, NVRAM and JFFS has always had plenty of free space, so I assume all the errors are related to RAM even though it is complaining about writing files.

 

[Dr Mono]

This is a controversial one. Few folks have reported the same issue with 388 firmware (multiple routers). However, RMerlin and Gnuton don't think it is a memory leak, but Asus changing how free memory is calculated/displayed. To play it safe, I'm still running the most recent 386.

 

[John Fitzgerald]

RT-AX68U

So I noticed my router kept using swap, which I had never seen it use before. So I removed the swap file.
Here is my other post: https://www.snbforums.com/threads/388-1-sysstate-is-chewing-up-memory.82328/

Now I am getting out of memory errors after running for a few days. Had to reboot and then it was back to normal for now.

Any issues in downgrading back to the previous firmware?
Anything else I should look at?

Dec 18 12:42:10 RT-AX68U-0DB8 conn_diag: Error locking /var/lock/sys_detect.lock: 28 No space left on device
Dec 18 12:42:10 RT-AX68U-0DB8 conn_diag: Error locking /var/lock/wifi_detect.lock: 28 No space left on device
Dec 18 12:42:10 RT-AX68U-0DB8 conn_diag: Error locking /var/lock/wifi_detect.lock: 28 No space left on device
Dec 18 12:42:10 RT-AX68U-0DB8 conn_diag: Error locking /var/lock/net_detect.lock: 28 No space left on device
Dec 18 12:42:10 RT-AX68U-0DB8 conn_diag: Error locking /var/lock/eth_detect.lock: 28 No space left on device
Dec 18 12:42:10 RT-AX68U-0DB8 conn_diag: Error locking /var/lock/eth_detect.lock: 28 No space left on device
Dec 18 12:42:10 RT-AX68U-0DB8 conn_diag: Error locking /var/lock/port_status_usb_change.lock: 28 No space left on device
Dec 18 12:42:11 RT-AX68U-0DB8 dnsmasq-dhcp[9132]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 8 s)
Dec 18 12:42:17 RT-AX68U-0DB8 hostapd: wl0.2: STA 2c:aa:8e:b9:75:1a IEEE 802.11: associated
Dec 18 12:42:17 RT-AX68U-0DB8 hostapd: wl0.2: STA 2c:aa:8e:b9:75:1a RADIUS: starting accounting session 2A3CD10ED856AD2D
Dec 18 12:42:17 RT-AX68U-0DB8 hostapd: wl0.2: STA 2c:aa:8e:b9:75:1a WPA: pairwise key handshake completed (RSN)
Dec 18 12:42:19 RT-AX68U-0DB8 dnsmasq-dhcp[9132]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 9 s)
Dec 18 12:42:20 RT-AX68U-0DB8 out of memory [1165]
Dec 18 12:42:20 RT-AX68U-0DB8 out of memory [1165]
Dec 18 12:42:28 RT-AX68U-0DB8 dnsmasq-dhcp[9132]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 8 s)

I am not running much for scripts. Diversion is just loaded for update checks and some settings, it is turned off for ad blocking

View attachment 46449

Is Entware installed properly, (ep) to update???

 

[jtp10181]

Yes Entware is installed and updated.
Just logged into the UI and count the network clients list said 0, then after clicking a few pages I got logged out, then when I logged in it was working again. I suspect the httpd or networkmap process crashed?

Checked my logs and this was about 15 minutes ago and repeated multiple times for 5 minutes. What interface/radio is wl1? I checked 'ip a' and I only see wl0 listed.
I have also noticed since updating the 388.1 that my alexa devices will randomly ALL go offline at least once every day, possibly when this crashing is happening?

I also ran at least one of the beta versions, I only noticed the issues after updating to the final 388.1 but it could have been there lurking in the beta and I never noticed.

@RMerlin is there any other info I can get off my router that would be useful before I revert back to 386?

Dec 18 20:30:20 RT-AX68U-0DB8 kernel: wl1: PSMx dump at 24487 seconds. corerev 131 reason:watchdog ucode revision 1626.38205 features 0x3106
Dec 18 20:30:20 RT-AX68U-0DB8 kernel: psmxdebug 0x0000050f pc_x 0x01ff001f macctl_x 0x403 maccmd_x 0x0
Dec 18 20:30:20 RT-AX68U-0DB8 kernel: psmx_brc 0x01c9 psmx_brc_1 0x0000 MX_UCODE_DBGST 0x2
Dec 18 20:30:20 RT-AX68U-0DB8 kernel: PC (psmdebug_x):
Dec 18 20:30:20 RT-AX68U-0DB8 kernel: 0x50f      0x50f      0x50f      0x50f    
Dec 18 20:30:20 RT-AX68U-0DB8 kernel: PC (psmdebug):
Dec 18 20:30:20 RT-AX68U-0DB8 kernel: 0x1ff0023  0x19301fd  0x1930063  0x1ff06b8
Dec 18 20:30:20 RT-AX68U-0DB8 kernel: 0x1ff0023  0x19301fd  0x1930063  0x1ff06f2
Dec 18 20:30:20 RT-AX68U-0DB8 kernel: PSMX_STACK_STATUS : 0x8101
Dec 18 20:30:20 RT-AX68U-0DB8 kernel: PSMX_STACK_ENTRIES:
Dec 18 20:30:20 RT-AX68U-0DB8 kernel: 0x1b02
Dec 18 20:30:20 RT-AX68U-0DB8 kernel: 0x050f
Dec 18 20:30:20 RT-AX68U-0DB8 kernel: 0x035f
Dec 18 20:30:20 RT-AX68U-0DB8 kernel: 0x07e3
Dec 18 20:30:20 RT-AX68U-0DB8 kernel: 0x19b3
Dec 18 20:30:20 RT-AX68U-0DB8 kernel: 0x3d85
Dec 18 20:30:20 RT-AX68U-0DB8 kernel: 0x0952
Dec 18 20:30:20 RT-AX68U-0DB8 kernel: 0x1345
Dec 18 20:30:20 RT-AX68U-0DB8 kernel: txe_vasip_intsts 0x0
Dec 18 20:30:20 RT-AX68U-0DB8 kernel: wl1: fatal error, reinitializing, total count of reinit's[1]
Dec 18 20:30:20 RT-AX68U-0DB8 kernel: CSIMON:  CSIMON[1.1.0] Initialization
Dec 18 20:30:20 RT-AX68U-0DB8 kernel: CSIMON: M2M usr already registered ...
Dec 18 20:30:20 RT-AX68U-0DB8 kernel: wl1: PSMx dump at 24487 seconds. corerev 131 reason:watchdog ucode revision 1626.38205 features 0x3106
Dec 18 20:30:20 RT-AX68U-0DB8 kernel: psmxdebug 0x0000050f pc_x 0x01ff001f macctl_x 0x403 maccmd_x 0x0
Dec 18 20:30:20 RT-AX68U-0DB8 kernel: psmx_brc 0x01c9 psmx_brc_1 0x0000 MX_UCODE_DBGST 0x2
Dec 18 20:30:20 RT-AX68U-0DB8 kernel: PC (psmdebug_x):
Dec 18 20:30:20 RT-AX68U-0DB8 kernel: 0x50f      0x50f      0x50f      0x50f    
Dec 18 20:30:20 RT-AX68U-0DB8 kernel: PC (psmdebug):
Dec 18 20:30:20 RT-AX68U-0DB8 kernel: 0x1ff0023  0x1fd01ff  0x1ff006c  0x1930709
Dec 18 20:30:20 RT-AX68U-0DB8 kernel: 0x1ff0023  0x1fd01ff  0x1ff006a  0x93071d
Dec 18 20:30:20 RT-AX68U-0DB8 kernel: PSMX_STACK_STATUS : 0x8101
Dec 18 20:30:20 RT-AX68U-0DB8 kernel: PSMX_STACK_ENTRIES:
Dec 18 20:30:20 RT-AX68U-0DB8 kernel: 0x1b02
Dec 18 20:30:20 RT-AX68U-0DB8 kernel: 0x050f
Dec 18 20:30:20 RT-AX68U-0DB8 kernel: 0x035f
Dec 18 20:30:20 RT-AX68U-0DB8 kernel: 0x07e3
Dec 18 20:30:20 RT-AX68U-0DB8 kernel: 0x19b3
Dec 18 20:30:20 RT-AX68U-0DB8 kernel: 0x3d85
Dec 18 20:30:20 RT-AX68U-0DB8 kernel: 0x0952
Dec 18 20:30:20 RT-AX68U-0DB8 kernel: 0x1345
Dec 18 20:30:20 RT-AX68U-0DB8 kernel: txe_vasip_intsts 0x0
Dec 18 20:30:20 RT-AX68U-0DB8 kernel: wl1: fatal error, reinitializing, total count of reinit's[2]
Dec 18 20:30:20 RT-AX68U-0DB8 kernel: CSIMON:  CSIMON[1.1.0] Initialization
Dec 18 20:30:20 RT-AX68U-0DB8 kernel: CSIMON: M2M usr already registered ...
Dec 18 20:30:20 RT-AX68U-0DB8 acsd: eth6 received event: tx pkt delay suddently jump

 

[RMerlin]

@RMerlin is there any other info I can get off my router that would be useful before I revert back to 386?

Look at the content of /tmp/ , this is the first place that may accidentally fill up and lead to out of space errors. You can also monitor memory usage by process by checking with top.

 

[jtp10181]

I will have to wait and see if it happens again, currently /tmp looks fine.
I was also using htop, in the other thread I linked to in the my post which may be related. Nothing was jumping out at me besides the sysstate process which seems to run for a while then go away. Killing any of the processes using a lot of RAM does not seem to make any difference in the free RAM.

Something is going on with this 388 firmware but I cannot figure out what it is. None of this happened before updating.

 

[nickie]

I’ve noticed and already reported ram issues with the 388 firmware and my tuf-ax5400. After about 10/12 days of usage I start getting some instability and out of memory messages. This happened with dirty upgrade from 386 and also full reset with clean update. Went back to 386 and never had a problem after 16 days with free ram at about 105/115mb. I think I saw something similar being reported by et8 or xt8 users.

 

[John Fitzgerald]

I will have to wait and see if it happens again, currently /tmp looks fine.
I was also using htop, in the other thread I linked to in the my post which may be related. Nothing was jumping out at me besides the sysstate process which seems to run for a while then go away. Killing any of the processes using a lot of RAM does not seem to make any difference in the free RAM.

Something is going on with this 388 firmware but I cannot figure out what it is. None of this happened before updating.

View attachment 46466

Entware has a new release today. Check and see if that helps.

 

[jtp10181]

Uptime of 22 hours and fatal errors again. According to this error it has retried whatever it is doing 739 times. Scrolling up it just goes on forever since about an hour ago.

No file space issues

Has not started using swap:

htop sorted by memory use

 

[RMerlin]

Are you also seeing OOM errors? If not, you might be suffering from two different issues, this one being a hardware issue with the wifi radio.

 

[jtp10181]

Are you also seeing OOM errors? If not, you might be suffering from two different issues, this one being a hardware issue with the wifi radio.

Searched the main log for OOM and nothing, did a grep search on all logs, also found nothing.
Is it worth nothing even after a fresh reboot 'ifconfig' does not even show a wl1 interface? Is that possibly for routers with 2x 5Ghz bands? I believe the AX68U is just a single 5Ghz, I only set one 5Ghz channel.

Looking like I should go back to 386 and see if it continues? Like I said this only started with 388.1

Also, I have not seen the out of memory return since I enabled the swap again, but it has not used the swap either.

 

[RMerlin]

Is it worth nothing even after a fresh reboot 'ifconfig' does not even show a wl1 interface?

The interface names vary between models, but it will be ethx, not wlx. To view the interface name for the second radio:

 nvram get wl1_ifname

Looking like I should go back to 386 and see if it continues?

Try an electrical reset instead. Unplug the power with the power switch still turned on. Wait 5 -10 secs, then turn the router off, plug the power back in, then turn it back on.

 

[jtp10181]

OK, so it is eth6, and based on some of the other info I got using 'nvram show | grep wl1' I can tell it is the 5Ghz band. Which, was still working while it was in its crashing loop. It stopped some time overnight and seems to fine for now.

Going to pull power on it this morning and then see how it goes. I had done a shutdown before but I used the switch, I see how pulling the power would be better, have to do the same thing on my home automation hub if the radio starts acting up to power it off fully.

 

[jtp10181]

Uptime almost two days now and no wl1 crashing since I did the electrical reset. Not seeing any swap use either.
I had disabled the AiProtection in case it was eating up extra memory, going to turn that back on and see if it starts swapping again eventually.
Possibly was all related to the hardware issue.

 

[jtp10181]

Uptime almost 7 days now. I noticed it started to use swap agian. Checking htop the networkmap process is using tons of memory. If I did not have the swap file I am guessing things would have crashed again from running out of memory. It is still running fine though, and has been much better since the "electrical reset".

 

[ColinTaylor]

networkmap and sysstate are using way too much memory. So unless your network topology or devices is unusual (e.g. very large subnet, vlans, etc.) I'd say this is a bug.

However, I can't say I've noticed any other RT-AX68U users reporting such a problem.

 

[jtp10181]

networkmap and sysstate are using way too much memory. So unless your network topology or devices is unusual (e.g. very large subnet, vlans, etc.) I'd say this is a bug.

However, I can't say I've noticed any other RT-AX68U users reporting such a problem.

That's what I am suspecting is some sort of bug. I have around 45 devices connected at all times, peak of around 50-52. I have 20 static IPs set in DHCP.
I have 2.4Ghz Guest 1 (isolation on) and Guest 2 (isolation off) enabled. Guest 1 is for true guests, usually nothing connected. Guest 2 is for IoT devices mainly just so I can force them over to the 2.4Ghz band easily (some get confused with the "Smart Connect" 2.4 and 5 sharing the same SSID).

I have seen people saying they had issues until they did a factory reset, I think on the AX86. I don't think a lot of people are using the AX68. So that may be another thing I could try but I dont do it often so it will take me hours to do it I'm sure and would be dreading it.

 

[nickie]

Some tuf-ax5400 users have experienced something similar in their routers. In my case I noticed this twice (first upgrade with no reset and then with a clean reset) where the router keeps showing less and less memory available until it only has abou 30mb free and starts behaving erratically. Going back to the last 386 based firmware has solved the issue (it has been 1 month since I reversed the firmware).