Out of Memory Errors 388.1 / No space left on device [SOLUTIONS]

[jtp10181]

Just updating again to track the memory use. networkmap seems to be slowly creeping up and sysstate is level at around 8mb.

From what I have read networkmap was changed to closed source at some point and RMerlin is not so much interested in fixing bugs for Asus for free anymore (don't blame him).
So likely this is an Asus issue. I do not have enough motivation to fight that fight with a large company. Hopefully enough people complain to Asus and they fix it in the next release.
I will keep watching it to see how high it goes.

Side thought: Is there a safe way to restart that process? I have killed it before and it does come back but I think something was wonky afterwards.

 

[ColinTaylor]

Side thought: Is there a safe way to restart that process? I have killed it before and it does come back but I think something was wonky afterwards.

You could just setup a cron job that kills the networkmap process in the middle of the night. The watchdog process will restart it within a couple of minutes.

I've not noticed any wonkiness after doing that but then I don't have your problem and have only done it a few times.

 

[jtp10181]

You could just setup a cron job that kills the networkmap process in the middle of the night. The watchdog process will restart it within a couple of minutes.

I've not noticed any wonkiness after doing that but then I don't have your problem and have only done it a few times.

Ok thanks maybe it was after I killed the sysstate I had issues, but I think it was also when the wl1 was crashing so I will have to try it again.
Going to let it stew for a few more days just to see how bad it gets then probably resort to just killing one or both of those processes daily.

 

[MrRojano97]

As an user affected by this issue on my TUF-AX5400 using the official Asus Firmware, I can confirm that the issue is still present on Gnuton fork of Merlin

 

[jtp10181]

So after further investigation I have also found other issues with networkmap. When restarting it, it gets errors trying to create lock files due to using an invalid path. Also gets errors when you sigterm it and it attempts to unlock itself. I decided easiest thing was to just create that path, which I did and then it populated lock files there. Not hopeful but going to test this out and see if it fixes it alone without killing the process every night.

This is what I am adding at boot, in the init-start script file
Going to test it without the networkmap cron job first to see if the lock files alone makes a difference.

# Fix for networkmap and Restart processes daily
mkdir -p /var/lock/usr/networkmap
killall -v networkmap
cru a kill_sysState "14 2 * * * killall -v sysstate"
cru a kill_NetworkMap "18 2 * * * killall -v networkmap"

 

[dave14305]

You can try watching what networkmap is doing.

touch /tmp/NMP_DEBUG /tmp/NMP_DEBUG_MORE /tmp/NMP_DEBUG_FUNCTION /tmp/NMP_DEBUG_VC /tmp/conn_debug /tmp/SM_DEBUG
networkmap

Ctrl-C once it starts repeating the scan over and over. maybe something out of the ordinary will stand out.

 

[jtp10181]

Looks like networkmap is still creeping up in Resident memory use at about the same rate as before.
Guess I will kick in my cron job to restart it regularly until Asus has this sorted out.

 

[ColinTaylor]

This is what I am adding at boot, in the init-start script file
Going to test it without the networkmap cron job first to see if the lock files alone makes a difference.

# Fix for networkmap and Restart processes daily
mkdir -p /var/lock/usr/networkmap
killall -v networkmap
cru a kill_sysState "14 2 * * * killall -v sysstate"
cru a kill_NetworkMap "18 2 * * * killall -v networkmap"

There's no real point specifying verbose output (-v) in an automated script or cron as the output isn't going anywhere (unless you redirect it).

 

[jtp10181]

There's no real point specifying verbose output (-v) in an automated script or cron as the output isn't going anywhere (unless you redirect it).

Yeah I realize that now after seeing it is not logging it, I was hoping it would just dump it to the log so I could see that it ran. Is there an easy way to direct it to do that with one of those fancy redirect commands.

 

[ColinTaylor]

Yeah I realize that now after seeing it is not logging it, I was hoping it would just dump it to the log so I could see that it ran. Is there an easy way to direct it to do that with one of those fancy redirect commands.

Something like this:

killall -v networkmap 2>&1 | logger

 

[jata]

Happy New Year all..

I have been watching this thread as I have a similar issue with my AX86S on 388.1. It has been happening since 388 beta and with 388.1 both with a dirty upgrade and full reset.

I have virgin merlin setup on main router. No scripts, no hard drive, no swap file. I have 2x AX58U mesh nodes both running latest stock.

Memory usage creeps up over approx. 5 days. then OOM and crash/reboot then back to normal for a few days... Repeat.

I'm sticking with 388.1 as it is working fine (apart from this issue) and 388.1 seems to have magically solved all the issues I had over the last 2 years with apple homekit / homepods.

Happy to help troubleshoot and/or confirm this is the same underlying issue.

@jtp10181 - can you please share your final/latest kill networkmap script please?

 

[jtp10181]

@jtp10181 - can you please share your final/latest kill networkmap script please?

Was actually just thinking about giving an update today. Found out the native killall is BusyBox and does not support -v (verbose). I have an entware package that supports it but you need to specify the full path to it in the cron job. This is definitely helping, killing this process every day.

So this should work for anyone, insert this into /jffs/init-start (then run the commands manually or reboot to apply it)

# Fix for networkmap and Restart processes daily
mkdir -p /var/lock/usr/networkmap
cru a kill_sysState "14 2 * * * killall sysstate"
cru a kill_NetworkMap "18 2 * * * killall networkmap"

If you want to install the psmisc entware package you can have it run in verbose and log it

opkg install psmisc

Then swap out the two cru commands for this:

cru a kill_sysState "14 2 * * * /opt/bin/killall -v sysstate 2>&1 | logger"
cru a kill_NetworkMap "18 2 * * * /opt/bin/killall -v networkmap 2>&1 | logger"

 

[jata]

Thanks.

Yesterday I did a full clean install of 388.1 with minimum config (fixed both wifi channels, cake QOS, a few static/manual dhcp clients, added 2x nodes)

So now I wait to see if I get a crash due to OOM - not using a swap so I expect system to crash/reboot as I have seen this happen since 388

If I confirm the crash issue, I will implement the script and set it to kill just networkmap (not sysstate) and see if this is enough...

 

[jtp10181]

If you want to avoid waiting for it to crash, install 'htop' in entware, run that and then sort by M_RESIDENT. networkmap will probably be at the top. Under 10k RES seems to be normal, after a day it will be over 10k and after a week it will be approaching 50k in my experience. After 2 days you will easily be able to see if it keeps creeping up and know its going to crash eventually if you let it run its course.

One further tip, if you kill networkmap manually, the web interface will not work correctly until you log out and back in again. Killing it in the middle of the night solves this problem since you likely wont be logged in.

 

[jata]

Thanks for the tips. I am using top to monitor and I am able to see the memory % increase/change so I will keep an eye on it.

Trying to keep everything super clean so no swap/usb drive/entware/scripts etc.

Honestly, my main concern is keeping all my homekit / homepod etc all working smoothly - as they are right now...

 

[AlbumMan]

Happy New Year all..

I have been watching this thread as I have a similar issue with my AX86S on 388.1. It has been happening since 388 beta and with 388.1 both with a dirty upgrade and full reset.

I have virgin merlin setup on main router. No scripts, no hard drive, no swap file. I have 2x AX58U mesh nodes both running latest stock.

Memory usage creeps up over approx. 5 days. then OOM and crash/reboot then back to normal for a few days... Repeat.

I'm sticking with 388.1 as it is working fine (apart from this issue) and 388.1 seems to have magically solved all the issues I had over the last 2 years with apple homekit / homepods.

Happy to help troubleshoot and/or confirm this is the same underlying issue.

@jtp10181 - can you please share your final/latest kill networkmap script please?

Similar issue here and was mentioned in the Release thread a couple of weeks ago. Since there is now an OOM thread, I just wanted to jump in too.
RT-AX3000 (aka AX58U), simple config, no mesh, no scripts, no hd, etc. Am on day 29 and what started off as 51% RAM usage, is now up to 72%; the highest the router has ever been. Unlike the others, the logs are good (no errors) and has not ran out of memory or rebooted. I may wait to see how high it will climb before doing a power cycle (pull plug, etc) as suggested. I'm glad to see others seeing the same as it appears to have been introduced in 388.1.

 

[jtp10181]

Am on day 29 and what started off as 51% RAM usage, is now up to 72%

I don't know how you pull that off, 2 minutes after I boot my AX68U I am at only 30Mb free RAM out of 512Kb, same RAM specs as your AX58U. I do have a bunch of static DHCP reservations and custom names and icons for all 50 or so clients. I have tried disabling scribe and some other things but the RAM still all gets used very quickly. It is possibly all the trend-micro stuff, if you have all that disabled, that could be it. I tried disabling some of it but the processes still launched, I think you have to un-confirm the privacy settings to totally shut it all down.

What really blows my mind is I ran nearly this same setup on an AC56U with 256Kb RAM and it never used swap. When I switched to the AX68U I expected to be running at 50% RAM initially but even with v386 firmware something just chews up all the RAM it can leaving a small extra buffer. Its definitely worse on 388 though with this networkmap mem leak.

 

[jtp10181]

So far so good with killing networkmap every night. Have also noticed my devices (especially Alexa) have been dropping off the network a lot less.
I may bump it up to killing it twice a day and see if that further helps the Alexa devices from dropping off.

I know the Alexa devices are dropping because I will get alerts for every echo in the house nearly all at once that they are offline. If I go to check any they are already back online so it must just be for a few seconds they get disconnected. This also only started happening with 388 fw.

 

[jata]

@jtp10181 - That is a pain if you are getting device disconnections. I think it might be a different issue that the memory related one - potentially wifi drivers used on 388.1. Why do you not go back to 386.5_2? This is a very stable FW for AX86U

I was working on my NAS and AdGuard servers yesterday and accidentally unplugged my router so the memory/reboot clock has been reset so I have no new information to report.

 

[jtp10181]

I have thought about reverting but it seems to be working pretty good now. I have not noticed any actual device issues, my laptop never gets disconnected for example. There is just something goofy with the Alexa devices and I am not sure if they even really get disconnected or just Amazon thinks they are for some reason.