failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)

[Marko Polo]

Were there any other "failed to write" messages in the log?

No, only /var/lib/misc/dnsmasq.leases.

find /tmp -exec ls -de {} \; | grep -v " /tmp/mnt/" outputs hundreds of such messages:

mmap of a spare page failed!
mmap of a spare page failed!
mmap of a spare page failed!
mmap of a spare page failed!
mmap of a spare page failed!

 

[ColinTaylor]

Ouch! Looks like you've run out of RAM.

What does this give you?

free

 

[Marko Polo]

What does this give you?

user@user:/tmp# free
             total       used       free     shared    buffers     cached
Mem:        255700     211080      44620          0       1608      86312
-/+ buffers/cache:     123160     132540
Swap:            0          0          0

 

[Marko Polo]

Looks like you've run out of RAM.

How was I able to achieve that? It's the first time I've faced it. How to detect the cause?

 

[ColinTaylor]

How was I able to achieve that? It's the first time I've faced it. How to detect the cause?

It was caused because the command you issued had to buffer the file list in memory before piping it through to the next command.

What's really strange is that according to free you still have plenty of memory available. That's the same conundrum as in post #2.

 

[Marko Polo]

Seems to be a memory overflow bug.

Odd, since the router does report there's still 82 MB of free RAM (plus 102 MB of cache that would be freed when required).

Yep, Merlin?

 

[ro0ter]

Same problem here... nvram is filled to the top and I must say that there are sume funny enrties (besides the fact that the nvram summary does not sup up to what the router interface says).

ASUS RT-AC68U

Main problem that had me writing here is the yellow exclamation sign that was informing me regarding nvram avilability (appeared after upgrade to 384.14):

Your router is running low on free NVRAM, which might affect its stability.<br>Review long parameter lists (like DHCP reservations), or consider doing a factory default reset and reconfiguring.
Review System Information now

yes, I`ll factory reset it in a minute, first I`ll backup everything (just in case, maybe I need some info later on -- although I can`t read read config files)
​

Logs with respect to this problem were also present:

Dec 31 16:12:31 kernel: nvram: consolidating space!​

Status in web interface is: NVRAM usage 65185 / 65536 bytes

What I get from nvram summary command is:

size: 65184 bytes (352 left)
4813 MULTIFILTER_TMP_T *** what the goose?!?!?
1089 sshd_hostkey
810 nc_setting_conf
624 sshd_dsskey
516 rc_support
515 share_link
411 sshd_authkeys
338 sshd_ecdsakey
205 wl1_chansps
194 custom_clientlist​

The respective field looks quite funny. Any hint on it`s usage? I`ll flush nvram in a second, see what I have after facory reset. Screenshot with its content attached



Besides that I also have the "dnsmasq.leases: No space left on device"... Ram is ok (now), half filled. Router reboot cron job configured every night, it`s much more stable this way.
I only updated on the 31st to 384.14 since the lets encrypt automated certificate renewal no longer worked (issue with new json fields support in client api), plus I always hope for stability improvements, as everyone does

What was also annoying is that the internet would fail with days (unless I manually un/re-plug the brick).. initially the reboot job was every 2-3 days (now every day), yet I had some issues with my home alarm system, it was unreachable for 2 days in a row and I was really away (it will however scream, should it detect motion)... since I have previous experiences with thieves, I was quite worried (neighbours were also away and power cycling my apartment would not fix it, router and amenities can live 10+ hours on ups...). I`m currently looking into solutions with GPRS-based modems, yet I don`t think it would have helped in the given situation... It seems the router ran out of ram, God knows why... I only have samba, ai cloud, transmission (which I just stopped, I`ve seeded some old gimp and open office versions for more than 10 months, lol), 1 vpn server...

On the 24th the router was really unreachable. I have some traces which seem interesting for me. Ivestigating the logs I noticed the following (reboot cron is 03:21):

Dec 26 03:20:20 dnsmasq-dhcp[231]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s) *** this actually floods the logs, it accounts for more than 75% of the logs....
Dec 26 03:21:08 rc_service: watchdog 263:notify_rc reboot
Dec 26 03:21:08 dnsmasq-dhcp[231]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)
Dec 26 03:21:09 pppd[589]: fork failed: Cannot allocate memory
Dec 26 03:21:09 pppd[589]: Failed to create child process for /tmp/ppp/ip-down: Cannot allocate memory
Dec 26 03:21:09 kernel: Attempt to kill tasklet from interrupt
Dec 26 03:21:10 ovpn-server1[3614]: event_wait : Interrupted system call (code=4)
Dec 26 03:21:10 miniupnpd[3786]: Failed to get IP for interface ppp0
Dec 26 03:21:10 miniupnpd[3786]: SendNATPMPPublicAddressChangeNotification: cannot get public IP address, stopping
Dec 26 03:21:11 WAN_Connection: Fail to connect with some issues.
Dec 26 03:21:12 ovpn-server1[3614]: openvpn_execve: unable to fork: Cannot allocate memory (errno=12)​

Some more ram-deprived processes:

Dec 31 12:01:19 smbd[3440]: smbd_accept_connection: sys_fork() failed: Cannot allocate memory
Dec 31 12:01:19 smbd[3440]: [2019/12/31 12:01:19.225982, 0] smbd/server.c:572(smbd_accept_connection)
[...]
Dec 31 13:52:23 dnsmasq-dhcp[237]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)
Dec 31 13:53:14 syslog: WLCEVENTD wlceventd_proc_event(420): eth1: Auth xx:xx:xx:xx:xx:xx, status: 0, reason: d11 RC reserved (0)
Dec 31 13:53:14 syslog: WLCEVENTD wlceventd_proc_event(449): eth1: Assoc xx:xx:xx:xx:xx:xx, status: 0, reason: d11 RC reserved (0)
Dec 31 13:53:15 dnsmasq-dhcp[237]: DHCPDISCOVER(br0) xx:xx:xx:xx:xx:xx
Dec 31 13:53:15 dnsmasq-dhcp[237]: DHCPOFFER(br0) xxx.xxx.xxx.xxx xx:xx:xx:xx:xx:xx
Dec 31 13:53:15 dnsmasq-dhcp[237]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)
Dec 31 13:53:15 dnsmasq-dhcp[237]: DHCPREQUEST(br0) xxx.xxx.xxx.xxx xx:xx:xx:xx:xx:xx
Dec 31 13:53:15 dnsmasq-dhcp[237]: DHCPACK(br0) xxx.xxx.xxx.xxx xx:xx:xx:xx:xx:xx xxxxxxxxxxxxxxxxxx
Dec 31 13:53:15 dnsmasq-dhcp[237]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)
Dec 31 13:53:36 dnsmasq[237]: failed to allocate 1028 bytes
Dec 31 13:54:15 dnsmasq-dhcp[237]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)
[...]
Dec 31 15:06:32 dnsmasq[237]: failed to allocate 1028 bytes
Dec 31 15:06:32 dnsmasq[237]: failed to allocate 1028 bytes
Dec 31 15:06:33 dnsmasq[237]: failed to allocate 1028 bytes
Dec 31 15:06:34 dnsmasq[237]: failed to allocate 1028 bytes
Dec 31 15:06:37 dnsmasq[237]: failed to allocate 1028 bytes
Dec 31 15:07:04 dnsmasq[237]: failed to allocate 1028 bytes
Dec 31 15:07:04 dnsmasq[237]: failed to allocate 1028 bytes
Dec 31 15:07:18 dnsmasq-dhcp[237]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)
Dec 31 15:08:03 dnsmasq[237]: script process exited with status 20
Dec 31 15:08:18 dnsmasq-dhcp[237]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)
Dec 31 15:09:07 dnsmasq[237]: failed to allocate 1028 bytes
[...]
and many others..
​

I am also facing a problem with traffic analyzer. Even after resetting the traffic monitoring settings and reinitializing the file, I still get:

Jan 1 09:10:55 rstats[270]: Problem loading /mnt/Data/Trafic/tomato_rstats_38d5471f9658.gz. Still trying...​

I`ll turn off logs for JFFS`s sake (were on debug for several days), if anything like this ever occurs, I`ll turn them on again.

Any help or hints are greatly appreciated. I have downloaded both config file (for debug) and jffs.

I`ll factory reset it now.

Ah, yes, have a Happy New Year! Shall we all grow younger, be healthier, wealthier, smarter and better than before!

 

[dave14305]

Any help or hints are greatly appreciated. I have downloaded both config file (for debug) and jffs.

MULTIFILTER_TMP_T *** what the goose?!?!?

related to Parental controls. Do you use it? If not:

nvram unset MULTIFILTER_TMP_T
nvram commit

For RAM issues also look for large files in /tmp (which sits in RAM alongside programs):

find /tmp -xdev -exec ls -ld {} + | sort -nk 5

 

[Grisu]

and before factory reset update to 384.14_1

 

[chavo]

killall rstats

use ncdu to find increasing folder

 

[chavo]

need restart samba and delete files
rm -rf /var/lock/*.tdb

 

[DTS]

I'm running into this issue on the latest Merlin fw with an AC86U. The values shown here are before a reboot and while the issue is occurring. Like a couple others in this thread, these results indicate that I am not out of memory.

admin@AC86U-Merlin:/tmp/home/root# df -h
Filesystem                Size      Used Available Use% Mounted on
ubi:rootfs_ubifs         77.2M     64.6M     12.6M  84% /
devtmpfs                214.9M         0    214.9M   0% /dev
tmpfs                   215.0M    260.0K    214.7M   0% /var
tmpfs                   215.0M    900.0K    214.1M   0% /tmp/mnt
mtd:bootfs                4.4M      3.3M      1.1M  75% /bootfs
tmpfs                   215.0M    900.0K    214.1M   0% /tmp/mnt
mtd:data                  8.0M    588.0K      7.4M   7% /data
tmpfs                   215.0M    900.0K    214.1M   0% /tmp
/dev/mtdblock9           47.0M      3.6M     43.4M   8% /jffs
tmpfs                   215.0M    900.0K    214.1M   0% /www/require/modules/menuTree.js

admin@AC86U-Merlin:/tmp/home/root# ls -lh /tmp
-rw-r--r--    1 admin root          70 Dec 13 06:01 2C:FD:A1:A2:17:98.cap
-rw-rw-rw-    1 admin root           0 Dec 13 06:01 addonwebui.lock
-rw-r--r--    1 admin root         227 Dec 22 21:56 allwclientlist.json
-rw-r--r--    1 admin root          72 Dec 13 06:01 aplist.json
drwxrwxrwx    2 admin root          40 May  5  2018 asusfbsvcs
drwxrwxrwx    3 admin root          80 May  5  2018 avahi
-rw-r--r--    1 admin root           0 Dec 22 06:01 blockfile_tmp
drw-rw-rw-    2 admin root          60 May  5  2018 bwdpi
-rw-r--r--    1 admin root         301 Dec 13 06:01 chanspec_all.json
-rw-r--r--    1 admin root         138 Dec 13 06:01 chanspec_avbl.json
-rw-rw-rw-    1 admin root         101 Dec 13 06:01 chanspec_avbl.txt
-rw-r--r--    1 admin root         253 Dec 13 06:01 chanspec_private.json
-rw-r--r--    1 admin root         481 Dec 22 21:57 clientlist.json
drwxr-xr-x    3 admin root          60 Jan  1  1970 confmtd
-rw-r--r--    1 admin root          81 Dec 22 21:57 current_wired_client_list.json
drwx------    2 admin root          60 Dec 22 21:57 db
-rw-rw-rw-    1 admin root        2.7K Dec 22 21:56 dev
lrwxrwxrwx    1 admin root           8 May  5  2018 dhcp6c -> /sbin/rc
drwxrwxrwx    4 admin root          80 May  5  2018 diag_db_cloud
-rwsr-x--T    1 admin root           0 May  5  2018 ebtables.lock
drwxr-xr-x   11 admin root        1.5K Dec 22 08:12 etc
-rw-rw-rw-    1 admin root         857 Dec 14 05:41 filter.default
-rw-rw-rw-    1 admin root         383 Dec 14 05:41 filter_ipv6.default
-rw-rw-rw-    1 admin root        1.9K Dec 14 05:41 filter_rules
-rw-rw-rw-    1 admin root          88 Dec 13 06:01 guest_vsie
drwxr-xr-x    3 admin root          60 Jan  1  1970 home
-rw-rw-rw-    1 admin root          14 Dec 13 06:01 hw_auth_clm
drwxr-xr-x    2 admin root          40 Jan  1  1970 inadyn.cache
-rw-rw-rw-    1 admin root          64 May  5  2018 lld2d.conf
-rw-rw-rw-    1 admin root          62 May  5  2018 lldpd_bind_ifnames
-rw-rw-rw-    1 admin root           5 May  5  2018 mastiff.pid
-rw-rw-rw-    1 admin root           0 May  5  2018 mastiff_log
-rw-rw-r--    1 admin root       24.5K Dec 13 06:01 menuTree.js
drwxrwxrwx    2 admin root          40 Jan  1  1970 mnt
lrwxrwxrwx    1 admin root          24 Dec 14 05:41 nat_rules -> /tmp/nat_rules_eth0_eth0
-rw-rw-rw-    1 admin root         494 Dec 14 05:41 nat_rules_eth0_eth0
drwxrwxrwx    2 admin root         140 May  5  2018 nc
drwxrwxrwx    2 admin root          40 May  5  2018 netool
-rw-r--r--    1 admin root        8.1K Dec 14 23:59 nmp_cache.js
drwxr-xr-x    3 admin root          60 Jan  1  1970 notify
-rw-rw-rw-    1 admin root           1 Dec 13 06:01 obstatus
-rw-rw-rw-    1 admin root          88 Dec 13 06:01 obvsie
drwxrwxrwx    3 admin root         260 May  5  2018 ppp
srwxrwxrwx    1 admin root           0 May  5  2018 ps_sock
-rw-rw-rw-    1 admin root         671 Dec 14 05:41 redirect_rules
-rw-r--r--    1 admin root           2 Dec 13 06:01 relist.json
-rw-r--r--    1 admin root          59 Dec 22 19:15 resolv.conf
-rw-r--r--    1 admin root          17 Dec 22 19:15 resolv.dnsmasq
-rw-rw-rw-    1 admin root         286 May  5  2018 run_lldpd.sh
-rw-r--r--    1 admin root           0 Jan  1  1970 settings
drwxr-xr-x    2 admin root          40 Jan  1  1970 share
-rw-rw-rw-    1 admin root        1.1K Dec 22 21:56 syslog.log
-rw-rw-rw-    1 admin root      256.0K Dec 22 21:55 syslog.log-1
lrwxrwxrwx    1 admin root           8 May  5  2018 udhcpc -> /sbin/rc
-rw-rw-rw-    1 admin root         270 Dec 15 00:01 usb.log
drwxr-xr-x    4 admin root          80 Jan  1  1970 var
-rw-rw-rw-    1 admin root           0 Dec 22 21:57 watchdog_heartbeat
-rw-r--r--    1 admin root         119 Dec 13 06:01 wchannel.json
-rw-rw-rw-    1 admin root          86 Dec 22 03:11 webs_upgrade.log
-rw-r--r--    1 admin root         175 Dec 22 21:57 wiredclientlist.json
lrwxrwxrwx    1 admin root           8 May  5  2018 wpa_cli -> /sbin/rc
lrwxrwxrwx    1 admin root           8 May  5  2018 zcip -> /sbin/rc


admin@AC86U-Merlin:/tmp/home/root# free
             total       used       free     shared    buffers     cached
Mem:        440324     408236      32088       1376          0      18956
-/+ buffers/cache:     389280      51044
Swap:            0          0          0

After rebooting:

 free
             total       used       free     shared    buffers     cached
Mem:        440324     201824     238500       1472          0      18328
-/+ buffers/cache:     183496     256828
Swap:            0          0          0

I wonder if this is related to my vpn issue:

Since upgrading to firmware 386.3_2, my Internet will not stay connected for even a day.

I have been running Merlin firmware on my RT-AC86U since 2018. It's been rock solid, rarely requiring my attention. I've been using ExpressVPN since then as well. Again, no problems. I upgraded to 386.3_2 in October. Since then, my Internet connection has not been reliable. Some of my...

www.snbforums.com

 

[explorer200]

I have been unable to login to a remote AC86U running the latest Merlin firmware 386.7_2 for the past 3 weeks. What was weird though was I was able to access the remote network via OVPN installed on Merlin, but absolutely no access to the UI at 192.168.50.1 or router.asus.com

Last night there was a storm that knocked out the power and eventually the router rebooted, I checked today and was able to log into the UI. I immediately checked the log and it was flooded with:

dnsmasq-dhcp[2606]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60 s)
dnsmasq-dhcp[2606]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60 s)
dnsmasq-dhcp[2606]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60 s)
dnsmasq-dhcp[2606]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60 s)
dnsmasq-dhcp[2606]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60 s)
dnsmasq-dhcp[2606]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60 s)
dnsmasq-dhcp[2606]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60 s)
dnsmasq-dhcp[2606]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60 s)
dnsmasq-dhcp[2606]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60 s)

This has happened a few times in the past. Not running any scripts or anything.

No clue how to prevent it from happening again

 

[gelinski]

Hello,

Same ASUS router model and issue here.
I believe it happens since 386.7_0 version because I didn't got any message like this one on 386.5_2 (and previous).

As it was also impacting the internet connection I reverted it back to the ASUS stock firmware.

:/

 

[chavo]

use ncdu every day
to determine what is eating up space

 

[StR]

I also had this issue in late November.
As I discovered from the log later, for several days in a row ( I saw 3 days, but I didn't see the onset of those , I was getting errors
dnsmasq-dhcp[30491]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60 s)
Those were written in the log every minute - for the entire period of time preserved in the log.
In the end, the router just rebooted by itself.

This is RT-AC86U running 386.7_2. I had not seen it on 386.5_* or earlier releases, and I skipped the earlier 386.7_x versions.
I am not running any extra scripts or packages.
The only difference from before is that I've added since updating to 386.7_2 was the time-based traffic blocking for two devices under parental control (through the Web-UI).

I didn't get to see what was the disk usage before it rebooted, but for the last two weeks, it's been low, and today it looks like this:

# df
Filesystem           1K-blocks      Used Available Use% Mounted on
ubi:rootfs_ubifs         79016     67252     11764  85% /
devtmpfs                220064         0    220064   0% /dev
tmpfs                   220176      1488    218688   1% /var
tmpfs                   220176      3560    216616   2% /tmp/mnt
mtd:bootfs                4480      3364      1116  75% /bootfs
tmpfs                   220176      3560    216616   2% /tmp/mnt
mtd:data                  8192      2276      5916  28% /data
tmpfs                   220176      3560    216616   2% /tmp
/dev/mtdblock9           48128     12764     35364  27% /jffs
/dev/sda1              7888868      1580   7887288   0% /tmp/mnt/ROUTER_USB

It's been about 20 days since the reboot.
At the time of the reboot on Nov. 29 it was running for about 2.5 months since the update was installed on September 17.
So, I do not think something filled up /var partition gradually, on a regular basis. It must have been something very sporadic.

 

[MakeItEasy]

I just got also this error (recurring since 4 hours) with the firmware 3.0.0.4.386_48260 (I do not use Merlin):

May 18 10:32:47 dnsmasq-dhcp[2042]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)

I had lost my Internet connection and a reboot resolved the issue: Internet connectivity is back but it seems the messages are still occurring in the log.

 

[Ripshod]

I just got also this error (recurring since 4 hours) with the firmware 3.0.0.4.386_48260 (I do not use Merlin):

May 18 10:32:47 dnsmasq-dhcp[2042]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)

I had lost my Internet connection and a reboot resolved the issue: Internet connectivity is back but it seems the messages are still occurring in the log.

It's related to this:
https://www.snbforums.com/threads/r...mory-leak-leads-to-lockup-every-10mins.85018/