failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)

[FlyByWire]

Hi there,

running Merlin 378.56_2 I found this message:

dnsmasq-dhcp[19831]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)

flooding the log since this morning 11:30. I attach a snapshot of the memory load from the gui. I have attached a USB harddisk and running just two jffs scripts to overclock (according to the wiki).

Any idea?

Thanks

FlyByWire

 

[RMerlin]

Odd, since the router does report there's still 82 MB of free RAM (plus 102 MB of cache that would be freed when required).

Try looking for more detailed info over ssh, with the following commands:

df -h
ls -lh /tmp
free

 

[FlyByWire]

Meanwhile I did a reboot (via gui) and the messages didn't recur, so I don't know if the that'll be helpfull...

 

[RMerlin]

Try doing the same thing after a few days (depending on how long it took the last time for the problem to occur) so you can compare. What you are seeing now is the state of your router after a fresh reboot. We'd have to compare both situations to determine if something is amiss.

One thing I do note already: you have a fairly large smb.log (19 MB). If you enabled logging through a custom config, you might want to either disable that, or move the log to the USB disk.

 

[FlyByWire]

If you enabled logging through a custom config, you might want to either disable that, or move the log to the USB disk.

Logging is not enabled. I did set up a parental control for one device. Do you think this might cause the large smb log? I'll keep an eye on the log...

 

[RMerlin]

Logging is not enabled. I did set up a parental control for one device. Do you think this might cause the large smb log? I'll keep an eye on the log...

I see that log is enabled by default, however mine is empty. Check what yours contain, something is causing it to fill up quite quickly, judging from its size.

 

[FlyByWire]

The error message didn't recur but smb.log is meanwhile 74 MB(!) showing this content:

Operate= 1
Operate= 2
Operate= 2
Operate= 2
Operate= 2
Operate= 2
Operate= 2
Operate= 2
Operate= 2
Operate= 2

continuing Operate=2 until eof...

 

[mattiL]

I had the same message for the first time this morning:

Nov 17 09:22:30 dnsmasq-dhcp[433]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)

It repeated for about two hours and then stopped.

The memory usage yesterday was in the region of 50% IIRC, and today it's just 20%, and the router has not restarted.

Uptime is:
17:23:52 up 9 days, 8:30, load average: 0.00, 0.01, 0.04

'free' shows:
matti@RT-AC87U-D5E8:/tmp/home/root# free

total used free shared buffers
Mem: 255720 54056 201664 0 2108
-/+ buffers: 51948 203772
Swap: 0 0 0


And 'df -h' shows:
matti@RT-AC87U-D5E8:/tmp/home/root# df -h

Filesystem Size Used Available Use% Mounted on
/dev/root 42.4M 42.4M 0 100% /
devtmpfs 124.8M 0 124.8M 0% /dev
tmpfs 124.9M 5.6M 119.3M 4% /tmp
/dev/mtdblock4 62.8M 1.7M 61.0M 3% /jffs

AC87@378.56_2

 

[rlcronin]

For what its worth, I am seeing the same message in my RT-AC68U logs from time to time. I am running Merlin 378.56_2 as well and am literally doing nothing with any special scripting or any such. No parental control, no overclocking, no nothing. Just the off the shelf Merlin build.
--
bc

 

[scyto]

did this get solved? I was troubleshooting why clients were not appearing on the client list or the DNS reservation list drop down for knew reservations but were getting logged in the DHCP list. I saw the errors the OP describes (i am running 380.57) with 108MB RAM free. I went to turn off anything that is intensive went to adaptive QOS and then clicked traffic manager page and the web page 404'd on me - i saw this in log when i logged back in (i.e error went away - this looks like some sort of buffer get exceeded?) also it corrected the client list being out of date and the client list on the drop down in the DHCP reservation feature being out of date (an:

Feb 18 17:54:48 dnsmasq-dhcp[2074]: DHCPREQUEST(br0) 192.168.1.151 00:1e:c0:10:86:8c
Feb 18 17:54:48 dnsmasq-dhcp[2074]: DHCPACK(br0) 192.168.1.151 00:1e:c0:10:86:8c PCMWIFIG
Feb 18 17:54:48 dnsmasq-dhcp[2074]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 2s)
Feb 18 17:54:50 dnsmasq-dhcp[2074]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 8s)
Feb 18 17:54:55 dnsmasq-dhcp[2074]: DHCPREQUEST(br0) 192.168.1.100 28:18:78:9a:78:4e
Feb 18 17:54:55 dnsmasq-dhcp[2074]: DHCPACK(br0) 192.168.1.100 28:18:78:9a:78:4e XboxOne
Feb 18 17:54:55 dnsmasq-dhcp[2074]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 3s)
Feb 18 17:54:57 dnsmasq-dhcp[2074]: DHCPREQUEST(br0) 192.168.1.151 00:1e:c0:10:86:8c
Feb 18 17:54:57 dnsmasq-dhcp[2074]: DHCPACK(br0) 192.168.1.151 00:1e:c0:10:86:8c PCMWIFIG
Feb 18 17:54:57 dnsmasq-dhcp[2074]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 1s)
Feb 18 17:54:58 dnsmasq-dhcp[2074]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 9s)
Feb 18 17:55:03 dnsmasq-dhcp[2074]: DHCPREQUEST(br0) 192.168.1.100 28:18:78:9a:78:4e
Feb 18 17:55:03 dnsmasq-dhcp[2074]: DHCPACK(br0) 192.168.1.100 28:18:78:9a:78:4e XboxOne
Feb 18 17:55:18 dnsmasq-dhcp[2074]: DHCPREQUEST(br0) 192.168.1.100 28:18:78:9a:78:4e
Feb 18 17:55:18 dnsmasq-dhcp[2074]: DHCPACK(br0) 192.168.1.100 28:18:78:9a:78:4e XboxOne
Feb 18 17:55:26 dnsmasq-dhcp[2074]: DHCPREQUEST(br0) 192.168.1.100 28:18:78:9a:78:4e
Feb 18 17:55:26 dnsmasq-dhcp[2074]: DHCPACK(br0) 192.168.1.100 28:18:78:9a:78:4e XboxOne
Feb 18 17:55:30 watchdog: restart httpd
Feb 18 17:55:30 rc_service: watchdog 454:notify_rc start_httpd
Feb 18 17:55:30 httpd: start httpd
Feb 18 17:55:31 hour monitor: daemon is starting
Feb 18 17:55:34 dnsmasq-dhcp[2074]: DHCPREQUEST(br0) 192.168.1.100 28:18:78:9a:78:4e
Feb 18 17:55:34 dnsmasq-dhcp[2074]: DHCPACK(br0) 192.168.1.100 28:18:78:9a:78:4e XboxOne
Feb 18 17:55:42 dnsmasq-dhcp[2074]: DHCPREQUEST(br0) 192.168.1.100 28:18:78:9a:78:4e
Feb 18 17:55:42 dnsmasq-dhcp[2074]: DHCPACK(br0) 192.168.1.100 28:18:78:9a:78:4e XboxOne

 

[dakotacody]

Not sure if you got your issue fixed or not. I was experiencing this and found a thread where Merlin said it looked to be a memory leak in lighttpd. I disabled AICloud via the web interface and checked that no lighttpd procs were still running. The issue was fixed for me after that. If you don't need AICloud running this may be the easiest fix. I didn't even need to reboot. I don't usually leave AICloud running but just log in via web interface when necessary. He may have fixed it in latest firmware too.

uptime
23:13:10 up 259 days, 8:19, load average: 0.00, 0.00, 0.0

Link to the other post about lighttpd http://www.snbforums.com/threads/memory-allocation-error-write-error.8562/

 

[MartinK]

I have the same problem; running the latest release 380.65; 2 clients connected; nothing enabled; no scripts, no upnp, no samba, no nothing; and still seeing an increase of memory usage over time; cleared log; the problem is currently occurring; rebooting resets the memory usage back to 18%; overtime i'm back at 45% and more.

 

[spalife]

Check with free
If it is due to caching, you can schedule a cron job to periodically clear the cache like every hour
cru a dropCaches "0 * * * * echo 3 > /proc/sys/vm/drop_caches"

 

[Marko Polo]

Still not solved. I am on 384.5. Any suggestions?

Jun 28 14:32:51 dnsmasq-dhcp[28077]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)
Jun 28 14:33:51 dnsmasq-dhcp[28077]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)
Jun 28 14:34:51 dnsmasq-dhcp[28077]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)
Jun 28 14:35:51 dnsmasq-dhcp[28077]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)
Jun 28 14:36:51 dnsmasq-dhcp[28077]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)
Jun 28 14:37:51 dnsmasq-dhcp[28077]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)
Jun 28 14:38:00 dnsmasq[28077]: failed to allocate 1028 bytes
Jun 28 14:38:51 dnsmasq-dhcp[28077]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)

 

[ColinTaylor]

Still not solved. I am on 384.5. Any suggestions?

What is the output of "df -h" when the problem is occurring?

 

[Marko Polo]

user@user:/tmp/home/root# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/root                33.9M     33.9M         0 100% /
devtmpfs                124.8M         0    124.8M   0% /dev
tmpfs                   124.9M     77.9M     46.9M  62% /tmp
/dev/mtdblock4           62.8M      3.9M     58.8M   6% /jffs
/dev/sda1               234.7G    103.9G    118.9G  47% /tmp/mnt/data

It seems hat my root partition is filled up. Is it the reason? How to find out what is filling it?

 

[Marko Polo]

It seems that /dev/root should be an internal memory user has no access to. Isn't it? Something is wrong with router services, supposedly.

 

[ColinTaylor]

root is always 100% because it's in ROM.

/var is symlinked to /tmp/var so we're interested in /tmp

tmpfs 124.9M 77.9M 46.9M 62% /tmp

It's not 100% but 77.9MB used doesn't sound right. There's probably another process that's trying to write a very large file somewhere under /tmp. Finding it is the challenge. Try looking for large files in /tmp and /tmp/var/log.

EDIT: Also /tmp/home/root

 

[Marko Polo]

There's probably another process that's trying to write a very large file somewhere under /tmp. Finding it is the challenge. try looking for large files in /tmp and /tmp/var/log.

Nothing interesting. var directory itself is only 900KB, all other files in /tmp are negligible. The only exception is /tmp/mnt but I suppose it is a mounting symlink, not a real directory.

 

[ColinTaylor]

You could try running the following command. It's a bit ugly but it does the job. Look for any unusually large files.

find /tmp -exec ls -de {} \; | grep -v " /tmp/mnt/"

Unfortunately I suspect we're looking for a file that's failed to be written, not one that has. Were there any other "failed to write" messages in the log?