[Diversion] failed to write /var/lib/misc/dnsmasq.leases: No space left on device

[BSOD2600]

Woke up to find no LAN devices could access the Internet. Eventually figured out the router wasn't handing out DHCP any longer. Found the following in the router logs

Oct 30 06:42:12 dnsmasq[24902]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)
Oct 30 06:42:12 dnsmasq[24902]: DHCPREQUEST(br0) 192.168.1.65 9c:eb:e8:24:f8:8a
Oct 30 06:42:12 dnsmasq[24902]: failed to allocate 124 bytes

What was using all of the space? dnsmasq log files.

Filesystem                Size      Used Available Use% Mounted on
/dev/root                36.1M     36.1M         0 100% /
devtmpfs                124.8M         0    124.8M   0% /dev
tmpfs                   124.9M      6.3M    118.5M   5% /tmp
/dev/mtdblock4           62.8M      8.7M     54.0M  14% /jffs
/dev/sda1               462.0M    437.4M      1.0K 100% /tmp/mnt/AB

admin@RT-AC87U:/tmp/mnt/AB/entware/var/log# ll -h
-rw-r-----    1 nobody   root      154.0K Oct 30 06:45 dnsmasq.log
-rw-r-----    1 nobody   root      383.8M Oct 30 05:21 dnsmasq.log1
-rw-r-----    1 nobody   root           0 Oct 30 02:01 dnsmasq.log2

Disabled logging in diversion, deleted the dnsmasq log files and all back to normal now.

I never had this problem when using AB-Solution. Why isnt the log rolling properly working?

 

[bitmonster]

Do you have Swap enabled and a thumb drive plugged in to the USB2 port? I'm not sure if this is related but you don't seem to.

Uninstall all the custom scripts, maybe even factory reset the whole thing (after saving settings file) and set the customisations back up *after* you have a flash drive plugged in.

 

[BSOD2600]

How would I check the swap setting?
USB is connected to the port in the rear (next to the enet ports).

 

[bitmonster]

As per the instructions for Diversion, do you have a flash drive plugged in to the router and did you enable swap as prompted by the installation, assuming you have a flash drive plugged in to the router for storage?

If not, do that, and start everything over again.

Use the USB2 port to keep the USB3 port free for other drives.

 

[BSOD2600]

A 512MB flash drive is plugged into the router. Pretty sure I said no to swap due to the storage constraints.
Due to physical space constraints, the USB 2 drive is plugged into the rear of the router.

 

[bitmonster]

It should be pretty obvious what your issue is.

Swap is needed for a reason.

I suggest you uninstall all customisation or just factory reset until you are able to plug in a 2GB+ flash drive and next time you install Diversion, don't skip over the prompt to install Swap, or at least install it via Diversion options (wherever it is).

The router internal storage is for the router only, not for anything else you add on top, when you can just use an external flash drive anyway.

Factory reset / wipe the router until you are able to plug in a proper flash drive with Swap.

I have 1GB for Swap on a 16GB flash drive, but 512MB Swap + 2GB storage should be more than enough.

 

[ColinTaylor]

Rather than going to the extreme of factory resetting the router he might want to first try clearing down some space on his USB drive and see if that fixes the problem in the short term.

# rm /tmp/mnt/AB/entware/var/log/dnsmasq.log*
# reboot

 

[bitmonster]

That won't fix the no swap issue..

Swap is still required, even if just 100MB, and a larger flash drive is strongly advised.

Sent from my SM-G965F using Tapatalk

 

[ColinTaylor]

That won't fix the no swap issue..

No but it might get his router working again while he sources a bigger flash drive.

EDIT: Oops, just looked back at post #1 and seen that's exactly what he did already.

 

[strangeluck]

The issue here isn't a lack of swap, it is a lack of storage space on the USB device. Adding a swap file will only consume more space on said USB device. The solution is more likely to involve a USB device with more storage or freeing space on the existing device.

 

[BSOD2600]

Odd, the Diversion requirements ( https://diversion.ch/diversion/requirements.html ) states min of 500MB.

It shouldn't be that hard to tweak the log size / roll over count for dnsmasq, right? Not that familiar with Linux to know where to look and change the setting...

 

[bitmonster]

Just out of curiosity, is there a pressing reason you just can't use a bigger drive?


Sent from my SM-G965F using Tapatalk

 

[john9527]

Something doesn't make sense here...../var/lib/misc/dnsmasq.leases resides in ram in the tmpfs partition, which showed only 5% utilization.

Does Diversion use ram tmp space then move things to the flash drive? Or could a background update of the blocking files cause a thread to timeout which fools the code into thinking it can't allocate space?

 

[strangeluck]

Odd, the Diversion requirements ( https://diversion.ch/diversion/requirements.html ) states min of 500MB.

It shouldn't be that hard to tweak the log size / roll over count for dnsmasq, right? Not that familiar with Linux to know where to look and change the setting...

Something doesn't make sense here...../var/lib/misc/dnsmasq.leases resides in ram in the tmpfs partition, which showed only 5% utilization.

Does Diversion use ram tmp space then move things to the flash drive? Or could a background update of the blocking files cause a thread to timeout which fools the code into thinking it can't allocate space?

With logging enabled in diversion the logs are saved to /opt/var/log/dnsmasq.log and used to generate statistics. Just as a point of reference, my log files for the last 4 days (new logs are started weekly for stats on friday morning for me) are currently totaling 320mb. 500mb is not going to be enough space for logging if you have a lot of network traffic.

Also note that logging can be disabled in diversion which will mean no additional space consumed on the USB for dnsmasq logs. If for some reason you are encountering a large number of errors (invalid DNSSEC signature keys comes to mind) your log will grow exponentially larger as well.

 

[john9527]

With logging enabled in diversion the logs are saved to /opt/var/log/dnsmasq.log

Exactly my point....../opt links to a directory on the USB and I can see where that may be getting filled.

But the failure he had...

Oct 30 06:42:12 dnsmasq[24902]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)
Oct 30 06:42:12 dnsmasq[24902]: DHCPREQUEST(br0) 192.168.1.65 9c:eb:e8:24:f8:8a
Oct 30 06:42:12 dnsmasq[24902]: failed to allocate 124 bytes

points to tmpfs in ram, which when he checked, was only 5% utilized...

 

[ColinTaylor]

Something doesn't make sense here...../var/lib/misc/dnsmasq.leases resides in ram in the tmpfs partition, which showed only 5% utilization.

We've had this message crop up a few times over the years and more often than not when investigated there always appears to be plenty of free memory. For example here.

Idle thought: I wonder whether (perhaps because the USB is 100% full and can't be written to) that when it tries to "roll over" the dnsmasq log file it is temporarily trying to hold all 383.8MB of it in memory but can't because there's only 118.5MB available. Just a thought.

 

[thelonelycoder]

Disabled logging in diversion, deleted the dnsmasq log files and all back to normal now.

I never had this problem when using AB-Solution. Why isnt the log rolling properly working?

The Diversion log rotation is working just fine:

-rw-r-----    1 nobody   root      154.0K Oct 30 06:45 dnsmasq.log
-rw-r-----    1 nobody   root      383.8M Oct 30 05:21 dnsmasq.log1
-rw-r-----    1 nobody   root           0 Oct 30 02:01 dnsmasq.log2

dnsmasq.log was rolled over to dnsmasq.log1 at the correct time by a cron job at 5:20. It does this every day, copying dnsmasq.log1 to dnsmasq.log2 and renaming dnsmasq.log to dnsmasq.log1.

Your USB device is full, look for a bigger device or disable logging in Diversion.
The dnsmasq.log's are kept for one week for the stats function to have some data to work with and are then reset.

 

[john9527]

dnsmasq.log was rolled over to dnsmasq.log1 at the correct time by a cron job at 5:20. It does this every day, copying dnsmasq.log1 to dnsmasq.log2 and renaming dnsmasq.log to dnsmasq.log1.

Do you really do a copy? I'm wondering if @ColinTaylor is on to something about buffers being allocated to do the copy. Could you do the rotation entirely with delete and rename instead?

 

[thelonelycoder]

Idle thought: I wonder whether (perhaps because the USB is 100% full and can't be written to) that when it tries to "roll over" the dnsmasq log file it is temporarily trying to hold all 383.8MB of it in memory but can't because there's only 118.5MB available. Just a thought.

During the (Diversion's own) dnsmasq log rotation log1 is appended to log2 which uses some memory but no usage of the /tmp/ filesystem is involved. Some small files are written to /tmp/ if the stats function in Diversion is enabled to preserve the /var/lib/misc/dnsmasq.leases and /et../hosts.dnsmasq client-names-t0-IP resolution. But that's all.
I suspect OP has a very busy router and therefore large dnsmasq log files.

 

[thelonelycoder]

Do you really do a copy? I'm wondering if @ColinTaylor is on to something about buffers being allocated to do the copy. Could you do the rotation entirely with delete and rename instead?

The procedure is the same ever since the early AB-Solution days.
Append content of log1 to log2, then move log to log1 and create a new empty log file.