[Diversion] failed to write /var/lib/misc/dnsmasq.leases: No space left on device

[BSOD2600]

Thanks all for the suggestions. Had a few hour power outage tonight so everything has been sufficiently reset

A 256MB swap has been created. logging still disabled. we'll see if this fixes the recent issue.

if/when this error occurs again, what are the correct command(s) so help properly identify the root problem?

 

[ColinTaylor]

if/when this error occurs again, what are the correct command(s) so help properly identify the root problem?

I'm not sure there is a root problem as such. My guess is that the combination of all of the things you're running is exceeding the memory capacity of the router. Who's to say that one particular process' memory requirements are any more or less valid that another's.

For example, your dnsmasq process is using 60MB of memory whereas mine is using less than 4MB. That's presumably because of all the blocking you're doing with Diversion. That's neither right or wrong it's just your choice on how you've decided to run your router.

With all that said there is one thing I noticed. Comparing your ps output from the working and non-working setup, they are practically identical except for the memory used by pixelserv-tls. All the other process' memory usage remains constant apart from pixelserv-tls which starts at 9MB and has grown to 29MB by the time the errors occur. I suspect it is this growth that has pushed the memory requirements past what the router has.

Assuming this memory usage is normal for pixelserv-tls and not a memory leak then adding your swap file should fix the problem. It might be worth checking how much memory pixelserv-tls is using from time to time just to make sure it isn't growing indefinitely.

ps w | grep pixelserv-tls

 

[BSOD2600]

24 hours uptime and no repeating errors...yet.

  634 nobody   33968 S    pixelserv-tls 192.168.1.11
  786 nobody   60804 S    dnsmasq --log-async

 

[ColinTaylor]

24 hours uptime and no repeating errors...yet.

You can use "free" to see the swap usage.

 

[ColinTaylor]

The timestamp of Oct 30 06:42:12 also tells me this is not a Diversion related event. The log rotation is at exactly 5:20 in the morning, on all Diversion installations when logging is enabled.

Just a side note that I forgot to mention before. When the USB disk gets to 100% full and the dnsmasq log file cannot be written to (like what happened at 06:42 above) logging does not stop. What happens (at least on my system) is that the logging reverts to the syslog file... which is in /tmp... which is held in memory... which is already borderline 100% used.

 

[sunvit]

I've been doing a bit more experimenting with this (creating OOM situations) and racking my brain from when I had to deal with a similar thing back in my Solaris days.

Even though I can't come to a definitive answer to the OP, there is one red herring that I'd forgotten about; tmpfs.

The router has ~250MB of memory. /tmp will use half that amount (~125MB) by way of tmpfs... sort of:

Filesystem                Size      Used Available Use% Mounted on
devtmpfs                124.8M         0    124.8M   0% /dev
tmpfs                   124.9M      6.5M    118.4M   5% /tmp

However (and this is the bit I forgot ) the "Size" (and by implication "Available") reported by df is not the amount of storage currently available. It is merely a limit on the maximum amount of storage that may be used if there is sufficient free memory.

So as we can see from the output of free the router is already using ~192MB. Even accounting for the 6.5MB of files in /tmp there is no way that the available space is 118.4MB. [I even managed to recreate the OP's dnsmasq errors by constantly sorting a large file in memory and df still reported /tmp as 90% available].

             total       used       free     shared    buffers     cached
Mem:        255700     196704      58996          0       3052      19064
-/+ buffers/cache:     174588      81112

So the unused memory is ~58MB (potentially ~80MB if it freed up all the buffers/cache).

So to sum up, all this proves is that you can't get any meaningful clues out of df. The best you can do is look at free.

Of course this still doesn't identify the original problem. But I suspect it's simply that the memory is over-committed. In my tests free was still reporting 36MB of free memory but processes were still crashing with OOM errors.

I too have the issue. I am using RT-AC87U and Asus Merlin Version- 384.9

I am getting the below error:
Feb 21 19:34:29 dnsmasq-dhcp[783]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)
Feb 21 19:35:29 dnsmasq-dhcp[783]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)
Feb 21 19:36:29 dnsmasq-dhcp[783]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)
Feb 21 19:37:29 dnsmasq-dhcp[783]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)
Feb 21 19:38:29 dnsmasq-dhcp[783]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)
Feb 21 19:39:29 dnsmasq-dhcp[783]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)
Feb 21 19:40:29 dnsmasq-dhcp[783]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)

 

[EmeraldDeer]

# df -k /var
# du -k /var | sort -n | tail
# find /var -size +100 -exec ls -al {} \; | sort -k 5,5 -n | tail
# df -k /opt/var/log
# du -k /opt/var/log | sort -n | tail
# find /opt/var/log -size +100 -exec ls -al {} \; | sort -k 5,5 -n | tail
# cat /proc/meminfo

 

[ColinTaylor]

# df -k /var
# du -k /var | sort -n | tail
# find /var -size +100 -exec ls -al {} \; | sort -k 5,5 -n | tail

Minor correction. /var is a link to /tmp/var so the command needs to use either "/tmp/var" or "/var/". i.e.

# du -k /tmp/var | sort -n | tail

or

# du -k /var/ | sort -n | tail