Need help tracking down recurring AC68U space issue

[fr33z0n3r]

I am getting an ongoing issue with space on my router. I'd like to know how I can track down the cause of this. I'm not using this for anything but standard router/wifi. I've already reset my config, to see if that helps.

If anyone has ideas on commands to run regularly, as well as anything I could do with alerting to be able to get in before space is too low to access it. Or logging to capture the proof of it would be appreciated.

How I see the problem appear in syslog (remote):

1. It seems like there are always topology change events occurring, then a few days before the device stop working well they stop coming in.

2. A few days after that this event appears:

dnsmasq-dhcp 250 - - dnsmasq-dhcp[250]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60s)

 

[ColinTaylor]

You appear to have run out of memory.

Are you running any third-party scripts, e.g. ad-blocking? What firmware version are you using?

When the problem occurs SSH into the router and run ps w and free to try and see what is consuming the memory.

 

[fr33z0n3r]

the issue is its unreachable when I find this. I am running the latest firmware. Again, not doing anything except wifi and firewall.

 

[fr33z0n3r]

I could open an ssh session and run top perpetually. But there doesn't seem to be any memory usage stats in there. Is there any way to tie memory usage to a process o nthis?

 

[bbunge]

the issue is its unreachable when I find this. I am running the latest firmware. Again, not doing anything except wifi and firewall.

Latest Firmware tells us nothing. There are several betas that folks consider latest. Merlin 384.19 is also latest. Just which firmware do you have and what add-ons?

 

[john9527]

It's not really process memory usage. You are running out of space on the /tmp partition in memory.
First thing is to try and find what file is using up the space. Running the following command will show the top 10 largest files in that partition.

du -ahx /tmp | sort -n -r | head -n 10

 

[fr33z0n3r]

Latest always means non beta latest - 384.19.

I will try to run that command. How can I run that command set with 'watch'?

 

[john9527]

How can I run that command set with 'watch'?

Just put the whole command in quotes...for example

watch -n 30 "du -ahx /tmp | sort -n -r | head -n 10"

 

[fr33z0n3r]

Thanks John! Not sure if the SSH session will die, but I'm going to use its session log to track the data.

 

[john9527]

I suspect you'll be able to spot a file that is growing before it actually gets to the point of causing a problem.

 

[fr33z0n3r]

So far only the syslog.log file is growing.

 

[fr33z0n3r]

So, I this watch command running for days. The top entry was always /tmp, and it got up to 1004K. I stopped the command to check some other stuff, but when I reran the same watch command the /tmp entry is not visible on the list. It should be the top entry still.

Any ideas why?

Still no issues with the router.

 

[ColinTaylor]

I think the command you were using was wrong, or perhaps your version of du formats its output differently than mine.

Try this instead (shows sizes in kB):

du -akx /tmp | sort -n -r | head -n 10

It's probably worth including the top command as well if your router supports the necessary options:

watch -n 30 "du -akx /tmp | sort -n -r | head -n 10; echo; top -b -m -n 1 | head -n 15"

 

[fr33z0n3r]

Both show /tmp for me, thanks. for the longer command, there is an error saying "top: no process info in proc"

 

[ColinTaylor]

there is an error saying "top: no process info in proc"

OK, it looks like your version of top doesn't support the memory option. John's firmware had the same problem until he patched it last year.

Oh well, you could use this:

watch -n 30 "du -akx /tmp | sort -n -r | head -n 10; echo; /bin/ps w | sort -n -r -k 3 | head -n 15"

 

[KsWoodsMan]

So far only the syslog.log file is growing.

What about the directories under /tmp ?
Are any of the directory sizes also growing ?
My (/tmp)/var/log direcory would grow to over 18MB. It did, untill I moved /var/log/dnsmasq.log to mounted storage at /opt/var/log/dnsmasq.log.

/tmp/syslog.log should be clamped in size to keep it and the backup from getting to big.
On mine the size is clamped to 256KB. Once it exceeds this size it gets rotated.
Unless it changed, you can check it with nvram get log_size .

Relocating /var/log/dnsmasq.log to /opt solved my problem like you are having.

 

[fr33z0n3r]

Yeah, nothing is growing very large. I'm just gonna keep this running for weeks and see if the issue reoccurs. Thanks for all the input everyone!