DNS problems

[fbreve]

I'm using firmware 3.0.0.4.260.

Once in a while, computers in the network complains about DNS problems and most of the Internet becomes unaccessible, even though some connections do not drop.

When this happens, I check the log file and it's full of these messages:
Nov 28 16:13:57 kernel: VFS: file-max limit 22684 reached
Nov 28 16:13:59 kernel: VFS: file-max limit 22684 reached
Nov 28 16:14:02 kernel: VFS: file-max limit 22684 reached
Nov 28 16:14:05 kernel: VFS: file-max limit 22684 reached
Nov 28 16:14:08 kernel: VFS: file-max limit 22684 reached
Nov 28 16:14:11 kernel: VFS: file-max limit 22684 reached
Nov 28 16:14:14 kernel: VFS: file-max limit 22684 reached
Nov 28 16:14:17 kernel: VFS: file-max limit 22684 reached
Nov 28 16:14:20 kernel: VFS: file-max limit 22684 reached
Nov 28 16:14:24 kernel: VFS: file-max limit 22684 reached
Nov 28 16:14:29 kernel: VFS: file-max limit 22684 reached
Nov 28 16:14:32 kernel: VFS: file-max limit 22684 reached
Nov 28 16:14:38 kernel: VFS: file-max limit 22684 reached
Nov 28 16:14:41 kernel: VFS: file-max limit 22684 reached
Nov 28 16:14:44 kernel: VFS: file-max limit 22684 reached

Does anyone have this same problem or how to fix it?

 

[RogerSC]

I'm using firmware 3.0.0.4.260.

Once in a while, computers in the network complains about DNS problems and most of the Internet becomes unaccessible, even though some connections do not drop.

When this happens, I check the log file and it's full of these messages:
Nov 28 16:13:57 kernel: VFS: file-max limit 22684 reached
Nov 28 16:13:59 kernel: VFS: file-max limit 22684 reached
Nov 28 16:14:02 kernel: VFS: file-max limit 22684 reached
Nov 28 16:14:05 kernel: VFS: file-max limit 22684 reached
Nov 28 16:14:08 kernel: VFS: file-max limit 22684 reached
Nov 28 16:14:11 kernel: VFS: file-max limit 22684 reached
Nov 28 16:14:14 kernel: VFS: file-max limit 22684 reached
Nov 28 16:14:17 kernel: VFS: file-max limit 22684 reached
Nov 28 16:14:20 kernel: VFS: file-max limit 22684 reached
Nov 28 16:14:24 kernel: VFS: file-max limit 22684 reached
Nov 28 16:14:29 kernel: VFS: file-max limit 22684 reached
Nov 28 16:14:32 kernel: VFS: file-max limit 22684 reached
Nov 28 16:14:38 kernel: VFS: file-max limit 22684 reached
Nov 28 16:14:41 kernel: VFS: file-max limit 22684 reached
Nov 28 16:14:44 kernel: VFS: file-max limit 22684 reached

Does anyone have this same problem or how to fix it?

That's a lot of files, so I have to wonder if this is corruption, perhaps from not clearing the nvram after updating your firmware to 260? I can't imagine what could really be creating all these files...Can you please try clearing the nvram in your router, and then manually re-entering your settings and see if that helps the situation?

 

[fbreve]

It was happening with the previous firmware too. And with that one, I fully reset the router after installing it to fix a problem where any wireless connection would kill the router (that was a NVRAM problem after upgrading firmware). So this time I don't think it's NVRAM related.

 

[RogerSC]

It was happening with the previous firmware too. And with that one, I fully reset the router after installing it to fix a problem where any wireless connection would kill the router (that was a NVRAM problem after upgrading firmware). So this time I don't think it's NVRAM related.

I'd be curious about this one, and log into the router. Find out where the large mass of those files are in the file system, and see what they're named, who the owner is, etc. That might tell you something. If some process is going wild creating lots of files, that might help to narrow it down. It might also help for you to mention what settings you have that aren't vanilla, like VPN, DLNA, etc. stuff that you're doing that's above and beyond just routing and wireless connections.

 

[RMerlin]

Download Manager allowing way too many inbound connections (both to BT and aMule) would be my first theory.

 

[fbreve]

I completely forgot I had telnet access. Thanks for remembering this. I spent too much time using routers that had only web access.

I just used ls -Rl | wc -l and it counted 18,126 files. So far, no problem.

I'll check again when the problem arises.

I'm not using anything unusual, QoS is enabled with default settings, I'm not using Download Master, Wireless WPA2-Personal, some manually assigned IP address, a single port trigger, a bunch of port forwarding, no IPv6.

I just noticed VPN Server is enabled, but I'm not using it (Samba and PPTP), I'll try disabling it after it happens one more time so I can check the filesystem.

 

[fbreve]

admin@RT-AC66U:/# du -ks ./* | sort -n
0 ./cifs1
0 ./cifs2
0 ./dev
0 ./etc
0 ./home
0 ./jffs
0 ./mmc
0 ./mnt
0 ./opt
0 ./proc
0 ./root
0 ./sys
0 ./var
802 ./sbin
1671 ./bin
6011 ./www
8230 ./lib
9218 ./rom
31506 ./usr
268920 ./tmp

 

[RogerSC]

admin@RT-AC66U:/# du -ks ./* | sort -n
0 ./cifs1
0 ./cifs2
0 ./dev
0 ./etc
0 ./home
0 ./jffs
0 ./mmc
0 ./mnt
0 ./opt
0 ./proc
0 ./root
0 ./sys
0 ./var
802 ./sbin
1671 ./bin
6011 ./www
8230 ./lib
9218 ./rom
31506 ./usr
268920 ./tmp

That's a lot of storage in /tmp, wondering how many files are there? I'm doing very vanilla stuff with my router and have this in /tmp (only 186 files with 2.58MB of storage):

admin@RT-N66U:/tmp# ls -lR | wc
186 1234 9508
admin@RT-N66U:/tmp# du -sk /tmp
2580 /tmp

I don't know what would be causing you to collect so many files in /tmp, and so much storage there. Can you find out what subdirectories the bulk of the files are in?

Could you have something enabled that you're not aware of that's running amok? Just a thought.

 

[RMerlin]

I completely forgot I had telnet access. Thanks for remembering this. I spent too much time using routers that had only web access.

I just used ls -Rl | wc -l and it counted 18,126 files. So far, no problem.

That error message referred to simultaneously open files (file handles, to be technical), not to the total amount of files you have on your system.

/tmp is located in RAM. Anything that requires writing by the router (such as syslog, dynamically generated config files, etc...) will be stored in /tmp.

 

[RogerSC]

That error message referred to simultaneously open files (file handles, to be technical), not to the total amount of files you have on your system.

/tmp is located in RAM. Anything that requires writing by the router (such as syslog, dynamically generated config files, etc...) will be stored in /tmp.

I wasn't aware that the number of open files was that large, nor that /tmp was a RAM file system, although it makes sense that it would be, considering all that seems to be in it. However, I'm still wondering where the bulk of that storage is in /tmp, that might tell one something about what's malfunctioning, since the subdirs in /tmp all have somewhat functional names.

Anyways, just a thought, thinking about ways to get to the bottom of this.

 

[fbreve]

Most of the files on /tmp were actually mounted from an USB stick I used during some tests yesterday. It was the first time I used an USB stick on RT-AC66U, so it's not related to the problem.

I unmounted and removed it now and these are the new amount of files in each directory:

admin@RT-AC66U:/# du -ks ./* | sort -n
0 ./cifs1
0 ./cifs2
0 ./dev
0 ./etc
0 ./home
0 ./jffs
0 ./mmc
0 ./mnt
0 ./opt
0 ./proc
0 ./root
0 ./sys
0 ./var
802 ./sbin
868 ./tmp
1671 ./bin
6011 ./www
8230 ./lib
9218 ./rom
31506 ./usr

I guess there is nothing wrong with those numbers. I'll have to wait until the problems happens again and check the filesystem while it's ocurring. The proccess that may be getting crazy is probably filling the file system really quickly because the time between the boot and the problem is random. Sometimes it's a few hours, sometimes it's a few days.

 

[fbreve]

And it happened again:

Mar 14 09:13:37 kernel: VFS: file-max limit 22684 reached
Mar 14 09:13:45 kernel: VFS: file-max limit 22684 reached
Mar 14 09:13:46 kernel: VFS: file-max limit 22684 reached
Mar 14 09:13:55 kernel: VFS: file-max limit 22684 reached
Mar 14 09:14:04 kernel: VFS: file-max limit 22684 reached
Mar 14 09:14:04 kernel: VFS: file-max limit 22684 reached
Mar 14 09:14:05 kernel: VFS: file-max limit 22684 reached
Mar 14 09:14:05 kernel: VFS: file-max limit 22684 reached
Mar 14 09:14:07 kernel: VFS: file-max limit 22684 reached
Mar 14 09:14:10 kernel: VFS: file-max limit 22684 reached
Mar 14 09:14:16 kernel: VFS: file-max limit 22684 reached
Mar 14 09:14:22 kernel: VFS: file-max limit 22684 reached
Mar 14 09:14:31 kernel: VFS: file-max limit 22684 reached
Mar 14 09:14:34 kernel: VFS: file-max limit 22684 reached
Mar 14 09:14:37 kernel: VFS: file-max limit 22684 reached
Mar 14 09:14:45 kernel: VFS: file-max limit 22684 reached
Mar 14 09:14:46 kernel: VFS: file-max limit 22684 reached
Mar 14 09:14:49 kernel: VFS: file-max limit 22684 reached
Mar 14 09:14:52 kernel: VFS: file-max limit 22684 reached
Mar 14 09:14:58 kernel: VFS: file-max limit 22684 reached
Mar 14 09:15:05 kernel: VFS: file-max limit 22684 reached
Mar 14 09:15:05 kernel: VFS: file-max limit 22684 reached
Mar 14 09:15:13 kernel: VFS: file-max limit 22684 reached
Mar 14 09:15:16 kernel: VFS: file-max limit 22684 reached
Mar 14 09:15:19 kernel: VFS: file-max limit 22684 reached
Mar 14 09:15:22 kernel: VFS: file-max limit 22684 reached
Mar 14 09:15:25 kernel: VFS: file-max limit 22684 reached
Mar 14 09:15:31 kernel: VFS: file-max limit 22684 reached
Mar 14 09:15:35 kernel: VFS: file-max limit 22684 reached
Mar 14 09:15:36 kernel: VFS: file-max limit 22684 reached

I checked the file system before resetting it and the numbers were almost the same as those above.

 

[fbreve]

It happened again:

Mar 16 00:44:11 kernel: VFS: file-max limit 22684 reached
Mar 16 00:44:13 kernel: VFS: file-max limit 22684 reached
Mar 16 00:44:20 kernel: VFS: file-max limit 22684 reached
Mar 16 00:44:23 kernel: VFS: file-max limit 22684 reached
Mar 16 00:44:31 kernel: VFS: file-max limit 22684 reached
Mar 16 00:44:38 kernel: VFS: file-max limit 22684 reached
Mar 16 00:44:41 kernel: VFS: file-max limit 22684 reached
Mar 16 00:44:41 kernel: VFS: file-max limit 22684 reached
Mar 16 00:44:44 kernel: VFS: file-max limit 22684 reached

RT-AC66U login: admin
Password:

And this is the files distribution:

admin@RT-AC66U:/# du -ks ./* | sort -n
0 ./cifs1
0 ./cifs2
0 ./dev
0 ./etc
0 ./home
0 ./jffs
0 ./mmc
0 ./mnt
0 ./opt
0 ./proc
0 ./root
0 ./sys
0 ./var
472 ./tmp
809 ./sbin
1670 ./bin
6034 ./www
8230 ./lib
9218 ./rom
31509 ./usr
admin@RT-AC66U:/#


admin@RT-AC66U:/usr# du -ks ./* | sort -n
0 ./local
0 ./share
0 ./tmp
7 ./bin
611 ./js
944 ./css
9417 ./sbin
20531 ./lib

I guess there is nothing wrong in the file system.

 

[RMerlin]

Looking at the filesystem with du won't give you any useful information. You need "lsof", which will tell you what file handles are currently open.

lsof isn't included with the firmware, but should be available on Optware or Entware.

 

[episyron]

I am also getting this with firmware 3.0.0.4.270 on RT-N66U. Last time it took eleven days uptime to reach the limit. Has anyone debugged this with lsof?

 

[jmaino]

I too have had this happen a few times.

I just found that I can clear the situation by turning off per-IP monitoring, and avoid a reboot of my AC66U. So, I'm going to venture a guess that it's the per-IP monitoring that's keeping these filehandles around, and they get cleaned-up when you turn it off.

Will let you all know if I have another occurrence now that per-IP monitoring is turned off on my AC66U.

-jm

 

[archon810]

Hitting the same problem today for the first time, running Firmware Version:3.0.0.4.372_1300 on the RT-AC66U.

Tons of kernel: VFS: file-max limit 22684 reached lines in the log.

 

[vdemarco]

Hitting the same problem today for the first time, running Firmware Version:3.0.0.4.372_1300 on the RT-AC66U.

Tons of kernel: VFS: file-max limit 22684 reached lines in the log.

If you have entware/optware set up on your router run this

opkg install lsof

then post the output of

running the command

lsof

 

[archon810]

If you have entware/optware set up on your router run this

opkg install lsof

then post the output of

running the command

lsof

Sorry, I don't, and I'm not familiar with it. I just run stock and, frankly, it does everything I need it to, so I don't want to install custom firmware or tinker with the low-level things. I just want it to work reliably as it should. I know it doesn't help the debugging steps though.

I'm hoping someone from ASUS will notice and escalate this issue https://plus.google.com/107797272029781254158/posts/TphyASLBF8q

 

[vdemarco]

Sorry, I don't, and I'm not familiar with it. I just run stock and, frankly, it does everything I need it to, so I don't want to install custom firmware or tinker with the low-level things. I just want it to work reliably as it should. I know it doesn't help the debugging steps though.

I'm hoping someone from ASUS will notice and escalate this issue https://plus.google.com/107797272029781254158/posts/TphyASLBF8q

Send a PM to CL-Jermemy here. He is a Asus rep.