Since upgrading to firmware 386.3_2, my Internet will not stay connected for even a day.

[DTS]

I assume you increased it believing that would result in fewer writes. But hanging on to leases well beyond when guests leave can lead to exhaustion of the DHCP pool, at least w/ heavy usage, and depending on the size of the pool. If anything, I would have probably decreased the lease duration so they could be removed more often and make space available (the dnmasq.leases error was about the lack of space).

I guess it just depends on the root cause of the problem w/ those leases. Increasing the lease duration makes sense if you believe the mere fact of writing to the file causes the problem. Whereas decreasing the lease duration makes sense (at least to me) if you take the error message at its word, and the storage is exhausted.

Thanks for sharing that explanation. I will let you know as soon as the problem happens again.

Maybe it will go another week or two?

If it doesn't happen naturally by then, I could test by logging several devices onto the guest network.

If you have other suggestions, such as anything that would cause a lot of writes to /var/lib/misc/dnsmasq.leases, I could try your suggestions when the time comes.

For the moment, I'm just happy it is running problem-free. But I do eventually want to discover the cause of this error:

dnsmasq-dhcp[1784]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60 s)

 

[DTS]

OK, the issue happened (same as before and after about 16 days this time). Clients cannot access the Internet and the log shows a lot of these errors:

Jan  8 19:47:35 dnsmasq-dhcp[18996]: failed to write /var/lib/misc/dnsmasq.leases: No space left on device (retry in 60 s)

I'm ready to try more troubleshooting...

 

[eibgrad]

As I mentioned previously, see if the NAT rule for tun11 is still there.

iptables -t nat -vnL POSTROUTING

If not, add it manually and see if things start working again (w/o a need for a restart).

iptables -t nat -I POSTROUTING -o tun11 -j MASQUERADE

 

[eibgrad]

Also, see what's in that dnsmasq.leases file (or if it's even readable).

cat /var/lib/misc/dnsmasq.leases

 

[DTS]

Unfortunately, the issue is slightly different this time. I noticed the usual issue that wireless clients could not access the Internet last night. I checked the router logs and saw the errors (`failed to write /var/lib/misc/dnsmasq.leases: No space left on device`). But it was late and I could not begin troubleshooting until today.

However, early this morning, before I started any troubleshooting, we had a very brief power outage. It lasted only a few seconds. All my devices are on UPS's (and the whole house has a generator).

As mentioned previously, I have more than one physically separate LAN and more than one router connected to my ISP (with different WAN IP addresses). In the past, this Asus issue was independent -- there were no issues with my other routers or WAN connectivity.

However, today my main (wired) router's VPN tunnel went down when the power glitched (even with a UPS).

And, this time, the Asus tun11 is not up either.

So it is a different issue -- at least it is different now, after the power glitch. Maybe it would have been the same if I had been able to start troubleshooting last night.

 

[DTS]

The file is zero bytes (empty)

# ls -la /var/lib/misc/dnsmasq.leases
-rw-r--r--    1 admin root             0 Jan  8 20:11 /var/lib/misc/dnsmasq.leases

This is the same as it was the last time the issue happened.

 

[ColinTaylor]

The file is zero bytes (empty)

# ls -la /var/lib/misc/dnsmasq.leases
-rw-r--r--    1 admin root             0 Jan  8 20:11 /var/lib/misc/dnsmasq.leases

This is the same as it was the last time the issue happened.

cat the contents of that file anyway as it may still contain information even though it says zero bytes.

 

[eibgrad]

I want to see if it's corrupted or perhaps unreadable.

 

[DTS]

cat the contents of that file anyway as it may still contain information even though it says zero bytes.

I did and it is empty

 

[DTS]

I want to see if it's corrupted or perhaps unreadable.

cat returns nothing

 

[DTS]

What else is of interest given that tun11 is down?

 

[eibgrad]

/var/lib/misc/dnsmasq.leases points to /tmp/var/lib/misc/dnsmasq.leases. Could /tmp space be exhausted?

Show the output from the df command.

 

[DTS]

/var/lib/misc/dnsmasq.leases points to /tmp/var/lib/misc/dnsmasq.leases. Could /tmp space be exhausted?

Show the output from the df command.

# df -h
Filesystem                Size      Used Available Use% Mounted on
ubi:rootfs_ubifs         77.2M     64.6M     12.6M  84% /
devtmpfs                214.9M         0    214.9M   0% /dev
tmpfs                   215.0M    256.0K    214.8M   0% /var
tmpfs                   215.0M      1.1M    213.9M   1% /tmp/mnt
mtd:bootfs                4.4M      3.3M      1.1M  75% /bootfs
tmpfs                   215.0M      1.1M    213.9M   1% /tmp/mnt
mtd:data                  8.0M    588.0K      7.4M   7% /data
tmpfs                   215.0M      1.1M    213.9M   1% /tmp
/dev/mtdblock9           47.0M      3.5M     43.5M   7% /jffs
tmpfs                   215.0M      1.1M    213.9M   1% /www/require/modules/menuTree.js

 

[DTS]

current free:

# free
             total       used       free     shared    buffers     cached
Mem:        440324     414648      25676       1592          0       5244
-/+ buffers/cache:     409404      30920
Swap:            0          0          0

compare to the values immediately after the last reboot 16 days ago:

# free
             total       used       free     shared    buffers     cached
Mem:        440324     201824     238500       1472          0      18328
-/+ buffers/cache:     183496     256828
Swap:            0          0          0

 

[ColinTaylor]

Your memory usage seems unusually high. Run the following command and see if there's anything odd:

ps w | sort -n -k 3

 

[eibgrad]

Just weird. I can't understand how that file could fail to write. And why it's even empty. There's plenty of storage. It's clearly writing leases to that file given we see the attempt. Can you write anything else to that same storage? Try the following script.

cat << 'EOF' > /var/lib/misc/temp.txt
123
456
789
EOF

cat /var/lib/misc/temp.txt

 

[DTS]

Your memory usage seems unusually high. Run the following command and see if there's anything odd:

ps w | sort -n -k 3

# ps w | sort -n -k 3
    2 admin     0 SW   [kthreadd]
    3 admin     0 SW   [ksoftirqd/0]
    5 admin     0 SW<  [kworker/0:0H]
    7 admin     0 RW   [rcu_preempt]
    8 admin     0 SW   [rcu_sched]
    9 admin     0 SW   [rcu_bh]
   10 admin     0 SW   [migration/0]
   11 admin     0 SW   [watchdog/0]
   12 admin     0 SW   [watchdog/1]
   13 admin     0 SW   [migration/1]
   14 admin     0 RW   [ksoftirqd/1]
   16 admin     0 SW<  [kworker/1:0H]
   17 admin     0 SW<  [khelper]
   18 admin     0 SW   [kdevtmpfs]
   19 admin     0 SW<  [writeback]
   20 admin     0 SWN  [ksmd]
   21 admin     0 SW<  [crypto]
   22 admin     0 SW<  [bioset]
   23 admin     0 SW<  [kblockd]
   24 admin     0 DW   [skbFreeTask]
   25 admin     0 SW   [bcmFapDrv]
   26 admin     0 SW   [kworker/0:1]
   27 admin     0 RWN  [kswapd0]
   28 admin     0 SW   [fsnotify_mark]
   50 admin     0 SW   [kworker/u4:1]
   53 admin     0 SW   [btnhandler0]
   54 admin     0 SW   [btnhandler1]
   55 admin     0 SW   [btnhandler2]
   57 admin     0 SW<  [linkwatch]
   58 admin     0 SW<  [ipv6_addrconf]
   59 admin     0 SW<  [deferwq]
   60 admin     0 SW   [ubi_bgt0d]
  170 admin     0 SWN  [jffs2_gcd_mtd2]
  204 admin     0 SW   [bcmFlwStatsTask]
  208 admin     0 SW   [bcmsw_rx]
  209 admin     0 SW   [bcmsw]
  217 admin     0 SW   [pdc_rx]
  361 admin     0 SWN  [jffs2_gcd_mtd9]
  533 admin     0 SW   [dhd_watchdog_th]
  534 admin     0 SW   [wfd0-thrd]
  541 admin     0 RW   [dhd_watchdog_th]
  542 admin     0 SW   [wfd1-thrd]
  837 admin     0 SW   [kworker/u4:2]
  974 admin     0 SW   [kworker/1:2]
  PID USER       VSZ STAT COMMAND
16085 admin     0 SW   [kworker/1:1]
20747 admin     0 SW   [kworker/0:2]
21829 admin     0 Z    [firewall-start]
1975 admin  130m S    httpds -s -i br0 -p 8443
  292 admin  1568 S    {wdtctl} wdtd
  296 admin  1712 S    hotplug2 --persistent --no-coldplug
  299 admin  1752 S    /usr/sbin/envrams
1054 admin  1988 S    /usr/sbin/jitterentropy-rngd -p /var/run/jitterentropy-rngd.pid
 1873 admin  2352 S    dropbear -p 192.168.10.1:2222 -s -j -k
 3646 admin  2480 S    dropbear -p 192.168.10.1:2222 -s -j -k
1772 admin  2816 S    /usr/sbin/dhd_monitor
1799 admin  2832 S    /usr/sbin/infosvr br0
1192 admin  2884 S    /bin/eapd
1770 admin  3080 S    /usr/sbin/acsd
1794 nobody    3092 S    avahi-daemon: running [AC86U-Merlin.local]
1823 admin  3168 S    lld2d br0
1029 admin  3316 S    lldpd -L /usr/sbin/lldpcli -I eth1,eth2,eth3,eth4,eth5,eth6,wl0.2,wds0.*.*,wds1.*.*,wds2.*.* -s RT-AC86U
1033 nobody    3316 S    lldpd -L /usr/sbin/lldpcli -I eth1,eth2,eth3,eth4,eth5,eth6,wl0.2,wds0.*.*,wds1.*.*,wds2.*.* -s RT-AC86U
  976 admin  3392 S    /sbin/syslogd -m 0 -S -O /tmp/syslog.log -s 256 -l 7
  978 admin  3392 S    /sbin/klogd -c 5
1059 admin  3392 S    sh -c ATE Get_WanLanStatus
1789 admin  3392 S    crond -l 9
3648 admin  3396 S    -sh
24421 admin  3396 R    ps w
24422 admin  3396 S    sort -n -k 3
1196 admin  3704 S    nas
7799 admin  3748 S    /usr/sbin/ntp -t -S /sbin/ntpd_synced -p north-america.pool.ntp.org -p pool.ntp.org
1806 admin  4236 R    rstats
1808 admin  4444 S    sysstate
1199 admin  4736 S    /usr/sbin/wlc_nt
1798 admin  4764 S    vis-datacollector
18996 nobody    4916 S    dnsmasq --log-async
18997 admin  4916 S    dnsmasq --log-async
1221 admin  4984 S    /usr/sbin/wlceventd
1329 admin  5012 S    nt_actMail
1795 admin  5372 S    vis-dcon
18993 admin  6008 S    /usr/sbin/stubby -g -C /etc/stubby/stubby.yml
1061 admin  6784 S    protect_srv
1055 admin  8204 S    asd
2034 admin  8604 S    cfg_server
15948 admin  9088 S    networkmap
1976 admin  9940 S    httpd -i br0
1828 admin 10492 S    vis-dcon
1838 admin 10948 S    mastiff
  925 admin 11104 S    console
  971 admin 11104 S    /sbin/PS_pod
1024 admin 11104 S    /sbin/wanduck
1063 admin 11104 S    ATE Get_WanLanStatus
1198 admin 11104 S    wpsaide
1810 admin 11104 S    check_watchdog
1839 admin 11104 S    bwdpi_check
2129 admin 11104 S    usbled
2154 admin 11104 S    disk_monitor
1809 admin 11108 S    watchdog
1842 admin 11108 S    pctime
1072 admin 11288 S    nt_center
    1 admin 12768 S    /sbin/init
1856 admin 13040 S    conn_diag
1062 admin 13152 S    /sbin/netool
1060 admin 13240 S    nt_monitor
1863 admin 14336 S    amas_lib
  285 admin 18508 S    /bin/swmdk
1854 admin 19316 S    roamast

 

[DTS]

Does this look odd to you?

1975 admin  130m S    httpds -s -i br0 -p 8443

I have had a connection to the web GUI since the last reboot, and I glance at it approx once or twice daily. I am generally leaving the GUI on the same page -- the system log page.

However, I have noticed since upgrading to this firmware version that the web GUI does become unstable over time.

A couple days ago I navigated to the LAN page (to check lease time config). After navigating back to the system log page, it did not display properly. I had to refresh the page twice before it rendered as expected.

In prior cases of this issue, the web GUI has become completely unresponsive.

 

[RMerlin]

/tmp uses tmpfs, which is in RAM. If you are low on RAM, them tmpfs will be unable to allocate space when writing to it.

Your web server is using an abnormally high amount of RAM. Try restarting the service.

service restart_httpd

Why that is so, I have no idea.

 

[DTS]

/tmp uses tmpfs, which is in RAM. If you are low on RAM, them tmpfs will be unable to allocate space when writing to it.

Your web server is using an abnormally high amount of RAM. Try restarting the service.

service restart_httpd

# service restart_httpd

Done.

# free
             total       used       free     shared    buffers     cached
Mem:        440324     291220     149104       1608          0       5848
-/+ buffers/cache:     285372     154952
Swap:            0          0          0

Why that is so, I have no idea.

Our messages crossed in cyberspace. This issue started with this version of the firmware.

I saw it when I upgraded my other Asus. Then I still saw it when I purchased a new Asus and installed this firmware first thing. In my experience it is related to this firmware version.