AC68U Core 1 or 2 goes 100% after a couple weeks

[HiVolt]

I have a an AC68U running Merlin 380.63_2, I've already updated a couple times from minor versions before...

Here's a problem I've got. After about 2 weeks runtime, one of the cores, sometimes its Core 1 sometimes Core 2 gets pegged at 100%. I telnet to the router, and run "top" and this is what it shows...

Mem: 73136K used, 182564K free, 0K shrd, 2120K buff, 8276K cached
CPU: 0.0% usr 0.1% sys 0.0% nic 47.3% idle 47.9% io 0.0% irq 4.5% sirq
Load average: 4.03 4.04 4.05 1/85 13805

Looks like NIC is showing nearly 50% usage so i assume whatever this is, its hogging the resources. None of the actual processes are showing high usage. The router still works fine I don't see any adverse effects but If i reboot it it will be fine for a couple weeks then it will happen again.

I'm running it in Dual WAN mode if it makes a difference, with primary WAN port being Fibre 250/250, and a failover Cable 30/5 connection on LAN port 4.

Any help would be appreciated, thanks.

 

[john9527]

labels are after the value....so you are

47.3% idle
47.9% io

Do you have a USB drive connected? Running Media Server? Also, check the syslog for any disk errors.

 

[HiVolt]

Oh, sorry. I thought it was NIC...

Nope, no media server, no usb connected... this is in an office purely as a router not running any fancy stuff...

 

[HiVolt]

I went thru all the options, there are no AI protection, no AI cloud turned on, no VPN, no media servers. As I was going thru the web interface crashed... So i had to reboot...

I would like to figure this out somehow, its gonna be a bit of a pain in the butt if it keeps happening.

 

[john9527]

Do you have a copy of the top output? If not, next time it happens see what process is at the top of the 'top' list.

 

[Reinvented]

When you were flashing different firmwares, I'm assuming you were dirty flashing, and also loading up a config file as well? Or were you setting up stuff from scratch after clearing nvram?

 

[HiVolt]

This is fairly new router only set up less than 2 months... When I first got it, I flashed it with latest Merlin and reset all values to factory... I have since flashed it twice with incremental updates hoping to fix this glitch, but did not clear or reset anything, nor did I load any config file.

I'm sure this will happen again in ~2weeks, so I will post the "top" output then. But from what I recall there was no process with higher than 1-2% thruout the list. Thats what baffled me.

 

[RMerlin]

Check your traffic meter while that happens. Generally, io usage will be either USB or network.

 

[HiVolt]

Check your traffic meter while that happens. Generally, io usage will be either USB or network.

I did, there was minimal traffic and there has never been any USB plugged in. This has happened 3 times alerady, and it's roughly 2 weeks after boot/reboot. If you have specific things I should check next time I'm willing to provide you with any details you need.

I remember checking the traffic before when maxing out the line at 250mbps, and only one of the cores was around 40%, so one core being stuck at 100% shouldn't be a result of any network traffic IMO.

 

[HiVolt]

I hate to reply to this old thread, but I'm still having this issue, on and off... It has once gone like 3-4 months without one of the cores being pegged at 100%, but sometimes it does it like a week after reboot...

I have upgraded the firmware several times and currently sit at Merlin 380.68 on the AC68u. If you google this problem, it seems it happens to all kinds of ASUS routers on stock or Merlin fw... It's unfortunate this particular revision C does not support Tomato or I would try it to see if it screws up.

The TOP readout is below.

Mem: 74252K used, 181456K free, 1116K shrd, 1072K buff, 7664K cached
CPU: 0.3% usr 0.2% sys 0.0% nic 46.4% idle 48.1% io 0.0% irq 4.7% sirq
Load average: 5.04 5.08 5.11 1/91 10273
PID PPID USER STAT VSZ %VSZ CPU %CPU COMMAND
1554 1 admin S 916 0.3 0 0.4 miniupnpd -f /etc/upnp/config
480 1 admin S 1320 0.5 1 0.2 protect_srv
504 1 admin S 3092 1.2 1 0.1 avahi-daemon: running [RT-AC68U-DB38.local]
511 1 admin S 7288 2.8 0 0.0 watchdog
506 1 admin S 6708 2.6 0 0.0 httpd -i br0
505 1 admin S 6364 2.4 0 0.0 httpds -s -p 8443 -i br0
1 0 admin S 6232 2.4 0 0.0 /sbin/preinit
954 1 admin S 6220 2.4 0 0.0 bwdpi_wred_alive
479 1 admin S 6220 2.4 0 0.0 /sbin/wanduck
531 1 admin S 6220 2.4 0 0.0 erp_monitor
847 1 admin S 6220 2.4 0 0.0 ntp
4271 1 admin S 6220 2.4 0 0.0 usbled
529 1 admin S 6220 2.4 1 0.0 hour_monitor
897 1 admin S 6220 2.4 1 0.0 disk_monitor
515 511 admin S 6220 2.4 0 0.0 ots
491 1 admin S 6220 2.4 1 0.0 wpsaide
528 1 admin S 6220 2.4 1 0.0 bwdpi_check
352 1 admin S 6212 2.4 0 0.0 console
10850 10849 admin D 5252 2.0 0 0.0 bwdpi_sqlite -e -s NULL
963 959 admin S 4004 1.5 1 0.0 data_colld -i 1800 -p 43200 -b -w /tmp/bwdpi/dc/
958 1 admin S 4004 1.5 0 0.0 data_colld -i 1800 -p 43200 -b -w /tmp/bwdpi/dc/
974 959 admin S 4004 1.5 0 0.0 data_colld -i 1800 -p 43200 -b -w /tmp/bwdpi/dc/
959 958 admin S 4004 1.5 0 0.0 data_colld -i 1800 -p 43200 -b -w /tmp/bwdpi/dc/
961 959 admin S 4004 1.5 0 0.0 data_colld -i 1800 -p 43200 -b -w /tmp/bwdpi/dc/
962 959 admin S 4004 1.5 0 0.0 data_colld -i 1800 -p 43200 -b -w /tmp/bwdpi/dc/
960 959 admin S 4004 1.5 0 0.0 data_colld -i 1800 -p 43200 -b -w /tmp/bwdpi/dc/
915 914 admin S 2724 1.0 0 0.0 wred -B
914 913 admin S 2724 1.0 1 0.0 wred -B
913 1 admin S 2724 1.0 0 0.0 wred -B
916 914 admin S 2724 1.0 0 0.0 wred -B
917 914 admin S 2724 1.0 1 0.0 wred -B
922 914 admin S 2724 1.0 1 0.0 wred -B
923 914 admin S 2724 1.0 0 0.0 wred -B
920 914 admin S 2724 1.0 1 0.0 wred -B
921 914 admin S 2724 1.0 0 0.0 wred -B
918 914 admin S 2724 1.0 1 0.0 wred -B
919 914 admin S 2724 1.0 0 0.0 wred -B
516 1 admin S 2296 0.9 1 0.0 rstats
4276 4269 admin S 2200 0.8 1 0.0 u2ec
4269 1 admin S 2200 0.8 0 0.0 u2ec
4277 4276 admin S 2200 0.8 1 0.0 u2ec
4266 1 admin D 1984 0.7 0 0.0 networkmap
4237 1 admin S 1832 0.7 1 0.0 nas
4238 1 admin S 1804 0.7 1 0.0 /bin/wps_monitor
1557 1 admin S 1452 0.5 0 0.0 /sbin/udhcpc -i vlan3 -p /var/run/udhcpc1.pid -s /tmp/udhcpc -O33 -O249
482 1 admin S 1452 0.5 0 0.0 telnetd -b 192.168.1.1
10169 482 admin S 1448 0.5 1 0.0 -sh
10246 10169 admin R 1448 0.5 0 0.0 top
354 352 admin S 1444 0.5 0 0.0 /bin/sh
10933 511 admin D 1444 0.5 0 0.0 cp /tmp/syslog.log /tmp/syslog.log-1 /jffs
508 1 admin S 1440 0.5 1 0.0 crond
369 1 admin S 1440 0.5 0 0.0 /sbin/klogd -c 5
10849 529 admin S 1440 0.5 1 0.0 sh -c bwdpi_sqlite -e -s NULL
367 1 admin S 1436 0.5 1 0.0 /sbin/syslogd -m 0 -S -O /tmp/syslog.log -s 256 -l 7
550 1 admin S 1436 0.5 1 0.0 /sbin/zcip -p /var/run/zcip0.pid vlan35 /tmp/zcip
553 1 admin S 1392 0.5 0 0.0 /usr/sbin/pppd file /tmp/ppp/options.wan0
15686 1 nobody S 1376 0.5 0 0.0 dnsmasq --log-async
486 480 admin S 1320 0.5 1 0.0 protect_srv
487 486 admin S 1320 0.5 0 0.0 protect_srv
4250 1 admin S 1316 0.5 1 0.0 /usr/sbin/acsd
4270 1 admin S 1236 0.4 0 0.0 lpd br0
523 1 admin S 1208 0.4 1 0.0 lld2d br0
4235 1 admin S 1192 0.4 1 0.0 /bin/eapd
509 1 admin S 1164 0.4 1 0.0 /usr/sbin/infosvr br0
293 1 admin S 664 0.2 0 0.0 hotplug2 --persistent --no-coldplug
admin@RT-AC68U-DB38:/tmp/home/root#

 

[VZ3]

I have the same issue with AC87R and 380.68. It something with disk IO, somewhere in kernel. There is nothing unusual to see in TOP command other that one CPU is waiting/looping/got stuck on IO.

PS: I have an impression that Web UI is kinda facilitated/triggered that bug. Or maybe it got stuck when trying to write log or traffic db into jffs. So, after that traffic.db got corrupted, so I have to delete that file.

 

[selenajohn]

labels are after the value....so you are

47.3% idle
47.9% io

 

[john9527]

Also, if you press the '1' key after starting top, it will show your the stats for each CPU individually.

 

[ColinTaylor]

labels are after the value....so you are

47.3% idle
47.9% io

As previously pointed out in post #2.

 

[VZ3]

Again, there is nothing to see in top command except that one core stuck with high IO.

 

[Vexira]

enable reboot scheduler and set it to a time when there's no running traffic, say 4am that what I did. But mines set to reboot once a day to keep every thing running smoothly you might what it to reboot once a week.

 

[HiVolt]

enable reboot scheduler and set it to a time when there's no running traffic, say 4am that what I did. But mines set to reboot once a day to keep every thing running smoothly you might what it to reboot once a week.

I thought about that, except I have mine set up in a Dual WAN config, Primary being FTTH with PPPoE authentication, and the Secondary backup being a Cable connection with DHCP, and after a reboot the secondary doesn't get its DHCP IP, i have to toggle it manually to static then back to DHCP and then it works...

 

[HiVolt]

Also, if you press the '1' key after starting top, it will show your the stats for each CPU individually.

Mem: 72940K used, 182768K free, 1148K shrd, 732K buff, 7164K cached
CPU0: 0.3% usr 0.9% sys 0.0% nic 96.8% idle 0.0% io 0.0% irq 1.7% sirq
CPU1: 0.0% usr 0.5% sys 0.0% nic 0.0% idle 96.0% io 0.0% irq 3.3% sirq
Load average: 5.00 5.02 5.05 1/91 9368

That just shows me the same thing as the main GUI page. That's how I notice that one of the CPU's has gone 100%, I don't check via terminal.

I just don't get why this problem has persisted for this long on these routers... There has to be some core flaw in the firmware, I don't think it would be hardware since there are many varieties.

I don't have any USB disks plugged in, I don't use it as any sort of media server, I have nothing connecting in or out of it. I just use it as a router.

 

[john9527]

Here's a SWAG.....do a backup of JFFS, then select 'Reformat at next boot', reboot the router twice in a row (sometimes it takes two passes for a successful reformat), then restore the JFFS backup and reboot again. Maybe there's a bad block in jffs that's not being handled correctly.

 

[HiVolt]

Whats the JFFS partition do anyway?

Format JFFS partition at next boot - NO
Enable JFFS custom scripts and configs - NO

Those are the settings I have. If there is no scripts/configs turned on, does that matter?