RT-AC87U with 380.65. One core stuck at 100%

[VZ3]

Happens out of the blue. Top command shows nothing. There are no activities, no traffic. Core 2 shows 100% utilization.

Reboot did not help. Only power cycle did reset CPU to normal. Log shows nothing unusual.

Well, this got me baffled... Have anybody had this issue?

 

[joro_abv]

I had this with 380.62_1 on AC56U, so it is not version specific nor router specific.

 

[chamberc]

It happens on various Merlin and stock versions for years. Happens on the AC88u as well.

 

[hggomes]

Please show us a printscreen of "top" command.

 

[VZ3]

I did not saved top printscreen when it happened but the top processes was httpd with about 1% utilization and top command with similar stats.
According to top command utilization should not be more then 2-3% but UI showed Core 2 pegged at 100% solid, Core 1 was at usual 1-2%. And like I said, reboot did not helped only power cycle.

 

[hoang51]

Do you happen to have a USB device plugged in? I experienced a single core at 100% CPU utilization rate on my RT-AC68R if I was trying to access my USB HDD with 380.65 firmware. I had to downgrade to 380.64_2 to have stabilized CPU utilization and so far, I haven't experienced any issue going back to 380.64_2.

 

[VZ3]

Do you happen to have a USB device plugged in? I experienced a single core at 100% CPU utilization rate on my RT-AC68R if I was trying to access my USB HDD with 380.65 firmware. I had to downgrade to 380.64_2 to have stabilized CPU utilization and so far, I haven't experienced any issue going back to 380.64_2.

Yes, I have. But at that moment I was not using it. And how to explain that there was no activity in top command. Also reboot did not helped.
Anyway, I will try to mingle with USB drive next time it happens.

 

[RMerlin]

Processes aren't the only thing that can use the CPU. The kernel and device drivers can, too. That can be seen by the info shown at the top of top's output, for instance under SIRQ.

 

[VZ3]

Processes aren't the only thing that can use the CPU. The kernel and device drivers can, too. That can be seen by the info shown at the top of top's output, for instance under SIRQ.

Thanks for the input. I cannot say for sure (just don't remember) regarding interrupts/software interrupts requests numbers but I am sure about CPU's usr/sys and Load averages numbers... nothing was at 100% even at 50%. Those numbers was looks like router was idling. That's what baffled me. Where did UI get it's stats? How to verify that Core 2 has indeed that kinda load? What is causing it?
I know, that's a lot's of questions... just sharing what I saw by speaking out loud in hope to find reasonable explanation.

 

[CivHunter]

I also had Core #1 stuck at 100% after recently flashing from 380.64_2 to 380.5. The only parameter I changed afterwards was to turn off per IP Logging and enabled CTF on the LAN page. It ran normally for a day or two and then I noticed Core #1 at 100%. I tried to run a scan on my 8GB USB thumb drive that plugged into the router and it would never unmount (about 15% of the way through the process). I powered down the router and ran a check disk on it using my Windows 10 PC. It found errors and fixed them. I plugged it back into the router and powered up. After that I was able to run a scan on the USB drive using the router. Also after reboot Core #1 is back to idle. I just now did all of this. If I see a change in Core #1 or the disk I will post again.

 

[Matteo Guglielmi]

I've been experiencing the same problem with all my 4 routers (2 x RTAC87U, 1 x RTAC3200, 1 x RTAC68U) with all firmware versions released so far.

I promised myself to stop upgrading the firmware (at least for a while...) as soon as I find a version which is "spinning-core" free... no luck so far but with earlier versions of the firmware it was much worse (once per week) than it's today with 380.64-2 (once per 1 or 2 months).

When the core spins @ 100%, rebooting the router via the web interface usually does not work.

A dirty solution is to ssh into the router and run this "kernel-reset" command:

echo 1 > /proc/sys/kernel/sysrq ; echo b > /proc/sysrq-trigger

The router will reboot immediately without attempting a clean shutdown and will come back online just fine.

 

[RMerlin]

The most likely cause is the USB disk plugged in, and the media server scanning its content. It's not a bug in the firmware, so you can't just "wait for a firmware without the issue".

 

[Matteo Guglielmi]

The most likely cause is the USB disk plugged in, and the media server scanning its content. It's not a bug in the firmware, so you can't just "wait for a firmware without the issue".

I have no usb disk plugged in... I never had one.

In my case I've been always using firmwares with only vpn turned on (nothing fancy, really)... but I found all of them with one core spinning several times over the last year or so.

In particular there was a specific version of the firmware (which i don't recall now) that was really giving me lot of problems... all the router were ending up quite quickly with a spinning core.. I also posted about that on the forum.

But the problem is not solved yet.

In fact, as of yesterday, I found two of my routers running on 380.64-2 (installed in very separate places actually) with a spinning core again... but for ~60 days they have been running fine so I don't mind that much anymore... a reset every two months is ok.

I still don't know why and when this happens though, as VZ3 says, there are no error messages in the logs.

EDIT:

Just like these random spikes of 1 and 2 GB of the Traffic Analyzer which never went away across all firmware versions:

 

[hoang51]

I believe I have narrowed down why one of my core CPU was at 100%, and it was due to a failing USB HDD that was plugged into the USB port on RT-AC68R router with firmwares 380.64_2 and 380.65. One can read it here: https://www.snbforums.com/threads/r...65-is-now-available.37295/page-21#post-311852