Relatively high load with idle CPU on ASUS RT-AC86U with ASUSWRT-Merlin

[StR]

I am not experiencing any problems, so, at this point it is just a curiosity.

I've just noticed that the "load", as reported by top (1) and can be seen in /proc/loadavg is consistently (over the past 20+ minutes) sitting above 2.70 (sometimes spiking to over 3.00), while the CPU is 99.x% idle, occasionally going down to 98.x% for very short time. (After about half an hour, I've seen the 1-min average coming down to 2.20 for a minute or less. but now I see it being 3.28 with 99.7% CPU idle).
During this period of time, the network activity is very low: I am reading some static text content, and my wife has her laptop on with her VPN client connected, but she is doing things locally, i.e. no large network traffic.

cat /proc/loadavg
2.87 2.85 2.84 1/163 17949

Mem: 348780K used, 91576K free, 17064K shrd, 4K buff, 62764K cached
CPU:  0.0% usr  0.0% sys  0.0% nic  100% idle  0.0% io  0.0% irq  0.0% sirq
Load average: 2.85 2.90 2.87 1/162 18289
  PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
 1919     1 admin    S    19848  4.5   1  0.0 roamast
  285     1 admin    S    18508  4.1   1  0.0 /bin/swmdk
...

(this is done by from the router's prompt via ssh connection).

Any idea why the load is reported being this high?
Do you see this on your router? Is is just on my router, or this is somehow a common "feature"?

I am currently running ASUSWRT-Merlin v.386.5_2, but don't know if it is version specific, as I don't think I've ever looked at the load numbers before.

 

[ColinTaylor]

IIRC that's normal for that router. There were various posts about it when the RT-AC86U was released. I don't know the reason but they (ARM) seem to have changed the formula used to create the load numbers.

EDIT: https://www.snbforums.com/threads/r...-6-is-now-available.47941/page-36#post-434812

 

[StR]

Thank you for that link!

(Having worked with different *nix systems for many moons, I feel weird seeing these erroneous load numbers.)

 

[Martinski]

FYI,

Here are a couple of screenshots showing the "Load Averages" as given by the top command from two RT-AC86U routers running at separate households. I've never seen the load averages go above 2.70, and that's even while I'm remotely connected to the routers using the OpenVPN Server with a lot of network activity going on. All other times whenever I've checked, the values are usually very close to 2.0 (mostly between 1.9x to 2.1x).

Within the context of the RT-AC86U router (which has a dual-core CPU), a "good" load average would be 2.0 or less since there are only 2 CPU cores to run jobs simultaneously. If the load averages are consistently much greater than 2.0, it means that at least one runnable process must wait in the run queue to take its turn (via the task scheduler) to get CPU time, while two other processes are actually being run. This may not necessarily be a big performance issue; it depends on what the expected response time is for the particular thread or task that's waiting in the queue, and how long it actually waits.

I do find it strange though that the Load Averages show the CPU (almost) "fully loaded" (i.e. 2.0 or close to it) while the CPU utilization shows very little CPU activity (e.g. 96.x% to 99.x% idle). Usually, the top command does show a lot of tasks/threads created, each of which may contend for a slice of CPU time. Granted, most of them appear to be mainly in a "sleep" state; but a few of them do keep taking their turn for CPU time fairly frequently. You can see this if you run the top command and look at its output for a couple of minutes.

Note that "high load averages" (e.g. much greater than 2.0 on the RT-AC86U) may not necessarily be a reflection of current/recent high network activity. It's better to think of CPU load averages more as a measurement of the demand that the overall system has on CPU resources at a particular point in time. The more runnable processes are kept waiting in the run queue the more overloaded the system can become. At a minimum, the 1-, 5-, and 15-minute averages can give you a trend of such CPU demand by the total number of processes contending for CPU time (i.e. number of processes waiting in the run queue *plus* those concurrently running on all existing CPU cores).

BTW, in Linux, "Load Averages" are time-dependent, exponentially-weighted moving averages of CPU loads calculated for the past 1, 5 & 15 minutes. These values are NOT calculated using the usual "arithmetic mean average" formula that you normally think of.

My 2 cents.

 

[Martinski]

I don't know the reason but they (ARM) seem to have changed the formula used to create the load numbers.

Are you saying that the chip manufacturer may have changed the Linux kernel "formula" used to calculate load averages?

I wonder which part of the "formula" they may have tweaked, and if so, that might also change the way the calculated values are interpreted.

 

[ColinTaylor]

Are you saying that the chip manufacturer may have changed the Linux kernel "formula" used to calculate load averages?

I wonder which part of the "formula" they may have tweaked, and if so, that might also change the way the calculated values are interpreted.

When this subject first came up I found something (that I can't find now ) on another forum that said something to that effect. But it's the internet so who knows how accurate that was. It may not have been something that ARM themselves had deliberately changed but rather the SDK for that processor using "different things" in the load calculation (that's my speculation). The Raspberry Pi had a kernel bug at one point that produced similarly high numbers. In that case certain types of idle processes were being counted as if they were active.

In any case the "high" load value seems to be an anomaly for the RT-AC86U as I'm not aware of it occurring on any other models. I'm inclined to think that the numbers are incorrect. Or at least don't represent what we all understand to be the system load.

 

[Martinski]

When this subject first came up I found something (that I can't find now ) on another forum that said something to that effect. But it's the internet so who knows how accurate that was. It may not have been something that ARM themselves had deliberately changed but rather the SDK for that processor using "different things" in the load calculation (that's my speculation). The Raspberry Pi had a kernel bug at one point that produced similarly high numbers. In that case certain types of idle processes were being counted as if they were active.

OK, understood. Thanks for the clarification.

In any case the "high" load value seems to be an anomaly for the RT-AC86U as I'm not aware of it occurring on any other models. I'm inclined to think that the numbers are incorrect. Or at least don't represent what we all understand to be the system load.

Yeah, I agree. The "load averages" reported on the RT-AC86U have often seemed "odd" to me since I bought the router back in 2019, particularly at the times when CPU utilization is showing an otherwise "idle" system. I have not seen the same issue either on other ASUS routers that I have access to.

Thanks again.