Asus Merlin SNMP doesn't report Number of CPU Cores (UCD-SNMP-MIB::ssCpuNumCpus.0) ?

[ninja888]

Hi There,

First off, thanks for the Asus Merlin Firmware - It's awesome!

As an SNMP server (v2c), although the Asus Merlin SNMP server does answer a remote snmpwalk of the following UCD-SNMP-MIB::systemStats table, it strangely does not include the UCD-SNMP-MIB::ssCpuNumCpus.0 OID (which returns the number of CPU Cores in the router - Should return 2 for my router)

$ snmpwalk -v 2c -c <SNMP COMMUNITY> <ROUTER IP> UCD-SNMP-MIB::systemStats
UCD-SNMP-MIB::ssIndex.0 = INTEGER: 1
UCD-SNMP-MIB::ssErrorName.0 = STRING: systemStats
UCD-SNMP-MIB::ssSwapIn.0 = INTEGER: 0 kB
UCD-SNMP-MIB::ssSwapOut.0 = INTEGER: 0 kB
UCD-SNMP-MIB::ssIOSent.0 = INTEGER: 0 blocks/s
UCD-SNMP-MIB::ssIOReceive.0 = INTEGER: 0 blocks/s
UCD-SNMP-MIB::ssSysInterrupts.0 = INTEGER: 188 interrupts/s
UCD-SNMP-MIB::ssSysContext.0 = INTEGER: 4896 switches/s
UCD-SNMP-MIB::ssCpuUser.0 = INTEGER: 1
UCD-SNMP-MIB::ssCpuSystem.0 = INTEGER: 1
UCD-SNMP-MIB::ssCpuIdle.0 = INTEGER: 97
UCD-SNMP-MIB::ssCpuRawUser.0 = Counter32: 323666
UCD-SNMP-MIB::ssCpuRawNice.0 = Counter32: 0
UCD-SNMP-MIB::ssCpuRawSystem.0 = Counter32: 303516
UCD-SNMP-MIB::ssCpuRawIdle.0 = Counter32: 28658661
UCD-SNMP-MIB::ssCpuRawWait.0 = Counter32: 22900
UCD-SNMP-MIB::ssCpuRawKernel.0 = Counter32: 0
UCD-SNMP-MIB::ssCpuRawInterrupt.0 = Counter32: 0
UCD-SNMP-MIB::ssIORawSent.0 = Counter32: 0
UCD-SNMP-MIB::ssIORawReceived.0 = Counter32: 0
UCD-SNMP-MIB::ssRawInterrupts.0 = Counter32: 283879485
UCD-SNMP-MIB::ssRawContexts.0 = Counter32: 787936477
UCD-SNMP-MIB::ssCpuRawSoftIRQ.0 = Counter32: 323897
UCD-SNMP-MIB::ssRawSwapIn.0 = Counter32: 0
UCD-SNMP-MIB::ssRawSwapOut.0 = Counter32: 0
UCD-SNMP-MIB::ssCpuRawSteal.0 = Counter32: 0
UCD-SNMP-MIB::ssCpuRawGuest.0 = Counter32: 0
UCD-SNMP-MIB::ssCpuRawGuestNice.0 = Counter32: 0


$ snmpwalk -v 2c -c <SNMP COMMUNITY> <ROUTER IP> UCD-SNMP-MIB::ssCpuNumCpus.0
UCD-SNMP-MIB::ssCpuNumCpus.0 = No Such Object available on this agent at this OID

This OID (to get the number of CPUs in the router) is useful when doing an snmpwalk across multiple (not just Asus Merlin routers) devices so that the average CPU Utilisation can be calculated as:

(HOST-RESOURCES-MIB::hrProcessorLoad.196608 + HOST-RESOURCES-MIB::hrProcessorLoad.196609 + HOST-RESOURCES-MIB::hrProcessorLoad.196610 + HOST-RESOURCES-MIB::hrProcessorLoad.196611) / UCD-SNMP-MIB::ssCpuNumCpus.0

Is there any reason why this OID is not returned? For other devices on my network, this OID is returned as expected...

FYI - I'm using Asus RT-AC88U - Firmware 384.6...

EDIT: Also updated to Firmware 384.7_2 and see the same issue...

Thanks!

 

[RMerlin]

I never used SNMP, and the net-snmp documentation is half useless to explain which of the 30+ modules supports what. You will have to tell me which net-snmp module is missing...

 

[ninja888]

Hi @RMerlin,

I'm assuming that all of the standard SNMP MIBs that are included in the Asus Merlin firmware releases are these exact versions at the following URL:

https://github.com/RMerl/asuswrt-merlin.ng/tree/master/release/src/router/net-snmp-5.7.2/mibs

If so, the issue seems to be that the standard (non-Asus-specific) MIBs that are included in the firmware are quite old and are missing some standard OIDs.

As an example (I'm sure there are others...) the UCD-SNMP-MIB.txt MIB at the above URL has the line LAST-UPDATED "200901190000Z" in the file yet, my Ubuntu and NAS versions of UCD-SNMP-MIB have the line LAST-UPDATED "201407310000Z", indicating the file at the URL above is 5 years older...

My UCD-SNMP-MIB.txt also has the following definition for ssCpuNumCpus which is missing from the UCD-SNMP-MIB.txt at the URL above:

ssCpuNumCpus OBJECT-TYPE
SYNTAX Integer32
MAX-ACCESS read-only
STATUS current
DESCRIPTION
"The number of processors, as counted by the agent.
This object's value may be useful in the management
of certain operating systems where notions such as
load average do not take into account the number of
processors in the system.
For other objects in the systemStats group whose
descriptions refer to 'N processors', this object's
value is N."
::= { systemStats 67 }

To get the latest versions of these MIB files, this is how I did it on my Ubuntu 18.10 install (and also on my NAS):

• Install SNMP (Client) - https://www.digitalocean.com/commun...ure-an-snmp-daemon-and-client-on-ubuntu-14-04:
o sudo apt-get install snmp-mibs-downloader
o sudo download-mibs

Running download-mibs then grabs the latest versions of the MIBs files and copies them to /usr/share/snmp/mibs)...

Given the above:

(A) Is it possible to update the MIBs to the latest versions in the Asus Merlin Firmware itself (possibly this may have to be an upstream fix from Asus)? Possibly also include an option in the Web GUI to run "download-mibs" to grab the latest MIB files?
(B) Is it possible for me to "inject" the updated MIBs into my current Asus Merlin Firmware (384.7_2), possibly by just scp'ing the latest MIB.txt files themselves over the existing ones using SSH, rather than installing snmp-mibs-downloader and running download-mibs on the router itself)? I assume that, if I get this working but the firmware isn't patched to include the latest MIB updates in the future, I'd have to do this on each new firmware release?

Thanks again!

 

[RMerlin]

I'm assuming that all of the standard SNMP MIBs that are included in the Asus Merlin firmware releases are these exact versions at the following URL:

No. The list of MIBs I enable are here:

https://github.com/RMerl/asuswrt-merlin.ng/blob/master/release/src/router/Makefile#L5436

(A) Is it possible to update the MIBs to the latest versions in the Asus Merlin Firmware itself (possibly this may have to be an upstream fix from Asus)? Possibly also include an option in the Web GUI to run "download-mibs" to grab the latest MIB files?

I am only one version behind the last net-snmp release (unless they issued a new one very recently).

(B) Is it possible for me to "inject" the updated MIBs into my current Asus Merlin Firmware (384.7_2), possibly by just scp'ing the latest MIB.txt files themselves over the existing ones using SSH, rather than installing snmp-mibs-downloader and running download-mibs on the router itself)? I assume that, if I get this working but the firmware isn't patched to include the latest MIB updates in the future, I'd have to do this on each new firmware release?

No, you need to enable the necessary modules at compile time. If the supported MIBs are not up-to-date, you will have to tell that to the net-snmp authors.

 

[ninja888]

No. The list of MIBs I enable are here:

https://github.com/RMerl/asuswrt-merlin.ng/blob/master/release/src/router/Makefile#L5436



I am only one version behind the last net-snmp release (unless they issued a new one very recently).



No, you need to enable the necessary modules at compile time. If the supported MIBs are not up-to-date, you will have to tell that to the net-snmp authors.

Hi @RMerlin,

From the makefile URL you sent it looks like net-snmp-5.7.2 is being used and this appears to be quite a bit older than the more recent versions (appears to be 6 years older than latest):

From https://sourceforge.net/projects/net-snmp/files/net-snmp/ :
5.7.2 = 2012-10-09 (this appears to be the version used in the Asus Merlin Firmware and this is missing the ssCpuNumCpus OID from UCD-SNMP-MIB.txt)
5.7.3 = 2014-12-08 (this includes the missing ssCpuNumCpus OID and a few more...)
5.8 = 2018-07-16 (this includes the missing ssCpuNumCpus OID and a few more...)

Is there any chance you could bump the net-snmp package to the latest (5.8)?

Many Thanks!

 

[RMerlin]

Hi @RMerlin,

From the makefile URL you sent it looks like net-snmp-5.7.2 is being used and this appears to be quite a bit older than the more recent versions (appears to be 6 years older than latest):

From https://sourceforge.net/projects/net-snmp/files/net-snmp/ :
5.7.2 = 2012-10-09 (this appears to be the version used in the Asus Merlin Firmware and this is missing the ssCpuNumCpus OID from UCD-SNMP-MIB.txt)
5.7.3 = 2014-12-08 (this includes the missing ssCpuNumCpus OID and a few more...)
5.8 = 2018-07-16 (this includes the missing ssCpuNumCpus OID and a few more...)

Is there any chance you could bump the net-snmp package to the latest (5.8)?

Many Thanks!

I can take a look, yes. At the time I enabled SNMP support, 5.7.3 was still the latest, and it didn't show much changes, so I didin't bother updating it at the time.

 

[RMerlin]

I successfully upgraded to 5.8, NumCpu is now properly returned.

(unfortunately, still no go on HND support, snmpd still causes a kernel crash as it tries to probe the Broadcom switch interfaces).

 

[ninja888]

I successfully upgraded to 5.8, NumCpu is now properly returned.

(unfortunately, still no go on HND support, snmpd still causes a kernel crash as it tries to probe the Broadcom switch interfaces).

Fantastic! Is there a beta release I can test with or will the new net-snmp package be released in the next update?

Also, what's HND support?

Many thanks again for taking a look into this!

 

[RMerlin]

Fantastic! Is there a beta release I can test with or will the new net-snmp package be released in the next update?

Also, what's HND support?

Many thanks again for taking a look into this!

It should be part of the next batch of alpha builds that I occasionally push to the Test Folder on Onedrive.

HND is the Broadcom platform used by the RT-AC86U and RT-AX88U. It has a newer kernel and a new SDK environment, which so far is not compatible with net-snmp. Starting snmpd causes the router to reboot while net-snmp tries to query all the available network interfaces.

 

[ninja888]

Hi @RMerlin.

Just wanted to say thanks for releasing 384.8_Beta1 with the new version of net-snmp. I've tested and it now returns UCD-SNMP-MIB::ssCpuNumCpus.0 as expected - Thanks!

Also - Recently noticed the following issue on both 384.7_2 and the new 384.8_Beta1 firmware - Not sure if I should raise this as a new issue or whether this is an issue with net-snmp itself?

I recently noticed that IF-MIB::ifHCInOctets (High Capacity Counter for Bytes-In-Since-Reboot for eth2 (5GHz Wifi)) and IF-MIB::ifInOctets (Counter for Bytes-In-Since-Reboot for eth2 (5GHz Wifi)) always returns 0 rather than the actual number of bytes/octets... These counters do increment as expected for eth0 (LAN) and eth1 (2.4GHz Wifi) - Indeed the ifHCOutOctets and ifOutOctets increment for eth2, just not the ifHCInOctets / ifInOctets counters? Example below including the interface descriptions for IF-MIB::ifHCInOctets (same is seen for the ifInOctets counter for eth2)...

$ snmpwalk -v 2c -c <COMMUNITY ID> <ROUTER IP> IF-MIB::ifDescr
IF-MIB::ifDescr.1 = STRING: lo
IF-MIB::ifDescr.2 = STRING: ifb0
IF-MIB::ifDescr.3 = STRING: ifb1
IF-MIB::ifDescr.4 = STRING: fwd0
IF-MIB::ifDescr.5 = STRING: fwd1
IF-MIB::ifDescr.6 = STRING: agg
IF-MIB::ifDescr.7 = STRING: eth0
IF-MIB::ifDescr.8 = STRING: dpsta
IF-MIB::ifDescr.9 = STRING: eth1
IF-MIB::ifDescr.10 = STRING: eth2
IF-MIB::ifDescr.11 = STRING: vlan1
IF-MIB::ifDescr.12 = STRING: vlan2
IF-MIB::ifDescr.13 = STRING: br0
IF-MIB::ifDescr.14 = STRING: ppp0
IF-MIB::ifDescr.15 = STRING: tun11

$ snmpwalk -v 2c -c <COMMUNITY ID> <ROUTER IP> IF-MIB::ifHCInOctets
IF-MIB::ifHCInOctets.1 = Counter64: 606162
IF-MIB::ifHCInOctets.2 = Counter64: 0
IF-MIB::ifHCInOctets.3 = Counter64: 0
IF-MIB::ifHCInOctets.4 = Counter64: 0
IF-MIB::ifHCInOctets.5 = Counter64: 0
IF-MIB::ifHCInOctets.6 = Counter64: 0
IF-MIB::ifHCInOctets.7 = Counter64: 24436600
IF-MIB::ifHCInOctets.8 = Counter64: 0
IF-MIB::ifHCInOctets.9 = Counter64: 4503
IF-MIB::ifHCInOctets.10 = Counter64: 0
IF-MIB::ifHCInOctets.11 = Counter64: 8976503
IF-MIB::ifHCInOctets.12 = Counter64: 0
IF-MIB::ifHCInOctets.13 = Counter64: 7660498
IF-MIB::ifHCInOctets.14 = Counter64: 14868819
IF-MIB::ifHCInOctets.15 = Counter64: 654226

$ snmpwalk -v 2c -c <COMMUNITY ID> <ROUTER IP> IF-MIB::ifHCOutOctets
IF-MIB::ifHCOutOctets.1 = Counter64: 1416345
IF-MIB::ifHCOutOctets.2 = Counter64: 0
IF-MIB::ifHCOutOctets.3 = Counter64: 0
IF-MIB::ifHCOutOctets.4 = Counter64: 0
IF-MIB::ifHCOutOctets.5 = Counter64: 3086728
IF-MIB::ifHCOutOctets.6 = Counter64: 0
IF-MIB::ifHCOutOctets.7 = Counter64: 810154770
IF-MIB::ifHCOutOctets.8 = Counter64: 0
IF-MIB::ifHCOutOctets.9 = Counter64: 659166
IF-MIB::ifHCOutOctets.10 = Counter64: 33972766
IF-MIB::ifHCOutOctets.11 = Counter64: 493763243
IF-MIB::ifHCOutOctets.12 = Counter64: 0
IF-MIB::ifHCOutOctets.13 = Counter64: 15516206
IF-MIB::ifHCOutOctets.14 = Counter64: 315929746
IF-MIB::ifHCOutOctets.15 = Counter64: 614916


Thanks again!

 

[RMerlin]

The Broadcom architecture is fairly complex, so I wouldn't be surprised if the counters wouldn't be accurate on some of the interfaces, as you factor various blackbox technologies such as CTF and the internal wireless forwarders (fwd0 and fwd1). For a long time, the eth0 counters were completely out of whack even when polling /proc/net/dev (and I don't know if it ever was fixed). Unless someone is an engineer with access to the proprietary Broadcom SDK documentation, there's nothing to be done about it, and it's certainly way beyond my field of expertise.

 

[Ari_el]

@RMerlin /proc/net/dev is still not fixed as of today (AC 3100 here with the latest asus-merlin), counters are all mixed up even with hardware acceleration disabled / CTF off. I understand this issue is related to the old kernel version that is used (2.6.x) when virtual bridges are in use.

Question - is there a way to access the broadcom driver counters through some tool/utility? I was thinking about extending SNMP (creating some private OIDs) to expose the data as an alternative to the broken default snmp behavior which uses /proc/net/dev of interface counters.

We know the web UI of the router is not affected by the /proc/net/dev issue with the counters, so I was hoping we could find a way to query the right counters

 

[RMerlin]

Question - is there a way to access the broadcom driver counters through some tool/utility?

Closed source/proprietary. Asus actually moved code related to that in libshared into a closed source portion a few years ago.