Stuck commands

[ColinTaylor]

Relatively freshly rebooted (few hours ago).

Ah, OK. That might not be helpful then.

If you have lsof installed you can try this:

grep " 50758 " /proc/net/netlink

Then take last the number on the output line (for example 51234) and use it in the lsof command:

lsof | grep " 51234 "

Here's an old example of mine:

# grep " 6891 " /proc/net/netlink
ffffffc02e265c00 31  6891   00000000 0        0        0 2        0        4712505
# lsof | grep " 4712505 "
wlceventd  6896                 admin    7u  netlink                         0t0    4712505 unknown protocol: 31

 

[Oracle]

I don't have lsof.
Let me know how and I'll add it.

I had to reboot because an addon got broken but no problem, errors will come again.
Matter of fact, one just came in:

Usage statistics of nvram and wl overrides
Uptime: 17:29:40 up 4:12, load average: 3.49, 3.22, 3.09
nvram use count: 572
nvram errors count: not available
wl use count: 1000
wl errors count: 1

 

[ColinTaylor]

I don't have lsof.
Let me know how and I'll add it.

If you have Entware installed, opkg install lsof

EDIT: You might have more luck if you add echo 4194304 > /proc/sys/kernel/pid_max at the beginning of your init-start script (and then reboot).

 

[Oracle]

Well, it looks like your example:

grep " 50758 " /proc/net/netlink
ffffffc008456c00 31  50758  00000000 0        0        0 2        0        97469

lsof | grep " 97469 "
httpd      17988                 admin    7u  netlink                         0t0   97469 unknown protocol: 31

 

[ColinTaylor]

OK that indicates that there might be an issue related to httpd. I see you've just created another thread about that anyway.

See the EDIT in my previous post. Might be worth doing that and then letting it run for a couple of days.

 

[Oracle]

Ok, added an init-start script in /jffs/scripts like this:

#!/bin/sh

echo 4194304 > /proc/sys/kernel/pid_max
logger -t init-start "Set pid_max to $(cat /proc/sys/kernel/pid_max)"

Rebooted.
pid_max got updated, the log line did not appear in the syslog.
I'll watch what happens now.

 

[Oracle]

This is after 16 hours of uptime. The output looks a bit different after the router had time to accumulate errors.

Print netlink sockets for which there is no process with the same number

Process 1092 not found
Process 1233 not found
Process 1239 not found
Process 1243 not found
Process 2089 not found
Process 2223 not found
Process 2381 not found
Process 2735 not found
Process 32771 not found
Process 33697 not found
Process associated with 4195398 not found (1092)
Process associated with 4195539 not found (1233)
Process associated with 4195545 not found (1239)
Process associated with 4195549 not found (1243)
Process associated with 4196395 not found (2089)
Process associated with 4196529 not found (2223)
Process associated with 4196687 not found (2381)
Process associated with 4197041 not found (2735)

A number in brackets is a *guess* at an associated process

ps w:

PID USER VSZ STAT COMMAND
1 admin 13440 S /sbin/init
2 admin 0 SW [kthreadd]
3 admin 0 SW [ksoftirqd/0]
5 admin 0 SW< [kworker/0:0H]
7 admin 0 SW [rcu_preempt]
8 admin 0 SW [rcu_sched]
9 admin 0 SW [rcu_bh]
10 admin 0 SW [migration/0]
11 admin 0 SW [watchdog/0]
12 admin 0 SW [watchdog/1]
13 admin 0 SW [migration/1]
14 admin 0 SW [ksoftirqd/1]
16 admin 0 SW< [kworker/1:0H]
17 admin 0 SW< [khelper]
18 admin 0 SW [kdevtmpfs]
19 admin 0 SW< [writeback]
20 admin 0 SW [kworker/0:1]
21 admin 0 SWN [ksmd]
22 admin 0 SW< [crypto]
23 admin 0 SW< [bioset]
24 admin 0 SW< [kblockd]
25 admin 0 DW [skbFreeTask]
26 admin 0 SW [bcmFapDrv]
27 admin 0 SWN [kswapd0]
28 admin 0 SW [fsnotify_mark]
53 admin 0 SW [btnhandler0]
54 admin 0 SW [btnhandler1]
55 admin 0 SW [btnhandler2]
56 admin 0 SW [kworker/1:1]
57 admin 0 SW< [linkwatch]
58 admin 0 SW< [ipv6_addrconf]
59 admin 0 SW< [deferwq]
60 admin 0 SW [ubi_bgt0d]
170 admin 0 SWN [jffs2_gcd_mtd2]
204 admin 0 SW [bcmFlwStatsTask]
208 admin 0 SW [bcmsw_rx]
209 admin 0 SW [bcmsw]
217 admin 0 SW [pdc_rx]
285 admin 18508 S /bin/swmdk
292 admin 1568 S {wdtctl} wdtd
296 admin 1712 S hotplug2 --persistent --no-coldplug
299 admin 1752 S /usr/sbin/envrams
362 admin 0 SWN [jffs2_gcd_mtd9]
538 admin 0 SW [dhd_watchdog_th]
539 admin 0 SW [wfd0-thrd]
546 admin 0 SW [dhd_watchdog_th]
547 admin 0 SW [wfd1-thrd]
927 admin 11756 S console
997 admin 0 SW [kworker/1:3]
1015 admin 11752 S /sbin/PS_pod
1076 admin 11752 S /sbin/wanduck
1083 admin 7660 S asd
1088 admin 2376 S /usr/sbin/haveged -r 0 -w 1024 -d 32 -i 32
1089 admin 13540 S nt_monitor
1090 admin 6824 S protect_srv
1091 admin 13800 S /sbin/netool
1101 admin 11460 S nt_center
1231 admin 2924 S /bin/eapd
1235 admin 11752 S wpsaide
1236 admin 3744 S nas
1238 admin 4648 S /usr/sbin/wlc_nt
1263 admin 5032 S /usr/sbin/wlceventd
1642 admin 5012 S nt_actMail
1954 admin 3120 S /usr/sbin/acsd
1956 admin 2856 S /usr/sbin/dhd_monitor
1968 admin 3324 S crond -l 9
1971 admin 5412 S vis-dcon
1975 admin 4804 S vis-datacollector
1976 admin 6956 S /usr/sbin/infosvr br0
1978 admin 3896 S sysstate
1979 admin 11756 S watchdog
1980 admin 11752 S check_watchdog
1981 admin 11752 S amas_lanctrl
1985 admin 4272 S rstats
1993 admin 3208 S lld2d br0
2022 admin 11556 S vis-dcon
2033 admin 12484 S networkmap --bootwait
2041 admin 11984 S mastiff
2042 admin 11752 S bwdpi_check
2047 admin 11756 S pctime
2088 admin 19964 S roamast
2090 admin 13808 S conn_diag
2103 admin 3316 S lldpd -L /usr/sbin/lldpcli -I eth1,eth2,eth3,eth4,eth5,eth6,wds0.*.*,wds1.*.* -s RT-AC86U
2107 nobody 3316 S lldpd -L /usr/sbin/lldpcli -I eth1,eth2,eth3,eth4,eth5,eth6,wds0.*.*,wds1.*.* -s RT-AC86U
2108 admin 0 SW [kworker/0:2]
2110 admin 12760 S cfg_server
2161 admin 14920 S amas_lib
2298 admin 0 SW [scsi_eh_0]
2299 admin 0 SW< [scsi_tmf_0]
2300 admin 0 SW [usb-storage]
2385 admin 1916 S /bin/mcpd
2470 admin 11752 S disk_monitor
2522 admin 6448 S /etc/openvpn/vpnserver1 --cd /etc/openvpn/server1 --config config.ovpn
2524 admin 6448 S /etc/openvpn/vpnserver1 --cd /etc/openvpn/server1 --config config.ovpn
2700 admin 3324 S /sbin/udhcpc -i eth0 -p /var/run/udhcpc0.pid -s /tmp/udhcpc_wan -t2 -T5 -A160 -H ****-AP -O33 -O249
2757 admin 0 SW [jbd2/sda1-8]
2758 admin 0 SW< [ext4-rsv-conver]
2770 admin 0 SW< [kworker/1:1H]
2834 admin 0 SW< [kworker/0:1H]
2849 admin 5248 S vnstatd -d --noadd --config /opt/share/dn-vnstat.d/vnstat.conf
2874 nobody 28860 S pixelserv-tls 192.168.20.2
3610 nobody 80904 S dnsmasq --log-async
3611 admin 2596 S dnsmasq --log-async
3838 nobody 3096 S avahi-daemon: running [RT-AC86U.local]
3866 admin 1856 S /usr/sbin/wsdd2 -d -w -i br0 -b sku:RT-AC86U,serial:************
3867 admin 11392 S /usr/sbin/nmbd -D -s /etc/smb.conf
3870 admin 11560 S /usr/sbin/smbd -D -s /etc/smb.conf
3883 admin 12880 S minidlna -f /etc/minidlna.conf -R -W
3917 admin 3324 S {tailtopd} /bin/sh /jffs/addons/scmerlin.d/tailtopd
3919 admin 3324 S N {tailtop} /bin/sh /jffs/addons/scmerlin.d/tailtop
28478 admin 0 SW [kworker/u4:1]
30158 admin 3680 S /usr/sbin/ntp -t -S /sbin/ntpd_synced -p pool.ntp.org
30161 admin 3324 S /sbin/syslogd -m 0 -S -O /tmp/syslog.log -s 256 -l 7
30163 admin 3324 S /sbin/klogd -c 5
30169 admin 2356 S dropbear -p 192.168.20.1:22 -j -k
30290 admin 1704 S /usr/sbin/sd-idle-2.6 -i 1800
31144 admin 10348 S httpds -s -i br0 -p 8443
31151 admin 11796 S httpd -i br0
31472 admin 2408 S miniupnpd -f /etc/upnp/config
32035 admin 6940 S vsftpd /etc/vsftpd.conf
47663 admin 2484 R dropbear -p 192.168.20.1:22 -j -k
47685 admin 3328 S -sh
57662 admin 0 SW [kworker/u4:0]
415192 admin 11752 S usbled
432190 admin 12048 S /usr/sbin/smbd -D -s /etc/smb.conf
503370 admin 3192 S sleep 5
503380 admin 3192 S N sleep 4
503386 admin 3328 R ps w

Now, which number shall I pick to run lsof?

 

[Oracle]

Sadly, I don't even understand what I'm looking at. This is so far away from my area of expertise...

grep " 32771 " /proc/net/netlink
ffffffc017083400 31  32771  00000000 0        0        0 2        0        1268

lsof | grep " 1268 "
init           1                 admin    4u  netlink                         0t0    1268 unknown protocol: 31
wdtctl       292                 admin    4u  netlink                         0t0    1268 unknown protocol: 31
hotplug2     296                 admin    4u  netlink                         0t0    1268 unknown protocol: 31
console      927                 admin    4u  netlink                         0t0    1268 unknown protocol: 31
PS_pod      1015                 admin    4u  netlink                         0t0    1268 unknown protocol: 31
wanduck     1076                 admin    4u  netlink                         0t0    1268 unknown protocol: 31
asd         1083                 admin    4u  netlink                         0t0    1268 unknown protocol: 31
haveged     1088                 admin    4u  netlink                         0t0    1268 unknown protocol: 31
nt_monito   1089                 admin    4u  netlink                         0t0    1268 unknown protocol: 31
nt_monito   1089 1099 nt_monito  admin    4u  netlink                         0t0    1268 unknown protocol: 31
nt_monito   1089 1638 nt_monito  admin    4u  netlink                         0t0    1268 unknown protocol: 31
protect_s   1090                 admin    4u  netlink                         0t0    1268 unknown protocol: 31
protect_s   1090 1097 protect_s  admin    4u  netlink                         0t0    1268 unknown protocol: 31
netool      1091                 admin    4u  netlink                         0t0    1268 unknown protocol: 31
netool      1091 1098 netool     admin    4u  netlink                         0t0    1268 unknown protocol: 31
nt_center   1101                 admin    4u  netlink                         0t0    1268 unknown protocol: 31
nt_center   1101 1105 nt_center  admin    4u  netlink                         0t0    1268 unknown protocol: 31
eapd        1231                 admin    4u  netlink                         0t0    1268 unknown protocol: 31
wpsaide     1235                 admin    4u  netlink                         0t0    1268 unknown protocol: 31
nas         1236                 admin    4u  netlink                         0t0    1268 unknown protocol: 31
wlc_nt      1238                 admin    4u  netlink                         0t0    1268 unknown protocol: 31
wlceventd   1263                 admin    4u  netlink                         0t0    1268 unknown protocol: 31
nt_actMai   1642                 admin    4u  netlink                         0t0    1268 unknown protocol: 31
nt_actMai   1642 1647 nt_actMai  admin    4u  netlink                         0t0    1268 unknown protocol: 31
acsd        1954                 admin    4u  netlink                         0t0    1268 unknown protocol: 31
dhd_monit   1956                 admin    4u  netlink                         0t0    1268 unknown protocol: 31
vis-dcon    1971                 admin    4u  netlink                         0t0    1268 unknown protocol: 31
vis-datac   1975                 admin    4u  netlink                         0t0    1268 unknown protocol: 31
infosvr     1976                 admin    4u  netlink                         0t0    1268 unknown protocol: 31
sysstate    1978                 admin    4u  netlink                         0t0    1268 unknown protocol: 31
watchdog    1979                 admin    4u  netlink                         0t0    1268 unknown protocol: 31
check_wat   1980                 admin    4u  netlink                         0t0    1268 unknown protocol: 31
amas_lanc   1981                 admin    4u  netlink                         0t0    1268 unknown protocol: 31
rstats      1985                 admin    4u  netlink                         0t0    1268 unknown protocol: 31
lld2d       1993                 admin    4u  netlink                         0t0    1268 unknown protocol: 31
vis-dcon    2022                 admin    4u  netlink                         0t0    1268 unknown protocol: 31
networkma   2033                 admin    4u  netlink                         0t0    1268 unknown protocol: 31
mastiff     2041                 admin    4u  netlink                         0t0    1268 unknown protocol: 31
mastiff     2041 2055 mastiff    admin    4u  netlink                         0t0    1268 unknown protocol: 31
mastiff     2041 2056 mastiff    admin    4u  netlink                         0t0    1268 unknown protocol: 31
bwdpi_che   2042                 admin    4u  netlink                         0t0    1268 unknown protocol: 31
pctime      2047                 admin    4u  netlink                         0t0    1268 unknown protocol: 31
roamast     2088                 admin    4u  netlink                         0t0    1268 unknown protocol: 31
roamast     2088 2679 roamast    admin    4u  netlink                         0t0    1268 unknown protocol: 31
roamast     2088 2791 roamast    admin    4u  netlink                         0t0    1268 unknown protocol: 31
roamast     2088 2792 roamast    admin    4u  netlink                         0t0    1268 unknown protocol: 31
conn_diag   2090                 admin    4u  netlink                         0t0    1268 unknown protocol: 31
conn_diag   2090 2098 conn_diag  admin    4u  netlink                         0t0    1268 unknown protocol: 31
lldpd       2103                 admin    4u  netlink                         0t0    1268 unknown protocol: 31
lldpd       2107                nobody    4u  netlink                         0t0    1268 unknown protocol: 31
cfg_serve   2110                 admin    4u  netlink                         0t0    1268 unknown protocol: 31
cfg_serve   2110 2629 cfg_serve  admin    4u  netlink                         0t0    1268 unknown protocol: 31
cfg_serve   2110 2630 cfg_serve  admin    4u  netlink                         0t0    1268 unknown protocol: 31
amas_lib    2161                 admin    4u  netlink                         0t0    1268 unknown protocol: 31
mcpd        2385                 admin    4u  netlink                         0t0    1268 unknown protocol: 31
disk_moni   2470                 admin    4u  netlink                         0t0    1268 unknown protocol: 31
vpnserver   2522                 admin    4u  netlink                         0t0    1268 unknown protocol: 31
udhcpc      2700                 admin    4u  netlink                         0t0    1268 unknown protocol: 31
pixelserv   2874                nobody    4u  netlink                         0t0    1268 unknown protocol: 31
pixelserv   2874 3104 pixelserv nobody    4u  netlink                         0t0    1268 unknown protocol: 31
wsdd2       3866                 admin    4u  netlink                         0t0    1268 unknown protocol: 31
nmbd        3867                 admin    4u  netlink                         0t0    1268 unknown protocol: 31
smbd        3870                 admin    4u  netlink                         0t0    1268 unknown protocol: 31
tailtopd    3917                 admin    4u  netlink                         0t0    1268 unknown protocol: 31
tailtop     3919                 admin    4u  netlink                         0t0    1268 unknown protocol: 31
ntp        30158                 admin    4u  netlink                         0t0    1268 unknown protocol: 31
syslogd    30161                 admin    4u  netlink                         0t0    1268 unknown protocol: 31
klogd      30163                 admin    4u  netlink                         0t0    1268 unknown protocol: 31
dropbear   30169                 admin    4u  netlink                         0t0    1268 unknown protocol: 31
sd-idle-2  30290                 admin    4u  netlink                         0t0    1268 unknown protocol: 31
httpds     31144                 admin    4u  netlink                         0t0    1268 unknown protocol: 31
httpd      31151                 admin    4u  netlink                         0t0    1268 unknown protocol: 31
miniupnpd  31472                 admin    4u  netlink                         0t0    1268 unknown protocol: 31
vsftpd     32035                 admin    4u  netlink                         0t0    1268 unknown protocol: 31
dropbear   47663                 admin    4u  netlink                         0t0    1268 unknown protocol: 31
usbled    415192                 admin    4u  netlink                         0t0    1268 unknown protocol: 31
smbd      432190                 admin    4u  netlink                         0t0    1268 unknown protocol: 31
sleep     507299                 admin    4u  netlink                         0t0    1268 unknown protocol: 31
sleep     507305                 admin    4u  netlink                         0t0    1268 unknown protocol: 31

 

[Oracle]

Some more:

#grep " 2089 " /proc/net/netlink
ffffffc01407b400 31  2089   00000000 0        0        0 2        0        3983
#lsof | grep " 3983 "
cfg_serve   2110                 admin    6u  netlink                         0t0    3983 unknown protocol: 31
cfg_serve   2110 2629 cfg_serve  admin    6u  netlink                         0t0    3983 unknown protocol: 31
cfg_serve   2110 2630 cfg_serve  admin    6u  netlink                         0t0    3983 unknown protocol: 31
# grep " 2223 " /proc/net/netlink
ffffffc0141cb800 31  2223   00000000 0        0        0 2        0        5308
# lsof | grep " 5308 "
mcpd        2385                 admin    6u  netlink                         0t0    5308 unknown protocol: 31
# grep " 2381 " /proc/net/netlink
ffffffc01417f000 31  2381   00000000 0        0        0 2        0        5526
# lsof | grep " 5526 "
pixelserv   2874                nobody    6u  netlink                         0t0    5526 unknown protocol: 31
pixelserv   2874 3104 pixelserv nobody    6u  netlink                         0t0    5526 unknown protocol: 31
# grep " 2735 " /proc/net/netlink
ffffffc014176000 31  2735   00000000 0        0        0 2        0        8223
# lsof | grep " 8223 "
pixelserv   2874                nobody    9u  netlink                         0t0    8223 unknown protocol: 31
pixelserv   2874 3104 pixelserv nobody    9u  netlink                         0t0    8223 unknown protocol: 31
# grep " 4195398 " /proc/net/netlink
ffffffc0197fb400 31  4195398 00000000 0        0        0 2        0        2535
# lsof | grep " 2535 "
eapd        1231                 admin    7u  netlink                         0t0    2535 unknown protocol: 31
# grep " 4195539 " /proc/net/netlink
ffffffc015a11c00 31  4195539 00000000 0        0        0 2        0        627
# lsof | grep " 627 "
nas         1236                 admin    7u  netlink                         0t0     627 unknown protocol: 31
# grep " 4196687 " /proc/net/netlink
ffffffc01417f400 31  4196687 00000000 0        0        0 2        0        5527
# lsof | grep " 5527 "
pixelserv   2874                nobody    7u  netlink                         0t0    5527 unknown protocol: 31
pixelserv   2874 3104 pixelserv nobody    7u  netlink                         0t0    5527 unknown protocol: 31
# grep " 4197041 " /proc/net/netlink
ffffffc014376c00 31  4197041 00000000 0        0        0 2        0        8224
# lsof | grep " 8224 "
pixelserv   2874                nobody   10u  netlink                         0t0    8224 unknown protocol: 31
pixelserv   2874 3104 pixelserv nobody   10u  netlink                         0t0    8224 unknown protocol: 31

 

[ColinTaylor]

@Oracle That all looks as expected. The only thing I hadn't seen mentioned before was pixelserv. Just for completeness you might want to check processes 1239 and 1243. I expect they'll be something like ceventd or wlceventd.

So now you just need to let it run and see how many stuck processes your script finds over time. If/when it finds a significant number check the processes again to see what's changed.

 

[Oracle]

Here are 1239 and 1243. Pixelserv came with Diversion.

# grep " 1239 " /proc/net/netlink
ffffffc015b9f800 31  1239   00000000 0        0        0 2        0        632
# lsof | grep " 632 "
wlceventd   1263                 admin    6u  netlink                         0t0     632 unknown protocol: 31
# grep " 1243 " /proc/net/netlink
ffffffc015b9f000 31  1243   00000000 0        0        0 2        0        634
# lsof | grep " 634 "
acsd        1954                 admin    6u  netlink                         0t0     634 unknown protocol: 31

@ColinTaylor, do you mind revealing what this means?
What are these "not found" processes and why are there so many instances of them?
What does the lsof output mean - 4u, 6u, 9u, unknown protocol: 31?

If all of this is normal, what do you expect to see as unusual in case of an error? Maybe I should disable the override scripts because they kill stuck commands and we can't see them.

 

[ColinTaylor]

Thanks for the extra info.

@ColinTaylor, do you mind revealing what this means?
What are these "not found" processes and why are there so many instances of them?
What does the lsof output mean - 4u, 6u, 9u, unknown protocol: 31?

For the background on this read the posts in this thread from @dave14305 and myself. At this time it's just information gathering to confirm our earlier hypothesis. In short, certain daemon processes (e.g. wlceventd) hold open netlink sockets with a nl_pid that is different to their pid. There's absolutely nothing wrong with them doing this. However there's a bug in the RT-AC86U's nvram module whereby it can't handle this situation, leading to hung nvram calls.

If all of this is normal, what do you expect to see as unusual in case of an error?

Even after you changed pid_max on your router you were still seeing a steady increase in the number of stuck wl commands but not nvram commands. I don't know why that is so again, it's just information gathering to try and see what's happening on your router.

Maybe I should disable the override scripts because they kill stuck commands and we can't see them.

Maybe. We'd still need some way of being aware of when processes have become stuck so that we can repeat the earlier script/commands.

 

[ttgapers]

Started following this thread and thanks @dave14305 and co. for bringing it to the top of the pile. I have 3x RT-AC86Us that I highly suspect are exhibiting these symptoms. I'll let you folks know what I find as hopefully it helps with the info gathering and analysis.

Results on the first (local router):

Print netlink sockets for which there is no process with the same number

Process 2448 not found
Process associated with 35218 not found (2448)
Process associated with 4294962358 not found (4294929588)
Process associated with 4294962470 not found (4294929700)
Process associated with 4294962580 not found (4294929810)
Process associated with 4294962639 not found (4294929869)
Process associated with 4294962712 not found (4294929942)
Process associated with 4294962897 not found (4294930127)
Process associated with 4294963031 not found (4294930261)
Process associated with 4294963074 not found (4294930304)

A number in brackets is a *guess* at an associated process

QQ: Should I be doing lookups on all the 429* processes above as well? If so, some more to run.....

Next

grep " 2448 " /proc/net/netlink
ffffffc013c28000 31  2448   00000000 0        0        0 2        0        5130

grep " 2448 " /proc/net/netlink
ffffffc013c28000 31  2448   00000000 0        0        0 2        0        5130

lsof | grep " 5130 "
cfg_serve  2467                  admin    6u  netlink                         0t0       5130 unknown protocol: 31
cfg_serve  2467  2491 cfg_serve  admin    6u  netlink                         0t0       5130 unknown protocol: 31
cfg_serve  2467  2492 cfg_serve  admin    6u  netlink                         0t0       5130 unknown protocol: 31

lsof | grep " 5131 "
cfg_serve  2467                  admin    7u  netlink                         0t0       5131 unknown protocol: 31
cfg_serve  2467  2491 cfg_serve  admin    7u  netlink                         0t0       5131 unknown protocol: 31
cfg_serve  2467  2492 cfg_serve  admin    7u  netlink                         0t0       5131 unknown protocol: 31

cfg_serve is my nemesis

 

[ColinTaylor]

Print netlink sockets for which there is no process with the same number

Process 2448 not found
Process associated with 35218 not found (2448)
Process associated with 4294962358 not found (4294929588)
Process associated with 4294962470 not found (4294929700)
Process associated with 4294962580 not found (4294929810)
Process associated with 4294962639 not found (4294929869)
Process associated with 4294962712 not found (4294929942)
Process associated with 4294962897 not found (4294930127)
Process associated with 4294963031 not found (4294930261)
Process associated with 4294963074 not found (4294930304)

A number in brackets is a *guess* at an associated process

QQ: Should I be doing lookups on all the 429* processes above as well? If so, some more to run.....

I have no idea why those process numbers are showing up. They look like a signed negative 32bit numbers, so 4294962358 would be -4938. There's probably something in the data that my script can't handle. If you're interested you could run this command to see if anything looks wrong with the format.

cat /proc/net/netlink | sort -nk3 | grep " 31 "

 

[ttgapers]

cat /proc/net/netlink | sort -nk3 | grep " 31 "

Here you go...

cat /proc/net/netlink | sort -nk3 | grep " 31 "
ffffffc0191d7800 31  0      00000000 0        0        0 2        0        134
ffffffc017a9a800 31  1      00000000 0        0        0 2        0        188
ffffffc014ec3c00 31  960    00000000 0        0        0 2        0        2136
ffffffc017267c00 31  1062   00000000 0        0        0 2        0        1744
ffffffc017707800 31  1208   00000000 0        0        0 2        0        1971
ffffffc0177dc800 31  1273   00000000 0        0        0 2        0        3114
ffffffc014de6000 31  1280   00000000 0        0        0 2        0        36361290
ffffffc0177dcc00 31  1281   00000000 0        0        0 2        0        2908
ffffffc017729c00 31  1282   00000000 0        0        0 2        0        3133
ffffffc01904c800 31  1308   00000000 0        0        0 2        0        2973
ffffffc017728000 31  1309   00000000 0        0        0 2        0        3630
ffffffc014e37c00 31  1310   00000000 0        0        0 2        0        2970
ffffffc017729000 31  1311   00000000 0        0        0 2        0        3633
ffffffc014e39c00 31  1312   00000000 0        0        0 2        0        2976
ffffffc01770b800 31  1321   00000000 0        0        0 2        0        3678
ffffffc01770b400 31  1898   00000000 0        0        0 2        0        3015
ffffffc01775f400 31  1899   00000000 0        0        0 2        0        3013
ffffffc014e39400 31  1908   00000000 0        0        0 2        0        3018
ffffffc014e0f800 31  1915   00000000 0        0        0 2        0        3913
ffffffc014fe8400 31  1916   00000000 0        0        0 2        0        3945
ffffffc014fad400 31  1918   00000000 0        0        0 2        0        3067
ffffffc01775f000 31  1919   00000000 0        0        0 2        0        3042
ffffffc014d18000 31  1921   00000000 0        0        0 2        0        3942
ffffffc014f20c00 31  1923   00000000 0        0        0 2        0        3064
ffffffc014c85400 31  1926   00000000 0        0        0 2        0        3950
ffffffc014e41000 31  1954   00000000 0        0        0 2        0        6784
ffffffc01e832000 31  2008   00000000 0        0        0 2        0        4033
ffffffc0197f8400 31  2014   00000000 0        0        0 2        0        4066
ffffffc0177a7800 31  2027   00000000 0        0        0 2        0        4075
ffffffc014de6800 31  2447   00000000 0        0        0 2        0        5123
ffffffc013c28000 31  2448   00000000 0        0        0 2        0        5130
ffffffc014d03000 31  2450   00000000 0        0        0 2        0        4321
ffffffc014d3f800 31  2511   00000000 0        0        0 2        0        5603
ffffffc01720a800 31  2657   00000000 0        0        0 2        0        6405
ffffffc014c62800 31  2938   00000000 0        0        0 2        0        6045
ffffffc0170f9800 31  3448   00000000 0        0        0 2        0        7555
ffffffc013c60000 31  3661   00000000 0        0        0 2        0        8430
ffffffc013f67800 31  4970   00000000 0        0        0 2        0        35274699
ffffffc017085c00 31  32771  00000000 0        0        0 2        0        189
ffffffc014ec3400 31  33730  00000000 0        0        0 2        0        2137
ffffffc017267800 31  33832  00000000 0        0        0 2        0        1745
ffffffc014e41c00 31  33978  00000000 0        0        0 2        0        1972
ffffffc014e7b400 31  34043  00000000 0        0        0 2        0        3115
ffffffc017749c00 31  34050  00000000 0        0        0 2        0        36361291
ffffffc0177a7000 31  34051  00000000 0        0        0 2        0        2909
ffffffc017729800 31  34052  00000000 0        0        0 2        0        3134
ffffffc014e35000 31  34078  00000000 0        0        0 2        0        2974
ffffffc017729400 31  34079  00000000 0        0        0 2        0        3631
ffffffc01720a000 31  34080  00000000 0        0        0 2        0        2971
ffffffc01770ac00 31  34081  00000000 0        0        0 2        0        3634
ffffffc014e39800 31  34082  00000000 0        0        0 2        0        2977
ffffffc01770bc00 31  34091  00000000 0        0        0 2        0        3679
ffffffc0177a7400 31  34668  00000000 0        0        0 2        0        3016
ffffffc01770b000 31  34669  00000000 0        0        0 2        0        3014
ffffffc014e7b000 31  34678  00000000 0        0        0 2        0        3019
ffffffc014d18400 31  34685  00000000 0        0        0 2        0        3914
ffffffc014fe8000 31  34686  00000000 0        0        0 2        0        3946
ffffffc014fad800 31  34688  00000000 0        0        0 2        0        3068
ffffffc014e39000 31  34689  00000000 0        0        0 2        0        3043
ffffffc014fe8800 31  34691  00000000 0        0        0 2        0        3943
ffffffc014f20800 31  34693  00000000 0        0        0 2        0        3065
ffffffc014c85000 31  34696  00000000 0        0        0 2        0        3951
ffffffc014eecc00 31  34724  00000000 0        0        0 2        0        6785
ffffffc0172d2400 31  34778  00000000 0        0        0 2        0        4034
ffffffc017707c00 31  34784  00000000 0        0        0 2        0        4067
ffffffc0172d2000 31  34797  00000000 0        0        0 2        0        4076
ffffffc01771f400 31  35217  00000000 0        0        0 2        0        5124
ffffffc014d04c00 31  35218  00000000 0        0        0 2        0        5131
ffffffc014e2c800 31  35220  00000000 0        0        0 2        0        4322
ffffffc014efec00 31  35281  00000000 0        0        0 2        0        5604
ffffffc014d3f000 31  35427  00000000 0        0        0 2        0        6406
ffffffc0197f8000 31  35708  00000000 0        0        0 2        0        6046
ffffffc0197fb000 31  36218  00000000 0        0        0 2        0        7556
ffffffc013e2f800 31  36431  00000000 0        0        0 2        0        8431
ffffffc014d3fc00 31  37740  00000000 0        0        0 2        0        35274701
ffffffc0172c9800 31  4294962358 00000000 0        0        0 2        0        17131625
ffffffc013e2fc00 31  4294962470 00000000 0        0        0 2        0        13746013
ffffffc013d81c00 31  4294962580 00000000 0        0        0 2        0        12630080
ffffffc013ca9800 31  4294962639 00000000 0        0        0 2        0        11679097
ffffffc013d9c800 31  4294962712 00000000 0        0        0 2        0        9463678
ffffffc013e27000 31  4294962897 00000000 0        0        0 2        0        4830603
ffffffc014c62000 31  4294963031 00000000 0        0        0 2        0        1740956
ffffffc014e0f400 31  4294963074 00000000 0        0        0 2        0        513880

Hope it helps. I will have a look at the 2x remote devices later tonight.

 

[ColinTaylor]

Here you go...

cat /proc/net/netlink | sort -nk3 | grep " 31 "
ffffffc0191d7800 31  0      00000000 0        0        0 2        0        134
ffffffc017a9a800 31  1      00000000 0        0        0 2        0        188
ffffffc014ec3c00 31  960    00000000 0        0        0 2        0        2136
ffffffc017267c00 31  1062   00000000 0        0        0 2        0        1744
ffffffc017707800 31  1208   00000000 0        0        0 2        0        1971
ffffffc0177dc800 31  1273   00000000 0        0        0 2        0        3114
ffffffc014de6000 31  1280   00000000 0        0        0 2        0        36361290
ffffffc0177dcc00 31  1281   00000000 0        0        0 2        0        2908
ffffffc017729c00 31  1282   00000000 0        0        0 2        0        3133
ffffffc01904c800 31  1308   00000000 0        0        0 2        0        2973
ffffffc017728000 31  1309   00000000 0        0        0 2        0        3630
ffffffc014e37c00 31  1310   00000000 0        0        0 2        0        2970
ffffffc017729000 31  1311   00000000 0        0        0 2        0        3633
ffffffc014e39c00 31  1312   00000000 0        0        0 2        0        2976
ffffffc01770b800 31  1321   00000000 0        0        0 2        0        3678
ffffffc01770b400 31  1898   00000000 0        0        0 2        0        3015
ffffffc01775f400 31  1899   00000000 0        0        0 2        0        3013
ffffffc014e39400 31  1908   00000000 0        0        0 2        0        3018
ffffffc014e0f800 31  1915   00000000 0        0        0 2        0        3913
ffffffc014fe8400 31  1916   00000000 0        0        0 2        0        3945
ffffffc014fad400 31  1918   00000000 0        0        0 2        0        3067
ffffffc01775f000 31  1919   00000000 0        0        0 2        0        3042
ffffffc014d18000 31  1921   00000000 0        0        0 2        0        3942
ffffffc014f20c00 31  1923   00000000 0        0        0 2        0        3064
ffffffc014c85400 31  1926   00000000 0        0        0 2        0        3950
ffffffc014e41000 31  1954   00000000 0        0        0 2        0        6784
ffffffc01e832000 31  2008   00000000 0        0        0 2        0        4033
ffffffc0197f8400 31  2014   00000000 0        0        0 2        0        4066
ffffffc0177a7800 31  2027   00000000 0        0        0 2        0        4075
ffffffc014de6800 31  2447   00000000 0        0        0 2        0        5123
ffffffc013c28000 31  2448   00000000 0        0        0 2        0        5130
ffffffc014d03000 31  2450   00000000 0        0        0 2        0        4321
ffffffc014d3f800 31  2511   00000000 0        0        0 2        0        5603
ffffffc01720a800 31  2657   00000000 0        0        0 2        0        6405
ffffffc014c62800 31  2938   00000000 0        0        0 2        0        6045
ffffffc0170f9800 31  3448   00000000 0        0        0 2        0        7555
ffffffc013c60000 31  3661   00000000 0        0        0 2        0        8430
ffffffc013f67800 31  4970   00000000 0        0        0 2        0        35274699
ffffffc017085c00 31  32771  00000000 0        0        0 2        0        189
ffffffc014ec3400 31  33730  00000000 0        0        0 2        0        2137
ffffffc017267800 31  33832  00000000 0        0        0 2        0        1745
ffffffc014e41c00 31  33978  00000000 0        0        0 2        0        1972
ffffffc014e7b400 31  34043  00000000 0        0        0 2        0        3115
ffffffc017749c00 31  34050  00000000 0        0        0 2        0        36361291
ffffffc0177a7000 31  34051  00000000 0        0        0 2        0        2909
ffffffc017729800 31  34052  00000000 0        0        0 2        0        3134
ffffffc014e35000 31  34078  00000000 0        0        0 2        0        2974
ffffffc017729400 31  34079  00000000 0        0        0 2        0        3631
ffffffc01720a000 31  34080  00000000 0        0        0 2        0        2971
ffffffc01770ac00 31  34081  00000000 0        0        0 2        0        3634
ffffffc014e39800 31  34082  00000000 0        0        0 2        0        2977
ffffffc01770bc00 31  34091  00000000 0        0        0 2        0        3679
ffffffc0177a7400 31  34668  00000000 0        0        0 2        0        3016
ffffffc01770b000 31  34669  00000000 0        0        0 2        0        3014
ffffffc014e7b000 31  34678  00000000 0        0        0 2        0        3019
ffffffc014d18400 31  34685  00000000 0        0        0 2        0        3914
ffffffc014fe8000 31  34686  00000000 0        0        0 2        0        3946
ffffffc014fad800 31  34688  00000000 0        0        0 2        0        3068
ffffffc014e39000 31  34689  00000000 0        0        0 2        0        3043
ffffffc014fe8800 31  34691  00000000 0        0        0 2        0        3943
ffffffc014f20800 31  34693  00000000 0        0        0 2        0        3065
ffffffc014c85000 31  34696  00000000 0        0        0 2        0        3951
ffffffc014eecc00 31  34724  00000000 0        0        0 2        0        6785
ffffffc0172d2400 31  34778  00000000 0        0        0 2        0        4034
ffffffc017707c00 31  34784  00000000 0        0        0 2        0        4067
ffffffc0172d2000 31  34797  00000000 0        0        0 2        0        4076
ffffffc01771f400 31  35217  00000000 0        0        0 2        0        5124
ffffffc014d04c00 31  35218  00000000 0        0        0 2        0        5131
ffffffc014e2c800 31  35220  00000000 0        0        0 2        0        4322
ffffffc014efec00 31  35281  00000000 0        0        0 2        0        5604
ffffffc014d3f000 31  35427  00000000 0        0        0 2        0        6406
ffffffc0197f8000 31  35708  00000000 0        0        0 2        0        6046
ffffffc0197fb000 31  36218  00000000 0        0        0 2        0        7556
ffffffc013e2f800 31  36431  00000000 0        0        0 2        0        8431
ffffffc014d3fc00 31  37740  00000000 0        0        0 2        0        35274701
ffffffc0172c9800 31  4294962358 00000000 0        0        0 2        0        17131625
ffffffc013e2fc00 31  4294962470 00000000 0        0        0 2        0        13746013
ffffffc013d81c00 31  4294962580 00000000 0        0        0 2        0        12630080
ffffffc013ca9800 31  4294962639 00000000 0        0        0 2        0        11679097
ffffffc013d9c800 31  4294962712 00000000 0        0        0 2        0        9463678
ffffffc013e27000 31  4294962897 00000000 0        0        0 2        0        4830603
ffffffc014c62000 31  4294963031 00000000 0        0        0 2        0        1740956
ffffffc014e0f400 31  4294963074 00000000 0        0        0 2        0        513880

Hope it helps. I will have a look at the 2x remote devices later tonight.

OK that all looks fine. So it's just something I've not seen before. You could try tracing one or two of the inodes to see if they're interesting.

lsof | grep " 513880 "
lsof | grep " 9463678 "
lsof | grep " 17131625 "

 

[Oracle]

Asus is making me learn totally unexpected things. When I bought this router I had no idea what I was getting into.

I'm still unclear how to report the problem to them. How do I make them pay attention to something that messes up 3rd party firmware and extensions?

For someone that has the netlink source code it will be relatively easy to put this in debug mode and see where the error comes from. For us it's like reverse engineering a black box.

 

[eibgrad]

Personally, given the complexity and uncertainty, I'd be more likely to dump the router and move on, esp. given all its other issues as well. Just sayin'. But I admire the persistence! True hackers here.

 

[ColinTaylor]

I'm still unclear how to report the problem to them. How do I make them pay attention to something that messes up 3rd party firmware and extensions?

You don't. Asus doesn't (and can't) support third party software that they have no control over. If you can recreate the problem with stock firmware you can use the Feedback option to report it.

 

[Oracle]

We can't run custom scripts on the stock firmware, can we?
For example the test loop that executes thousands of nvram get commands.