XT8 troubleshooting log, cfg_server possibly an issue

[arrgh]

Just under three days of uptime, cfg_server is behaving itself.

 

[bagofchips]

arrgh, thank you so much for looking into this! I had just posted a thread on this issue I've been facing and when I saw cfg_server in your thread's subject, I wasn't disappointed. Learned quite a bit! ASUS just had their work cut out for them.

 

[arrgh]

dammit, it just started happening again!

admin@router:/tmp/home/root# netstat -tnap |grep cfg_server |wc -l
1006
admin@router:/tmp/home/root# ps wT |grep cfg_server |wc -l
510

 

[tabormeister]

I got a beta firmware from Asus engineers for xt8s while troubleshooting something different, try this

https://www.asuswebstorage.com/navigate/a/#/s/CB7C3DDA15B740AD9F3D08A398A2A836Y

 

[tabormeister]

Lol nope mine just started doing it again. Rolling back to 386 again.

 

[tabormeister]

OK so interestingly I'm running one of the XT8 nodes on that beta firmware above, and one on 386 firmware and am not getting the cfg_server memleak anymore. Maybe try mismatching firmware versions to see if it does that nonsense?

 

[tabormeister]

Going back and forth with asus, not promising

 

[tabormeister]

So Asus took my XT8's back to review and see if there's an issue, I have a new set now (still hw 1.0) and am going to try hooking one up as a node. Updates to follow!

 

[arrgh]

I've been on 3.0.0.4.388_23012-g1166fba for a month or so and it's been good, ...

Recent Asus beta firmware release notes mentioned a "cfg server vulnerability" which probably refers to https://talosintelligence.com/vulnerability_reports/TALOS-2022-1592. I'm pretty confident my problem wasn't due to this vuln being exploited, but any attention this codebase gets is probably net positive. :/

 

[arrgh]

WELP

23285 just started misbehaving again... 500+ threads, 800 connections in CLOSE_WAIT, cfg_server using lots of RAM

admin@router:/tmp/home/root# ps wT |grep cfg |wc -l
510
admin@router:/tmp/home/root# netstat -tnap |grep cfg |wc -l
806
admin@router:/tmp/home/root# netstat -tnap |grep cfg |grep -v CLOSE_WAIT |wc -l
1
admin@router:/tmp/home/root# ps w |grep cfg_server |grep -v grep
32087 admin     148m R    cfg_server

There are some segfaults in dmesg, including one for cfg_server:

potentially unexpected fatal signal 11.
CPU: 2 PID: 31775 Comm: cfg_server Tainted: P           O    4.1.52 #2
Hardware name: Generic DT based system
task: ccfb0c00 ti: c852a000 task.ti: c852a000
PC is at 0xb692855c
LR is at 0xb6ef4e9c
pc : [<b692855c>]    lr : [<b6ef4e9c>]    psr: 60010010
sp : acffe8c8  ip : b6f4c7cc  fp : acffef9c
r10: 000c5184  r9 : 000a0d93  r8 : b1a3ec82
r7 : 00000018  r6 : acfff920  r5 : 00000004  r4 : 00000000
r3 : 00000010  r2 : 00000001  r1 : 00000000  r0 : 00000000
Flags: nZCv  IRQs on  FIQs on  Mode USER_32  ISA ARM  Segment user
Control: 10c5387d  Table: 11a4404a  DAC: 00000015
CPU: 2 PID: 31775 Comm: cfg_server Tainted: P           O    4.1.52 #2
Hardware name: Generic DT based system
[<c00270a0>] (unwind_backtrace) from [<c0022cf8>] (show_stack+0x10/0x14)
[<c0022cf8>] (show_stack) from [<c04c7bfc>] (dump_stack+0x8c/0xa0)
[<c04c7bfc>] (dump_stack) from [<c003ad2c>] (get_signal+0x490/0x558)
[<c003ad2c>] (get_signal) from [<c0022290>] (do_signal+0xc8/0x3ac)
[<c0022290>] (do_signal) from [<c0022718>] (do_work_pending+0x94/0xa4)
[<c0022718>] (do_work_pending) from [<c001f58c>] (work_pending+0xc/0x20)

 

[arrgh]

I didn't restart or kill anything since my last post, and it seems to have recovered itself. Uptime 18 days. WELP^2.

I would guess cfg_server got restarted by something, but its proc/*/status indicates the current instance was once large:

admin@router:/tmp# grep VmPeak /proc/$(pidof cfg_server)/status
VmPeak:   152224 kB

 

[arrgh]

Still on 23285 with wired backhaul, uptime 73 days, no noticeable UI slowdowns or cfg_server shenanigans. I'm calling this one good.

 

[glens]

I loaded up (I believe) that firmware and poked around in it. Preferred the Merlin environment so reverted. Uptime on the router has got to be 60 days by now, so it's been at least that long ago.

Impertinent, but was notified of new thread content. Refreshing memory, saw that above. For reference that was gnuton's Merlin, the initial version (not 388). A fairly-regional power loss overcame my UPS at only a handful of days shy a year uptime.

The only issue that ever caught my attention was that local names quit being resolved once. And I poke around probably more than most - can be a "knob twiddler" at times. Evidently any running-configuration changes were benign-enough.

 

[arrgh]

In the nearly-a-year since my last post here, I've upgraded a lot of my setup; I now have an AXE16000 as the main router, and have relegated two XT8s to satellite duty. It's been stable.