Solved RT-AX86U Reboot Loops In Merlin 386.7_2

[breadlord]

Hey all - I'm new here and I've been trying to read up all I can on reasons for my 86's constant reboot after moving from stock to Merlin firmware. I went through all of the normal shanenaigans - multiple wipes, reflashes, building config from fresh, etc. No dice.

I set my output to debug and found this at the end of a crashlog:

May 5 06:05:08 crashlog: <2>CPU2: stopping
May 5 06:05:08 crashlog: <4>CPU: 2 PID: 0 Comm: swapper/2 Tainted: P O L 4.1.52 #2
May 5 06:05:08 crashlog: <4>Hardware name: Broadcom-v8A (DT)
May 5 06:05:08 crashlog: <0>Call trace:
May 5 06:05:08 crashlog: <4>[<ffffffc000087398>] dump_backtrace+0x0/0x150
May 5 06:05:08 crashlog: <4>[<ffffffc0000874fc>] show_stack+0x14/0x20
May 5 06:05:08 crashlog: <4>[<ffffffc0005589d0>] dump_stack+0x90/0xb0
May 5 06:05:08 crashlog: <4>[<ffffffc00008d710>] handle_IPI+0x190/0x1a0
May 5 06:05:08 crashlog: <4>[<ffffffc000080c68>] gic_handle_irq+0x88/0x90
May 5 06:05:08 crashlog: <4>Exception stack(0xffffffc03e8cfdc0 to 0xffffffc03e8cfef0)
May 5 06:05:08 crashlog: <4>fdc0: b2bab4bc 000000c5 00000000 00000080 3e8cff10 ffffffc0 00381594 ffffffc0
May 5 06:05:08 crashlog: <4>fde0: b2bab4bc 000000c5 70134b50 004842b8 0306b051 00000000 1400007e 00000000
May 5 06:05:08 crashlog: <4>fe00: 0017650a 00000000 00000018 00000000 9ec88372 000bbcf0 c93ff75c 000000c5
May 5 06:05:08 crashlog: <4>fe20: 3e8c3a70 ffffffc0 3e8cfec0 ffffffc0 3e801248 ffffffc0 2883ff80 ffffffc0
May 5 06:05:08 crashlog: <4>fe40: 00000018 00000000 ff8dd940 00000000 f7483400 00000000 00000000 00000000
May 5 06:05:08 crashlog: <4>fe60: 003e31e8 ffffffc0 00000000 00000000 00000000 00000000 b2bab4bc 000000c5
May 5 06:05:08 crashlog: <4>fe80: 3ffd4338 ffffffc0 00000001 00000000 00000001 00000000 b2b8a5a0 000000c5
May 5 06:05:08 crashlog: <4>fea0: 3e8cc000 ffffffc0 00919000 ffffffc0 00746000 ffffffc0 3ffd4338 ffffffc0
May 5 06:05:08 crashlog: <4>fec0: 00789300 ffffffc0 3e8cff10 ffffffc0 0038158c ffffffc0 3e8cff10 ffffffc0
May 5 06:05:08 crashlog: <4>fee0: 00381594 ffffffc0 60000145 00000000
May 5 06:05:08 crashlog: <4>[<ffffffc000083f00>] el1_irq+0x80/0xf8
May 5 06:05:08 crashlog: <4>[<ffffffc000381698>] cpuidle_enter+0x18/0x20
May 5 06:05:08 crashlog: <4>[<ffffffc0000c5e5c>] cpu_startup_entry+0x1ec/0x250
May 5 06:05:08 crashlog: <4>[<ffffffc00008d190>] secondary_start_kernel+0x150/0x178
May 5 06:05:08 crashlog: <4>SMP: failed to stop secondary CPUs
May 5 06:05:08 crashlog:
May 5 06:05:08 crashlog:
May 5 06:05:08 crashlog:
May 5 06:05:08 crashlog:
May 5 06:05:08 crashlog:
May 5 06:05:08 kernel: ^[[0;33;41m[ERROR pktrunner] runnerUcast_inet6addr_event,148: Could not rdpa_system_ipv6_host_address_table_find ret=-5^[[0m

I suspect that what was happening is that a newer host (Probably my work mac, running OSX Monterey, and my Windows 10 PC, doing all kinds of crazy network stuff) are sending out IPv6 multicast packets - as a guess they might be mDNS over IPv6. Looks like the router has some kernel code to handle them, but it's not being processed, so is causing a lockup, which is health-checked into a reboot.

I've turned ipv6 passthrough on, and it has solved the bootloop issue - or at least the specific boot loop I was having.

 

[ColinTaylor]

FYI The final pktrunner message is not part of the crashlog. If you look at the complete crashlog there might be clues there (but usually not).

 

[breadlord]

FYI The final pktrunner message is not part of the crashlog. If you look at the complete crashlog there might be clues there (but usually not).

No, it isn't. But it is the last message in the log before the next boot process starts logging, and it does come from the kernel referring to an error processing IPv6 multicast, when IPv6 was disabled.

I don't know how crashlogs are implemented. If the kernel message is the bottom of the output stack during an exception condition then printing the error log before the log line that causes it makes some degree of sense.

I've not looked at the way the kernel is set up, so I can't confirm it - but if IPv6 is handled by a dynamically loaded kernel module that is removed when you disable it then this tracks for me.

 

[ColinTaylor]

I don't know how crashlogs are implemented. If the kernel message is the bottom of the output stack during an exception condition then printing the error log before the log line that causes it makes some degree of sense.

It's the standard Linux crash dump mechanism. So any lines you see in the syslog that start with crashlog: are being retrieved from the dump file and were create prior to the router rebooting. So the apparent timestamp for these messages is meaningless. It's slightly confusing because these old messages are being mixed in with the current syslog messages. You just have to be aware that the crashlog: messages are unrelated to the other messages around them in the log.

You usually (but not always) need to look at the first set of crashdump messages for a given event. The following crashdump messages are usually just the kernel terminating the processes running on the other cores.


But regardless of all that you appear to have identified the source of the problem.

 

[breadlord]

Honestly, the crash dump is useless. I only used it as a marker for "something bad happened" - the last log before the restart being an error from the kernel was the smoking gun.

But yeah, seems to have solved it. Only annoyance is having to fully disable ipv6 on my k3s nodes. For some reason they're reporting their Mac as the client id and that's being reported in the clients list as a hostname, so I'm unable to route via a FQDN.

Ah well, fun and games.

 

[clvk07]

Have the same issue. Since upgrading to 386.7.2 on ax86u it randomly rebooted 3 times. Might try to downgrade and see.

 

[IcecreamVille]

Have the same issue. Since upgrading to 386.7.2 on ax86u it randomly rebooted 3 times. Might try to downgrade and see.

Had a similar issue (but adding IPv6 in Passthru didn't fix it for me).

What appears to have fixed it (the jury is still out though) was downgrading to 386.5_2

Good luck

 

[clvk07]

Had a similar issue (but adding IPv6 in Passthru didn't fix it for me).

What appears to have fixed it (the jury is still out though) was downgrading to 386.5_2

Good luck

Was going to downgrade to 386.7.0, so that didn't work for you either? Saw 3-4 other threads where people having same issue with 7.2, so maybe something is not right.

 

[IcecreamVille]

Was going to downgrade to 386.7.0, so that didn't work for you either? Saw 3-4 other threads where people having same issue with 7.2, so maybe something is not right.

Because _2 is a patch (assuming naming logic is <Major Version>.<Minor Version>_<Patch>), I didn't try 386.7 - I would be curious if 386.7 resolves your issue.

386.5_2 has been rock-solid for me for over 3 days, so at this point, I don't want to touch it...

 

[clvk07]

Because _2 is a patch (assuming naming logic is <Major Version>.<Minor Version>_<Patch>), I didn't try 386.7 - I would be curious if 386.7 resolves your issue.

386.5_2 has been rock-solid for me for over 3 days, so at this point, I don't want to touch it...

Think 386.7_2 has

- UPDATED: RT-AX86U driver + SDK updated to latest upstream version

Which is not on 7.0

 

[DAOWAce]

Just had this same thing happen to me on my GT-AX6000. Huge huge crash log, but that same ipv6 thing was at the end.

I don't use IPv6, nor does my ISP support it.

Another thread about it: https://www.snbforums.com/threads/rt-ax86u-random-reboots.80646/

Weird 'cause I don't really recall this happening before. The log only has 3600 lines, and goes back to September 28, so there's limited info to cross verify.

Enabled the passthrough option regardless, hopefully that workaround works for me as well, but this is definitely looking to be a firmware issue.