Sporadic GT-AXE11000 reboots due to kernel panic

[strunker]

I have the same issue on my RT-AX86S running Merlin 386.7. I setup Lan-Secure Syslog Center for my Mac and got a similar kernel panic response:

2022-07-01 06:59:03    User.Error    RT-AX86S-C32D515-C kernel: bcm63xx_nand ff801800.nand: timeout waiting for command 0x1
    2022-07-01 06:59:03    User.Error    RT-AX86S-C32D515-C kernel: bcm63xx_nand ff801800.nand: intfc status 700000e0
    2022-07-01 06:59:03    User.Warning    RT-AX86S-C32D515-C kernel: BUG: failure at drivers/mtd/nand/brcmnand/brcmnand.c:1339/brcmnand_send_cmd()!
    2022-07-01 06:59:03    User.Emergency    RT-AX86S-C32D515-C kernel: Kernel panic - not syncing: BUG!
    2022-07-01 06:59:03    User.Warning    RT-AX86S-C32D515-C kernel: CPU: 1 PID: 1398 Comm: WebHistory Tainted: P           O    4.1.52 #2
    2022-07-01 06:59:03    User.Warning    RT-AX86S-C32D515-C kernel: Hardware name: Broadcom-v8A (DT)

This happens randomly every 2-3 days

Yup sure looks like the same problem. Do you get router lockup\reboots when this takes place?

I reached out to Linux developer, the person who actually developed this specific driver and he stated that the linux version they are using on these routers, and presumably this driver version, is old.

This is interesting though to see this impacting other models. @RMerlin may have interest in examining this outside of the research done on the 11000

 

[vlord]

Yup sure looks like the same problem. Do you get router lockup\reboots when this takes place?

I reached out to Linux developer, the person who actually developed this specific driver and he stated that the linux version they are using on these routers, and presumably this driver version, is old.

This is interesting though to see this impacting other models. @RMerlin may have interest in examining this outside of the research done on the 11000

Yes. The next set of messages in the log file is CPU 0 and CPU 1 panicking and then the router hangs and reboots. The whole process can take upwards of 7 minutes. I love the router but can’t have these reboots so I’ve ordered a AX86U hoping the issue will go away. As you said, hopefully we can get some attention to this issue and get the firmware sorted with the updated driver.

 

[strunker]

Yes. The next set of messages in the log file is CPU 0 and CPU 1 panicking and then the router hangs and reboots. The whole process can take upwards of 7 minutes. I love the router but can’t have these reboots so I’ve ordered a AX86U hoping the issue will go away. As you said, hopefully we can get some attention to this issue and get the firmware sorted with the updated driver.

I hear the ax86u is non problematic. But at this point I have a feeling a lot of the issue here boils down to just luck\chance. Some people with the same models never see this problem, others run into it very frequently.

In any event, good luck to you, I would definitely ticket it with them and see if they will buy it back or exchange towards the alternative model.

 

[Forsaken]

I have the same issue on my RT-AX86S running Merlin 386.7. I setup Lan-Secure Syslog Center for my Mac and got a similar kernel panic response:

2022-07-01 06:59:03    User.Error    RT-AX86S-C32D515-C kernel: bcm63xx_nand ff801800.nand: timeout waiting for command 0x1
    2022-07-01 06:59:03    User.Error    RT-AX86S-C32D515-C kernel: bcm63xx_nand ff801800.nand: intfc status 700000e0
    2022-07-01 06:59:03    User.Warning    RT-AX86S-C32D515-C kernel: BUG: failure at drivers/mtd/nand/brcmnand/brcmnand.c:1339/brcmnand_send_cmd()!
    2022-07-01 06:59:03    User.Emergency    RT-AX86S-C32D515-C kernel: Kernel panic - not syncing: BUG!
    2022-07-01 06:59:03    User.Warning    RT-AX86S-C32D515-C kernel: CPU: 1 PID: 1398 Comm: WebHistory Tainted: P           O    4.1.52 #2
    2022-07-01 06:59:03    User.Warning    RT-AX86S-C32D515-C kernel: Hardware name: Broadcom-v8A (DT)

This happens randomly every 2-3 days

Really strange things are happening here. For me it is opposite: panics have completely gone after I switched to Merlin. The only difference - I'm still on 386.5. Planning to switch to .7 shortly. Will keep an eye on this after update.

 

[strunker]

Really strange things are happening here. For me it is opposite: panics have completely gone after I switched to Merlin. The only difference - I'm still on 386.5. Planning to switch to .7 shortly. Will keep an eye on this after update.

I think Merlin and Asus kinda worked in tandem on the fix that is likely why you dont run into issue using that firmware. Only really Merlin could comment on that but he was involved earlier in the thread.

 

[RMerlin]

I think Merlin and Asus kinda worked in tandem on the fix that is likely why you dont run into issue using that firmware. Only really Merlin could comment on that but he was involved earlier in the thread.

I didn't. I just pointed Asus at the issue thread, I believe they sent test firmwares to them, and that's the last I heard of it. The suggested fix that was posted on my Github was dismissed by Asus as it wasn't resolving the core issue, merely hiding it.

 

[Tvlz]

The suggested fix that was posted on my Github was dismissed by Asus as it wasn't resolving the core issue, merely hiding it.

It should still be added as it points to an upstream linux kernel commit.

 

[Forsaken]

I didn't. I just pointed Asus at the issue thread, I believe they sent test firmwares to them, and that's the last I heard of it. The suggested fix that was posted on my Github was dismissed by Asus as it wasn't resolving the core issue, merely hiding it.

And this is the point. I flashed my AXE11000 with 386.5 with that suggested fix included and got zero panics since that time. Neither I got any related warnings in the logs. The error completely gone. I can speculate 386.5 has this "AVS" feature disabled (by default?).
But recent reports that panic occurs again on 386.7 concern me. I saw you did some merges from ASUS sources into 386.7. Is it possible one of those turned "AVS" feature on somehow?

 

[RMerlin]

It should still be added as it points to an upstream linux kernel commit.

Asus's engineer disagreed. Hiding an error message does not resolve an issue, and may lead to harder to diagnose problems.

Is it possible one of those turned "AVS" feature on somehow?

I have no idea if/how AVS can be turned on or off.

 

[strunker]

I didn't. I just pointed Asus at the issue thread, I believe they sent test firmwares to them, and that's the last I heard of it. The suggested fix that was posted on my Github was dismissed by Asus as it wasn't resolving the core issue, merely hiding it.

The firmware from Asus really masked the problem as well by disabling AVS, it doesnt really fix it by turning this feature off, it just hides it.

 

[mystik1]

The 16000 has remained stable, I left on a trip 4 days ago and havent had to worry that I wouldnt be able to connect back at all its been a complete rock, and ultimately I am sure Merlin will work with the 16000. Likely will take a bit but this router uses asus wrt same as the others, so there isnt really a reason Merlin wouldnt work on it. Just food for thought.

I am not certain if the issue I am having is the same as everyone else. I play a lot of video from my local network (connected NAS). Several times a day, it will seemingly disconnect or restart. In a couple minutes, you can navigate back to the video and restart it, but the interruption is totally unacceptable given the time it takes before you can restart and having to navigate back to the correct folder/video. I fought with it for a month, trying various things, but am finally returning it. I bought the 16000. Really encouraged to read your comments on it here. Can't wait to get it (tomorrow).

 

[strunker]

I am not certain if the issue I am having is the same as everyone else. I play a lot of video from my local network (connected NAS). Several times a day, it will seemingly disconnect or restart. In a couple minutes, you can navigate back to the video and restart it, but the interruption is totally unacceptable given the time it takes before you can restart and having to navigate back to the correct folder/video. I fought with it for a month, trying various things, but am finally returning it. I bought the 16000. Really encouraged to read your comments on it here. Can't wait to get it (tomorrow).

Its been pretty awesome.. WAY better than the 11000 ever was, I had problems with it of varying kinds even prior to the panics. 16000 has been much more solid. Its probably the same issue, thats the same behavior, obviously wouldnt be able to confirm without logs, but thus far I am a MUCH bigger fan of the 16000.

 

[Tvlz]

Asus's engineer disagreed. Hiding an error message does not resolve an issue, and may lead to harder to diagnose problems.

That makes no sense, from my understanding, changing from BUG_ON to WARN_ON doesn't hide any error message it just logs the error and continues instead of crashing the router.

 

[vlord]

That makes no sense, from my understanding, changing from BUG_ON to WARN_ON doesn't hide any error message it just logs the error and continues instead of crashing the router.

I ended up returning my AX86S due this issue. Seems like an easy enough "fix" to implement but I don't have the ability to fire up an Ubuntu instance to recompile with that flag set. It was easier to me to pay the difference and get an AX86U. So much better with zero crashes (with the exact same config). Plus, inexplicably, my wifi speeds are faster on the 5G band with the AX86U than the AX86S using the exact same channel and settings. Go figure.

 

[strunker]

I ended up returning my AX86S due this issue. Seems like an easy enough "fix" to implement but I don't have the ability to fire up an Ubuntu instance to recompile with that flag set. It was easier to me to pay the difference and get an AX86U. So much better with zero crashes (with the exact same config). Plus, inexplicably, my wifi speeds are faster on the 5G band with the AX86U than the AX86S using the exact same channel and settings. Go figure.

Nice, i had a similar experience going from the 11000 to the 16000, everything has just "worked" since and been nice and stable\fast. Its everything the 11000 should have been but wasnt due to this issue. Glad to hear that you were able to return and replace with something better.

 

[samfisher]

I ended up returning my AX86S due this issue. Seems like an easy enough "fix" to implement but I don't have the ability to fire up an Ubuntu instance to recompile with that flag set. It was easier to me to pay the difference and get an AX86U. So much better with zero crashes (with the exact same config). Plus, inexplicably, my wifi speeds are faster on the 5G band with the AX86U than the AX86S using the exact same channel and settings. Go figure.

My AX86U is also experiencing these crashes since the upgrade to 386.7

 

[L&LD]

Try the latest alpha. The RT-AX86U_386.7_1-g8fd8deefbb_49599.w version seems stable in my setup as well as many others reported here.

 

[alan6854321]

This fault is SO sporadic, it's really difficult to test.

I've been running 386.7 for 31 days without a crash, including getting through the 40C inferno here, and today - Bang! Kernel panic.

The worrying thing is that this time it didn't even reboot, just refused to allow any wireless clients to connect (They couldn't connect to the AiMesh node either). I had to power it off for a reboot.

I've loaded up 386.7_1-gcbedbbd5b0 from the test builds folder, we'll see how that goes.

 

[Forsaken]

Yep, now I can confirm AXE11000's 386.7 is also affected. Took about 16 days to catch one.

 

[L&LD]

Time to test the 386.7_2 release then.