Sporadic GT-AXE11000 reboots due to kernel panic

[strunker]

Couple updates from my side.

First, I got another Kernel panic last Monday ~14h after reboot, which is definitely bad news indicating that even everyday reboot will not guarantee 100% reliability.
Second, ASUS released another firmware, but this is rather neutral since there is nothing specific about this particular issue in their changelog.
And the last, but not least, @RMerlin released 386.5 firmware with added support for our model!

So I went ahead and checked out sources, searched for brcmnand.c files and found one with BUG_ON call in line 1339, which essentially triggers panic.
As I mentioned above, the reason for this is NAND controller reports itself as not ready 100ms after previous "read_page" operation, but I also found following commit in the link, provided by @strunker :

mtd: nand: brcmnand: Change BUG_ON in brcmnand_send_cmd · torvalds/linux@422485d

Change the BUG_ON() condition in brcmnand_send_cmd() which checks for the interrupt status "controller ready" bit to a WARN_ON. There is no good reason to kill the system when this condi...

github.com

Decided to give it a try, spend couple of hours to setup wsl-based build environment and created firmware with above change applied.
Flashed it today, disabled reboot scheduler and crossed my fingers.

Obviously, it might not help at all, since if it really problem with NAND controller gets stuck - router will lose ability to read/write from/to NAND, but at least it should not panic/reboot right away anymore.

Yeah been meaning to comment here, after a longer period of stability on the beta firmware they provide I was back to the exact same error again. They agreed to give me a NIB replacement (new in box) last week..

They wrote back today so state they have none in stock and arent sure when they will again, and that they would offer me a buy back instead. So I am kind of really.. stuck here. I dont want to flash on merlin for testing and then void any type of buy back they would otherwise be willing to do. I am an engineer for a living, but I wasnt really wanting\expecting to have to do all that just to get a 600 dollar piece of equipment to work properly.

Not entirely sure where to turn would REALLY love to try the newer gt-axe16000 mostly cuz it has 10gb lan ports, but I have no concept of how long of a wait that is going to be before that becomes available. May have to dump Asus all together and go with something else, maybe ubiquity?

 

[strunker]

Do you happen to know if all the settings are preserved going to merlin? Meaning will I have to redo all my static address assignments (I have loads of these), and other related settings?

Is it easy to go back to stock firmware in the event the panics continue? Does merlin support snmp logging lol so we are able to actually continue to log the potential panics?

Lots of questions... sigh..

 

[Forsaken]

Do you happen to know if all the settings are preserved going to merlin? Meaning will I have to redo all my static address assignments (I have loads of these), and other related settings?

Most of mine were preserved. Just had to play with Samba configuration a little bit.

Is it easy to go back to stock firmware in the event the panics continue?

AFAIK - just simple as flashing stock back, factory reset + restore previously saved config.

Does merlin support snmp logging lol so we are able to actually continue to log the potential panics?

Yep.

Lots of questions... sigh..

Again, just to emphasize: I would not anticipate it will magically gone with just flashing "original" 386.5. They real issue is likely in closed-sources part. My hope is that BUG_ON -> WARN_ON change to at least give some time to react.
In this case it would be possible to setup some sort of "bad happened" notification to gracefully end "very important Zoom call" and reboot. Not a brilliant solution, but at least can live with it.

But overall, now I would probably agree with buyback and get AX86 model.

 

[strunker]

Most of mine were preserved. Just had to play with Samba configuration a little bit.

AFAIK - just simple as flashing stock back, factory reset + restore previously saved config.

Yep.

Again, just to emphasize: I would not anticipate it will magically gone with just flashing "original" 386.5. They real issue is likely in closed-sources part. My hope is that BUG_ON -> WARN_ON change to at least give some time to react.
In this case it would be possible to setup some sort of "bad happened" notification to gracefully end "very important Zoom call" and reboot. Not a brilliant solution, but at least can live with it.

But overall, now I would probably agree with buyback and get AX86 model.

Isnt merlin a completely different firmware though? I guess I dont understand enough to get why it potentially wouldnt fix this issue.

I was also looking at the below, but just like the axe16000 there is no firm release date which makes it hard to plan for. I had a 68u for many years and it never gave me any issue so perhaps you are right there. I wanted to get the best, but at this point I just want\need something that is good and stable.

GT-AX11000 Pro Review: Asus's Best Tri-band Wi-Fi 6 Router | Dong Knows Tech

The ROG Rapture GT-AX11000 Pro is Asus's most complete Wi-Fi 6 router thanks to its support for the 5.9GHz band and multiple Multi-Gig ports. Get one today!

dongknows.com

You talking about the below yeah?

RT-AX86 Series(RT-AX86U/RT-AX86S)

ASUS gaming routers provide the very best gaming experience, with an arsenal of features and tools designed to improve online gaming performance and give you the competitive edge you need to win.

www.asus.com

 

[Forsaken]

Isnt merlin a completely different firmware though?

Well, it is pretty close to stock: https://github.com/RMerl/asuswrt-merlin.ng/wiki/Installation

I guess I dont understand enough to get why it potentially wouldnt fix this issue.

Linux kernel version is the same as stock, at least. So as closed-sourced broadcomm components, I would believe. So I would think it is not going to magically go away. But it at least gives a flexibility to try to play with open source part in hope it'll help to do something.

I was also looking at the below, but just like the axe16000 there is no firm release date which makes it hard to plan for. I had a 68u for many years and it never gave me any issue so perhaps you are right there. I wanted to get the best, but at this point I just want\need something that is good and stable.

Had ac68 (T-Mobile-branded ac1900) as well and two n56u before and wl-520gu back in the days. Never had any issues with software part at all, so decided to move forward with top notch just to be kind of future proof. My mistake.

You talking about the below yeah?

RT-AX86 Series(RT-AX86U/RT-AX86S)

ASUS gaming routers provide the very best gaming experience, with an arsenal of features and tools designed to improve online gaming performance and give you the competitive edge you need to win.

www.asus.com

AX86U particularly. Seems like their best/problemless model at the moment. Lacks 6Ghz, obviously, but even with AXE11000 I currently have one 6Ghz client at all. And for some reason it has significantly slower upload comparing to 5Ghz. Like: 30-50% better download vs 2-3 times slower upload .

 

[RMerlin]

There's a pending pull request that might resolve this issue. I have forwarded the information upstream for feedback.

 

[strunker]

Well, it is pretty close to stock: https://github.com/RMerl/asuswrt-merlin.ng/wiki/Installation


Linux kernel version is the same as stock, at least. So as closed-sourced broadcomm components, I would believe. So I would think it is not going to magically go away. But it at least gives a flexibility to try to play with open source part in hope it'll help to do something.

Had ac68 (T-Mobile-branded ac1900) as well and two n56u before and wl-520gu back in the days. Never had any issues with software part at all, so decided to move forward with top notch just to be kind of future proof. My mistake.


AX86U particularly. Seems like their best/problemless model at the moment. Lacks 6Ghz, obviously, but even with AXE11000 I currently have one 6Ghz client at all. And for some reason it has significantly slower upload comparing to 5Ghz. Like: 30-50% better download vs 2-3 times slower upload .

Right, I chose this model for the same reason. Updating network equip is.. annoying.. for people who have a lot of customizations, so I really held out for along while and chose this model due to it being "the best". Pretty frustrating, but thats good to know on the ax86u and its like half the cost.. And it has 2.5gb network too.

I also understand more what you mean on the linux kernel bit. Asus really needs to update the actual kernel itself, according to the actual Linux project member who helped code the driver, Asus is using an outdated linux kernel\build. Dont know what the fix really is for that outside Asus doing the right thing and updating the software.

Oh well, im glad we were able to suffer through this together. Let me know how stable merlin is (with your edits) and I will keep you posted on this buy back scenario, perhaps the same offer can be extended to you. I did make them aware of this thread in the many.. many.. conversations I had with them.

 

[strunker]

There's a pending pull request that might resolve this issue. I have forwarded the information upstream for feedback.

Thank you for that.

 

[RMerlin]

@Forsaken can you contact Asus through the Feedback form on the router? They would like you to try a test build. Mention in the feedback form that I'm referring you to them.

 

[strunker]

@Forsaken can you contact Asus through the Feedback form on the router? They would like you to try a test build. Mention in the feedback form that I'm referring you to them.

I would be willing to try a test build. they sent me test firmware with build numbers below, is there something new \ additional to test?

GT-AXE11000_9.0.0.4_386_47871-gd872af0_cferom_pureubi.w

 

[RMerlin]

I would be willing to try a test build. they sent me test firmware with build numbers below, is there something new \ additional to test?

GT-AXE11000_9.0.0.4_386_47871-gd872af0_cferom_pureubi.w

Yes, if you have the same kernel panic issue, contact them through the Feedback form. The test build they have specific to that issue is 386_48212. Specify that it's for the kernel panic issue, and that I referred you to them on this case.

 

[strunker]

Yes, if you have the same kernel panic issue, contact them through the Feedback form. The test build they have specific to that issue is 386_48212. Specify that it's for the kernel panic issue, and that I referred you to them on this case.

Got it, and I do, and they supplied this build originally (the one below) which looks older than the 386_48212. Thanks.

GT-AXE11000_9.0.0.4_386_47871-gd872af0_cferom_pureubi.w

 

[strunker]

Yes, if you have the same kernel panic issue, contact them through the Feedback form. The test build they have specific to that issue is 386_48212. Specify that it's for the kernel panic issue, and that I referred you to them on this case.

I reached out and they provided firmware 3.0.0.4.386_48253 to test with. Which I installed this AM, so far so good but this error is so random we wont know for a while if its better at all or not. Thanks @RMerlin

 

[RMerlin]

I reached out and they provided firmware 3.0.0.4.386_48253 to test with. Which I installed this AM, so far so good but this error is so random we wont know for a while if its better at all or not. Thanks @RMerlin

Thank you.

If @Forsaken can also get in touch with them, that would help gathering more feedback.

 

[Forsaken]

@RMerlin,

For some reasons stopped getting notifications about new replies for this thread, so just noticed your reply.
Sure will try to contact them. Living great on your 386_5 for almost a week, but as @strunker said the issue is too random to make conclusions.
Thank you for helping us!!!

Just realized there is no Feedback form in custom firmware (or am I missed something?).
Anyways, will switch back to official one and contact them afterwards.

 

[Forsaken]

@strunker ,

Just got notification from ASUS another user who is being testing firmware got an error so they prepared another firmware version. Is that you or does it mean somebody else has same problem?

 

[strunker]

@strunker ,

Just got notification from ASUS another user who is being testing firmware got an error so they prepared another firmware version. Is that you or does it mean somebody else has same problem?

Probably me. I just got back to them yest on a new panic that occurred. Installed the newest build they provided this AM. I am not running on the below. Really starting to wonder if this problem is really fixable with firmware or of its really just something hardware related that just plagues this model or a certain batch of them.

Current Version : 9.0.0.4.386_48449-gc53a19c

 

[strunker]

@strunker ,

Just got notification from ASUS another user who is being testing firmware got an error so they prepared another firmware version. Is that you or does it mean somebody else has same problem?

If you need the test firmware let me know and I can share the link they provided.

 

[Forsaken]

Probably me. I just got back to them yest on a new panic that occurred. Installed the newest build they provided this AM. I am not running on the below. Really starting to wonder if this problem is really fixable with firmware or of its really just something hardware related that just plagues this model or a certain batch of them.

Current Version : 9.0.0.4.386_48449-gc53a19c

That what they gave me as well. Is this still "BUG: failure at drivers/mtd/nand/brcmnand/brcmnand.c:1339/brcmnand_send_cmd()!" panic message?

 

[strunker]

That what they gave me as well. Is this still "BUG: failure at drivers/mtd/nand/brcmnand/brcmnand.c:1339/brcmnand_send_cmd()!" panic message?

Yup, same old problem. Although I did get further on the last build. I started having this weird sporadic wan issue though. The router wouldnt kernel panic but the wan connection drops (the wan light turns red on the router) even though the modem is perfectly fine. Then it comes back a minute or two later. Its the same symptom of a kernel panic just without the router actually crashing. Personally, its the same behavior without the panic, so I think its related. But, I cant really prove that, but the panics unfortunately continue for me.

I will post back in a week or so. I made it a little past 7 days this time before getting the most recent panic. Just dont understand how more people dont suffer from this issue. Or if they are, how they arent taking notice.