Sporadic GT-AXE11000 reboots due to kernel panic
- Thread starter Forsaken
- Start date
Im testing their latest beta firmware now, just to close that thread out. Their product team has been responsive\great\patient so I feel I owe them a valid test.
Will 100% be giving Merlin a whirl before accepting the buyback. Trying to hold out on that because there isn't much on the market today that really competes with this router so not like I can just go upgrade\sub it for something else.
You can see the below kernel panic from yest evening. Same old same old, posting so you cant call me a liar
___________________
2022-03-29 20:00:23,User.Error,10.1.1.1,Mar 29 20:00:23 Router-25F7EE8-C kernel: bcm63xx_nand ff801800.nand: timeout waiting for command 0x1
2022-03-29 20:00:23,User.Error,10.1.1.1,Mar 29 20:00:23 Router-25F7EE8-C kernel: bcm63xx_nand ff801800.nand: intfc status 700000e0
2022-03-29 20:00:24,User.Warning,10.1.1.1,Mar 29 20:00:24 Router-25F7EE8-C kernel: BUG: failure at drivers/mtd/nand/brcmnand/brcmnand.c:1348/brcmnand_send_cmd()!
2022-03-29 20:00:24,User.Emerg,10.1.1.1,Mar 29 20:00:24 Router-25F7EE8-C kernel: Kernel panic - not syncing: BUG!
2022-03-29 20:00:24,User.Warning,10.1.1.1,Mar 29 20:00:24 Router-25F7EE8-C kernel: CPU: 0 PID: 458 Comm: jffs2_gcd_mtd9 Tainted: P O 4.1.52 #2
2022-03-29 20:00:24,User.Warning,10.1.1.1,Mar 29 20:00:24 Router-25F7EE8-C kernel: Hardware name: Broadcom-v8A (DT)
2022-03-29 20:00:24,User.Emerg,10.1.1.1,Mar 29 20:00:24 Router-25F7EE8-C kernel: Call trace:
2022-03-29 20:00:24,User.Warning,10.1.1.1,Mar 29 20:00:24 Router-25F7EE8-C kernel: [<ffffffc000087398>] dump_backtrace+0x0/0x150
2022-03-29 20:00:24,User.Warning,10.1.1.1,Mar 29 20:00:24 Router-25F7EE8-C kernel: [<ffffffc0000874fc>] show_stack+0x14/0x20
2022-03-29 20:00:24,User.Warning,10.1.1.1,Mar 29 20:00:24 Router-25F7EE8-C kernel: [<ffffffc00057fea0>] dump_stack+0x90/0xb0
2022-03-29 20:00:24,User.Warning,10.1.1.1,Mar 29 20:00:24 Router-25F7EE8-C kernel: [<ffffffc00057db5c>] panic+0xe0/0x230
2022-03-29 20:00:24,User.Warning,10.1.1.1,Mar 29 20:00:24 Router-25F7EE8-C kernel: [<ffffffc000344c9c>] brcmnand_send_cmd+0xbc/0x178
2022-03-29 20:00:24,User.Warning,10.1.1.1,Mar 29 20:00:24 Router-25F7EE8-C kernel: [<ffffffc000344f40>] brcmnand_read_by_pio+0xc8/0x318
2022-03-29 20:00:24,User.Warning,10.1.1.1,Mar 29 20:00:24 Router-25F7EE8-C kernel: [<ffffffc000345248>] brcmnand_read+0xb8/0x458
2022-03-29 20:00:24,User.Warning,10.1.1.1,Mar 29 20:00:24 Router-25F7EE8-C kernel: [<ffffffc00034576c>] brcmnand_read_page+0x2c/0x38
2022-03-29 20:00:24,User.Warning,10.1.1.1,Mar 29 20:00:24 Router-25F7EE8-C kernel: [<ffffffc000340ebc>] nand_do_read_ops+0x1b4/0x640
2022-03-29 20:00:24,User.Warning,10.1.1.1,Mar 29 20:00:24 Router-25F7EE8-C kernel: [<ffffffc000341674>] nand_read+0x5c/0x88
2022-03-29 20:00:24,User.Warning,10.1.1.1,Mar 29 20:00:24 Router-25F7EE8-C kernel: [<ffffffc000336fac>] part_read+0x34/0x90
2022-03-29 20:00:24,User.Warning,10.1.1.1,Mar 29 20:00:24 Router-25F7EE8-C kernel: [<ffffffc0003348c8>] mtd_read+0x48/0x80
2022-03-29 20:00:24,User.Warning,10.1.1.1,Mar 29 20:00:24 Router-25F7EE8-C kernel: [<ffffffc000208758>] jffs2_flash_read+0x58/0x198
2022-03-29 20:00:24,User.Warning,10.1.1.1,Mar 29 20:00:24 Router-25F7EE8-C kernel: [<ffffffc00057e9d8>] jffs2_garbage_collect_pristine+0xa4/0x3bc
2022-03-29 20:00:24,User.Warning,10.1.1.1,Mar 29 20:00:24 Router-25F7EE8-C kernel: [<ffffffc00057f1e0>] jffs2_garbage_collect_live+0x37c/0xec8
2022-03-29 20:00:24,User.Warning,10.1.1.1,Mar 29 20:00:24 Router-25F7EE8-C kernel: [<ffffffc000203af8>] jffs2_garbage_collect_pass+0x408/0x830
2022-03-29 20:00:24,User.Warning,10.1.1.1,Mar 29 20:00:24 Router-25F7EE8-C kernel: [<ffffffc000205370>] jffs2_garbage_collect_thread+0xf8/0x1e8
2022-03-29 20:00:24,User.Warning,10.1.1.1,Mar 29 20:00:24 Router-25F7EE8-C kernel: [<ffffffc0000ae840>] kthread+0xd8/0xf0
djtheraven
New Around Here
I am having this exact same issue with random reboots. And even something worse. The router will just lock up. All lights on but both Wi-Fi and ethernet completely out until I pull the power. I have already returned it and new one is doing the same thing. I see some of you testing beta firmware but that was almost two months ago. Any progress? I bought two of these things in a mesh system I feel like I'm out $1,200. I have 70 smart gadgets and Wi-Fi things going. And when this happens I lose my entire house including my alarm system. Never had a router do this before.
Hate to tell you this, but I think this model is just buggy... with probably some type of hardware\software issue that is not being acknowledged. Most people likely dont notice it because they arent actively looking for it and the reboots are mostly quick. I spent months with support and the problem was never really solved. They gave me a firmware that disables "AVS", which im not entirely sure what that means or why it would cause my router to crash the way it was with kernel errors. In any event, its totally disabled on my router a this point and I havent had kernel panics for a good while now.
If you can return these and get your money back I would, the gt-axe16000 is out now and I will likely look at moving to that and dumping this 11000 it has been an extremely poor experience for me from start to finish and I wouldnt honestly recommend keeping it. With that said, if you have to, I can give you my case number and you can ask them for the version of the firmware I am on that disables AVS. Maybe that would be an alternate path to take. There is also merlin firmware for this model now that may be more stable, that you could attempt to try as well.
What I dont like about any of this though, is for 600 dollars I had to waste like 3-4 months of time, and now my router has a feature thats disabled... For a premium product I dont expect that type of experience. Are you sure your issue is the same as ours was here? Kernel panics? Do you have logging turned on to review what the logs are stating when the reboot\lock up is taking place?
alan6854321
Senior Member
Hi there,
I'm new to this thread, I found it because my AX86S rebooted this morning with exactly the same syslog entries as you've posted.
I got the following sent to my syslog server (But not in the syslog file on the router)
I don't know if this gives any clues as to what's happening, but I just made a change to the wireless config.
I bought the router about 4 weeks ago to upgrade an AC86U, I set it up exactly as the AC86U with the guest networks on slot #2 controlled by YazFi, it's worked fine without any issues.
I decided I wanted to use the old router as an Aimesh node so, in preparation, I moved the guest networks from slot #2 to #1 so they would propagate across Aimesh.
Everything seemed to work fine and all my devices connected as expected.
Then it reboots itself for the first time ever.
Just wondering if my reconfiguring the guest networks provoked it.
I'm new to this thread, I found it because my AX86S rebooted this morning with exactly the same syslog entries as you've posted.
I got the following sent to my syslog server (But not in the syslog file on the router)
Code:
23-May 06:00:20 AX kernel: bcm63xx_nand ff801800.nand: timeout waiting for command 0x1
23-May 06:00:20 AX kernel: bcm63xx_nand ff801800.nand: intfc status 700000e0
23-May 06:00:20 AX kernel: BUG: failure at drivers/mtd/nand/brcmnand/brcmnand.c:1339/brcmnand_send_cmd()!
23-May 06:00:20 AX kernel: Kernel panic - not syncing: BUG!
23-May 06:00:20 AX kernel: CPU: 0 PID: 422 Comm: jffs2_gcd_mtd9 Tainted: P O 4.1.52 #2
23-May 06:00:20 AX kernel: Hardware name: Broadcom-v8A (DT)I don't know if this gives any clues as to what's happening, but I just made a change to the wireless config.
I bought the router about 4 weeks ago to upgrade an AC86U, I set it up exactly as the AC86U with the guest networks on slot #2 controlled by YazFi, it's worked fine without any issues.
I decided I wanted to use the old router as an Aimesh node so, in preparation, I moved the guest networks from slot #2 to #1 so they would propagate across Aimesh.
Everything seemed to work fine and all my devices connected as expected.
Then it reboots itself for the first time ever.
Just wondering if my reconfiguring the guest networks provoked it.
djtheraven
New Around Here
I would think that it's the same thing. Every couple of days I get a random reboot. And then I guess randomly the reboot isn't successful and it drops all connections. Lots of people have this router and say they have no issues. Honestly the rebooting is not the issue for me. But when my whole house goes down including my alarm system hooked up to it until I get home to turn it off and on again is just unacceptable in any way. I've seen other people online with this issue but most people say they have no issues so I'm going to return again and I'm going to keep returning them until I guess I get one that works. Is there any way to manually turn off the AVS that you're talkin about without having to flash a different firmware?
I cant say that I ever noticed the guest network provoking it. Maybe we can\should try disabling it all together? Ill also note that the bug is VERY sporadic. There have been times I thought the issue was resolved on a test firmware build, and then 1.5-2 weeks later it rebooted again.
I would say if you do decide to tests turning off the guest wifi, you will want to wait 1 week minimum to see if it panics or not, two weeks to be sure that the settings change had any impact.
I dont think so. I sent them an email to questions what "AVS" actually is. And I also advised them more people are having issues. Got it, actually I went back and re-read my email. AVS = adaptive voltage scaling, assuming its related to AVS on the Nand memory on these devices. I have attached the below firmware, which is what they provided to me. This particular version did have a positive impact for me, I haven't had a kernel panic or reboot in the past 4 weeks or so which is significant as it was happening multiple times a week on my devices.
I asked them in the email I sent what can be provided to the general public as there are many of us now suffering from this problem, will report back once they email me.
TY.
1. We have fixed a critical issue in the last test firmware, which also cause the wireless clients not being in the network map.
2. Regarding the kernel panic issue, discussed with chipset vendor, it may not just a kernel issue. It may also be related to adaptive voltage scaling. Please download the following test firmware and make sure to do power cycling after firmware upgrade.
Thanks.
alan6854321
Senior Member
Hmm, that would be a right pain - many IoT devices to reconfigure, and some of them require climbing ladders and poking buttons with a paperclip!
Think I'll just hang fire and see what happens, it got through the night without a reboot.
Very interesting. As I mentioned previously, about 2 moth ago I compiled and flashed @RMerlin 's v386.5 with that BUG_ON line changed to WARN_ON. Didn't have any reboot since that time.
That said, I didn't have any NAND-related warnings in the remote logs as well. Suspecting Asuswrt-Merlin 386.5 might have AVS disabled either.
So, my advice would be to try to switch to Asuswrt-Merlin - it is really rock-solid and I like it even more than stock. For all who do not want to switch or uses features it does not provide - wait for next official release. Giving their release schedule it should be soon.
Only thing is the AVS disable isnt ever going to be put into the base firmware. They generated a special version of the firmware that disabled it for me, so even if I flash new stock firmware AVS will remain disabled probably forever. Since they dont want to disable this feature across the board only can get the special firmware from Asus to turn it off.
nzwayne
Senior Member
Just want to put my hand up, as an owner of 2x GT-AXE11000 (One in AiMesh mode) I'm also getting the random locks ups. Can be every few days, or up to say ten days. Nothing in the syslog prior to the lockup. Can't use Web Gui or SSH (no response from Router) to subtly recover via reboot, have to Power off/on.
Currently running GT-AXE11000_386.7_beta2 which is based on 386_48966 GPL. I may have to ditch these AXE11000's and go with GT-AX6000 instead as it's frustrating, especially when traveling and losing access to your IT setup.
Currently running GT-AXE11000_386.7_beta2 which is based on 386_48966 GPL. I may have to ditch these AXE11000's and go with GT-AX6000 instead as it's frustrating, especially when traveling and losing access to your IT setup.
So there is a fix for it BUT its not something they plan to put into the firmware, its a feature disable that they gave me specific firmware for. If you start a ticket we can likely get you setup with that.
Honestly, i put a ticket in with them and I insisted on a buy back from them. These types of issues arent acceptable on a premium product, period. I just purchased an axe16000 and set it up a few days ago, already its FAR more solid than the 11000 ever was. I travel this week so going to connect back to my home network and try to stress test it, but so far its better and I havent had a single panic or reboot.
As we said earlier in this thread I think there is simply a hardware issue with this router that is not acknowledged, the 16000 uses a different soc\cpu\chip that so far at least hasnt caused me any problems. This newer router also has 10gb ports which are really nice and def future proof it more so than the 11000.
nzwayne
Senior Member
Thanks for the detailed reply. Had another lockup yesterday, nothing in logs...err. Frustrating that a top-end HW product acts this way and your indicating not a formal fix or correction pending, just a circumvention. I do note this issue been with me since I purchased the routers early this year. I'd like to continue with a router product model that runs @RMerlin code as many advantages over the ASUS product release, especially when you include the Add-Ons. It's the home & business rational which myself and others I've shown have purchased the applicable ASUS routers to meet our needs. ie AXE16000 not supported, therefore N/A.
Let me contact ASUS support to hear their story and then direction to resolve. Thks again.
Aniraf
New Around Here
@strunker did they give you a firmware that disables AVS and fixes this issue? I have been fighting with this damn router for so long...I would gladly run the firmware if you can share it. I don't feel like doing a Merlin swing at the moment, although last I checked they added in the asus mesh stuff which I like.
If you arent seeing kernel panics in your logs your issue may be a bit different. However, I would suggest getting a ticket created. Next time it happens immediately do a 'submit feedback' from the router and put your ticket number in the subject, that will allow them to review the logs and link it back up with your ticket.
The 16000 has remained stable, I left on a trip 4 days ago and havent had to worry that I wouldnt be able to connect back at all its been a complete rock, and ultimately I am sure Merlin will work with the 16000. Likely will take a bit but this router uses asus wrt same as the others, so there isnt really a reason Merlin wouldnt work on it. Just food for thought.
alan6854321
Senior Member
My experience is that you don't see anything in the logs on the router.
The "Kernel panic - not syncing: BUG!" messages only appear if you are running a remote syslog server.
The "Kernel panic - not syncing: BUG!" messages only appear if you are running a remote syslog server.
Yes sir, they can and should provide direct to you. They had provided me a link, which has since expired.
I wouldnt feel comfortable providing it, for your benefit, because I have 0 clue if they custom tailored it to my specific router, or how it even really works. They were very vague with the details. All they said is it disables AVS and that it would be disabled from that point on. 100% contact them and definitely be ready to provide the logs showing the panics.
This is accurate.
I would highly recommend everyone of you do snmp logging on a separate machine. If you dont run your primary desktop 24/7 you can run this on a laptop for a while or an rpi, etc, and you can collect the logs, its really very simple and should take like 5-10 minutes to get up and running. Link is below.
https://www.solarwinds.com/free-tools/kiwi-free-syslog-server
The logs can grow rather large so I would recommend keeping an eye on that. Would also recommend that you go to firewall page on the router, and change the logging to none on the firewall. It injects a TON more data into the snmp logging and makes it harder for their support to sift through the logs.
Last edited:
vlord
Regular Contributor
I have the same issue on my RT-AX86S running Merlin 386.7. I setup Lan-Secure Syslog Center for my Mac and got a similar kernel panic response:
Code:
2022-07-01 06:59:03 User.Error RT-AX86S-C32D515-C kernel: bcm63xx_nand ff801800.nand: timeout waiting for command 0x1
2022-07-01 06:59:03 User.Error RT-AX86S-C32D515-C kernel: bcm63xx_nand ff801800.nand: intfc status 700000e0
2022-07-01 06:59:03 User.Warning RT-AX86S-C32D515-C kernel: BUG: failure at drivers/mtd/nand/brcmnand/brcmnand.c:1339/brcmnand_send_cmd()!
2022-07-01 06:59:03 User.Emergency RT-AX86S-C32D515-C kernel: Kernel panic - not syncing: BUG!
2022-07-01 06:59:03 User.Warning RT-AX86S-C32D515-C kernel: CPU: 1 PID: 1398 Comm: WebHistory Tainted: P O 4.1.52 #2
2022-07-01 06:59:03 User.Warning RT-AX86S-C32D515-C kernel: Hardware name: Broadcom-v8A (DT)This happens randomly every 2-3 days
Last edited:
Similar threads
Similar threads
| Thread starter | Title | Forum | Replies | Date |
|---|---|---|---|---|
| J | Asus router reboots same time each night | ASUS Wi-Fi | 1 |