RT-AX3000 crashing every few days in bcmsw_rx (Tried hw/sw reset/rebuilds and even new replacement router)

[icodehead]

The issue may be with Guest Network 1: https://www.snbforums.com/threads/i...addresses-in-any-386-build-on-rt-ax56u.75301/ Try using only Guest Networks 2 and 3 and see if your problem goes away.

Thanks JWoo for the feedback. That was a good find. I thought I had IPV6 disabled but the syslog shows "compile time options: IPv6" so I'm not sure. I checked the syslogs for any IPV6 addresses and didn't find any. Do you still think it would be worth a test to go back to merlin and not use Guest Network 1? A lot of work!

 

[JWoo]

Yes. The guest network function is closed source, so unlike Ragu we don't know what is in there. For sure IPv6 and Guest Network 1 do not play nice due to the AIMesh code Asus put into the new Guest Network function in the 386 based builds. There could be other bad interactions too, and that is what I am thinking. Try Guest Networks 2 and 3 and see if your router stabilizes. Take care!

 

[JWoo]

I have been up stable and running on the 386.4 Alpha 1 build now for 8 days since moving my IoT devices off Guest Network 1. This level of stability was not possible when using Guest Network 1 + IPv6 on my RT-AX56U. Hope that Asus will go ahead and fix the coding in the Guest Network function as the recode Asus did to support AIMesh is what was causing all the crashes for me.

 

[JWoo]

No telling what Asus stuffed into Guest Network 1 as their intent was to allow it to be used with AIMesh. My suggestion would be to move to Guest Network 2 and see if that helps your situation. It is easy to change guest networks so not a lot of work to test this.

 

[seanwo]

I have never enabled IPv6 on my configurations but I have used guest network 1. For the past 3? weeks I have slowly been building up my router (feature by feature) to see what might crash it on the latest build since I had previously identified the problems starting with build 386.2_6 (where new broadcom drivers were added). The tests have been stable so far (3 router changes with 7 days between them to ensure it is stable) but I just got to the point where I am re-enabling the guest network feature. When I reach the point where the router starts crashing every few days I will try to switch it from guest network 1 to 2 to see if it makes a difference. The only thing I have on guest network 1 is 2 Chromecast ultras and 2 FireTV sticks both running on 5ghz. Were your guest networks 2.4ghz or 5ghz or both @JWoo ?

 

[JWoo]

Both bands. But my devices on 2.4Ghz are older and do not support IPv6. I can state factually that on Guest Network 1, I was getting crashes from several times a day to in a best case a crash every 1-2 days. Using Guest Network 2 on the latest 386.4 alpha 1 build, I have been running for several weeks now with no crashes. So I am very sure that the closed source code from Asus on the Guest Network 1 is the issue as they changed that Guest Network 1 code for all the 386 builds to support their AIMesh implementation. So in my case, the interaction between IPv6, Guest Network 1 and the Broadcom drivers on any 386 based build was causing the crashes and I could also see in the IPV6 log incorrectness in the IPv6 LAN side assignments. There could of course be other interactions on Guest Network 1 that cause crashes and you may find some new scenario on Guest Network 1 that is wonky.

 

[seanwo]

As a long shot, I just put my whole configuration back on the latest stable build, and moved my guest network from guest network 1 to guest network 2 using a complete rebuild form the ground up again. Unfortunately, it still crashed with "Process bcmsw_rx (pid: 239, stack limit = 0xd77a8210)" after 2 days. I am glad it solved the IPV6 issues but it seems like there is still something wrong with the Broadcom driver stability in my opinion staring with the driver update that happened 386.2_6. As one of the last tests before I just give up on this dumpster fire and go back to 386.2_4 (or maybe even 384.19) I have just turned off the guest network to see if it makes any difference.

 

[seanwo]

Here are my notes so far on what crashes it and what does not:

All tests were using 396_3.2. Each week I added a bit more...

Week 1: (stable 7 days)
----------------

lan:lan ip
-ip address: 192.168.0.1

administration:system
-router login name: [admin username]
-new password: [new password]
-retype password: [new password]

wireless:general
-2.4ghz network name: [ssid]
-2.4ghz wpa pre-shared key: [key]
-5ghz network name: [ssid]
-5ghz wpa pre-shared key: [key]

lan:dhcp server
-[add manual assignments]

week 2: (stable 7 days)
----------------

administration:system
-enable jffs custom scripts and configs: yes
-enable ssh: lan only
-allow password login: no
-authorized keys: [public key]

wireless:general
-2.4ghz control channel: 1
-5ghz control channel: 36

jffs scripts
-ssh -i ~/.ssh/id_rsa_router admin@192.168.0.1
-create: jffs/scripts/wan-start (to email me when there is a crash)

week 3: (stable 7 days)
---------------------

guest network 1 (5ghz)
-[ssid], wpa2-personal, [key], unlimited access, enable

wireless:general
-5ghz channel bandwidth: 40mhz

week 4: (experiment; crashed after 2 days)

Full configuration using guest network 2 instead of guest network 1 as per conversations above.

The only things left that were not added prior to a minimal crashing setup included changing the DNS and setting the 2.4ghz network bandwidth to 20hz, so since I have a full config right now as part of the one off experiment this weekend, I just removed the guest network, set 2.4ghz to the default of 20/40, and removed the custom DNS to see if one of those are the trigger. Baring any positive results I can pickup the testing from where I ended on stable week 3 when I find the will to keep this madness up

 

[JWoo]

I would bet that if you go back to the latest 384 build that your router will be stable. You might want to do that and run for a week or so on 384 just to rule out any HW failure.

 

[seanwo]

I would bet that if you go back to the latest 384 build that your router will be stable. You might want to do that and run for a week or so on 384 just to rule out any HW failure.

I would be stable given I have done that. I also have 2 of these routers and it happens on either one I use

This is the testing I had done weeks ago with a minimal setup that could crash it. Each build was a complete wipe and rebuild.
+ means stable for 7 days, - means crashed before 7 days, and ? means not tested.

+ 384.19 (14-Aug-2020)
? 386.1 (30-Jan-2021)
+ 386.1_2 (12-Feb-2021)
? 386.2 (2-Apr-2021)
+ 386.2_2 (13-Apr-2021)
+ 386.2_4 (30-Apr-2021)
- 386.2_6 (6-June-2021) <-- problem starts with this version of the firmware
- 386.3 (23-July-2021)
- 386.3_2 (6-Aug-2021)

386.2_6 is the build that contains the broadcom driver updates for fragattack

Thanks for your posts. At least I know to stay away from guest network 1 now in the future along with anything else I find.

 

[seanwo]

Another crash within 1 day of pulling a few features out of full buildout I did testing the difference between using guest network 1 and guest network 2. I could go back to my minimal config and keep building it up week by week until I find something but there have been no comments from the dev team acknowledging a problem with 386 or integration with the closed source broadcom drivers so I just went back to 384.19 last night on my router and my two APs. I will check back in 6 months to see if there have been stability improvements made in 386 or new any new broadcom drivers integrated. I would have liked to use 386 in case there are critical security updates but a router rebooting every 2-4 days in the middle of business meetings is not something I am willing to trade off for another few months? of debugging. If anyone is having this problem and comes across a solution please post it. If the development team identifies an issue in this area and has a potential fix please post and I will try that as well. As of now I am certain the problem started with build 386.2_6 and is related to the details I gave in this post http://www.snbforums.com/threads/rt...even-new-replacement-router.74503/post-717803. Thanks to those that commented and I am glad that some workarounds for similar issues have been resolved. Unfortunately, I continue to be plagued by "bcmsw_rx" so I just want my network to work at this point. 384 is remarkably stable and is why I came to and enjoyed merlin builds to start with across many routers over the years. I hope that stability returns in the future. Cheers.

 

[Suzib6sw]

Another crash within 1 day of pulling a few features out of full buildout I did testing the difference between using guest network 1 and guest network 2. I could go back to my minimal config and keep building it up week by week until I find something but there have been no comments from the dev team acknowledging a problem with 386 or integration with the closed source broadcom drivers so I just went back to 384.19 last night on my router and my two APs. I will check back in 6 months to see if there have been stability improvements made in 386 or new any new broadcom drivers integrated. I would have liked to use 386 in case there are critical security updates but a router rebooting every 2-4 days in the middle of business meetings is not something I am willing to trade off for another few months? of debugging. If anyone is having this problem and comes across a solution please post it. If the development team identifies an issue in this area and has a potential fix please post and I will try that as well. As of now I am certain the problem started with build 386.2_6 and is related to the details I gave in this post http://www.snbforums.com/threads/rt...even-new-replacement-router.74503/post-717803. Thanks to those that commented and I am glad that some workarounds for similar issues have been resolved. Unfortunately, I continue to be plagued by "bcmsw_rx" so I just want my network to work at this point. 384 is remarkably stable and is why I came to and enjoyed merlin builds to start with across many routers over the years. I hope that stability returns in the future. Cheers.

So one of the major changes with the 386 code was the ability to extend the guest network to the nodes.. Do you have that enabled? Try turning off the feature in guest networks, then see if you still have problems..

 

[seanwo]

So one of the major changes with the 386 code was the ability to extend the guest network to the nodes.. Do you have that enabled? Try turning off the feature in guest networks, then see if you still have problems..

I can get it to crash without any guest network running unfortunately. Thanks.

 

[icodehead]

Update since Oct 21, 2021
For what its worth, since going back to ASUSWRT 3.0.0.4.386_45898, I finally had 3 crashes after about 3 weeks running steady. Just 2 days before the first crash I had added the last bits to complete my setup which was DNS-over-TLS (DoT Strick or Opportunistic) using cloudflare, set Guest 2 ap_isolate=0 and turned on AiProtection. IPv6 was never enabled. I'm working with ASUS T2 support but have yet to hear from them about this Feedback data.

As a side note, I have also had issues with DNS-over-TLS when connected to 2.4G Guest 1 or 2 Network. I experienced this running Google TV Chromecast or Sony Bravia TV. Every time I start a video from Hulu or HBOMax, the app would consistently freeze or give a timeout error after 15 plus seconds. Apple TV also was slower then normal and wasn't able to post images on all the content.
I tried other DNS services without success. Also other apps all work fine like Netflix, Amazon Prime, Apple TV, Philo, Peacock, Youtube, PBS, etc.
BUT ... everything works fine if the Google TV Chromecast or Sony Bravia TV are connected to the 2.4G eth5 main wireless network!

I have since gone back to unsecured DNS using cloudflare and can live with this a short time but I have 9 months of warranty left so it better get fixed by then.
Until ASUSWRT is stable I don't see a point to move back to Merlin. At least ASUSWRT don't crash as often.
I'll post back if this doesn't stabilize.

 

[m33ts4k0z]

I know this is not ideal, but all of you trying to do DNS work on the router, I would suggest to try to do DoT off the router like on a rpi for example. Just a suggestion.

 

[JWoo]

I went through RT-AX58U warranty replacement refurb routers and worked with Asus support for a number of weeks last year trying to prevent the crashes on the RT-AX58U; Asus eventually wound up buying back my RT-AX58U as in their own testing they found a problem that their techs called "slow WIFI" which implies either a chipset or driver issue. I am sure others on this forum will differ with me, but I think the RT-AX58U router has a systemic defect in design or manufacturing as no configuration I tried would fully prevent all the crashes. It seems like many others on this forum have had similar results with this model.

That is why I have an RT-AX56U right now because this HW platform is stable and as long as I don't run IPv6 with Guest Network 1 everything runs for weeks and weeks with no crashes at all. The quad core runs very cool and under low CPU load.

 

[Donga120]

Isolate Ethernet and Wi-Fi devices if not already done so. I had a similar issue with random reboots and crashes, new router, flashed multiple firmwares, initialised etc, was ongoing for about 4 months. In the end it was because my Marantz amp was hardwired and for some reason did not like it. I unplugged the Ethernet cable and switched it to Wi-Fi and just like that, it stopped.

 

[JWoo]

Isolate Ethernet and Wi-Fi devices if not already done so. I had a similar issue with random reboots and crashes, new router, flashed multiple firmwares, initialised etc, was ongoing for about 4 months. In the end it was because my Marantz amp was hardwired and for some reason did not like it. I unplugged the Ethernet cable and switched it to Wi-Fi and just like that, it stopped.

Not doubting your troubleshooting, but if an external device can cause a router to crash that is a major design shortcoming of the router. Using Ethernet or WIFI is a very basic router function; if the RT-AX58U cannot manage that it is really astounding and violates every principle of HW and SW design.

 

[Donga120]

Not doubting your troubleshooting, but if an external device can cause a router to crash that is a major design shortcoming of the router. Using Ethernet or WIFI is a very basic router function; if the RT-AX58U cannot manage that it is really astounding and violates every principle of HW and SW design.

Agreed, but everything I tried (multiple cables, DHCP, static IP, multiple DNS servers) still caused the issue. It could also be the Marantz at fault. Either way, it fixed it for me and I am happy I was able to resolve it or at least knew what the cause was.

 

[JWoo]

T

Agreed, but everything I tried (multiple cables, DHCP, static IP, multiple DNS servers) still caused the issue. It could also be the Marantz at fault. Either way, it fixed it for me and I am happy I was able to resolve it or at least knew what the cause was.

The Marantz may have violated some signaling or voltage requirement, sure it is possible. I don't know which chipset(s) Marantz is using, but for sure they are not making their own chipset. Out of curiosity had you used your Marantz device with any other router without crashing it prior to the RT-AX58U? Just curious.....