RT-AX3000 crashing every few days in bcmsw_rx (Tried hw/sw reset/rebuilds and even new replacement router)

[L&LD]

Sometimes, it's not just the firmware version, it's also what you do after, too.

 

[seanwo]

Sometimes, it's not just the firmware version, it's also what you do after, too.

yup. do you see anything that might be a culprit in the minimal setup I do after a flash and hardware reset/jffs format: https://www.snbforums.com/threads/r...even-new-replacement-router.74503/post-717803

 

[L&LD]

You're changing too many things. That is not a minimal setup to me. Follow the links I provided and see if you can methodically track down the step(s) you're doing that are causing you issues.

I just checked on a customer this morning that has been using an RT-AX58U for the last year (I called him back to verify). I updated Diversion and considered restarting the network too. This was hours earlier, but there is no need. The router/network has been working stable since Sep 17 @ 18:11.

There are no issues with 386.xx firmware at all, in any RMerlin supported router that has been properly set up.

 

[seanwo]

You're changing too many things. That is not a minimal setup to me. Follow the links I provided and see if you can methodically track down the step(s) you're doing that are causing you issues.

I just checked on a customer this morning that has been using an RT-AX58U for the last year (I called him back to verify). I updated Diversion and considered restarting the network too. This was hours earlier, but there is no need. The router/network has been working stable since Sep 17 @ 18:11.

There are no issues with 386.xx firmware at all, in any RMerlin supported router that has been properly set up.

I will definitely try to to minimize my setup even more but at some point I don't see the point of using the router if I can't take advantage of basic features like DNS and Guest networks. Since all these features (and more) worked quite well in older builds I am not sure how my router is "not properly setup". I have been very careful to do hardware resets and jffs formating with every build I tested for months to narrow down the actual github commit that introduced instability in my case. My setup is a compromise between trying to find what is wrong and having a usable network in my home during that time. I have a friend that has the same router and it works well for him, so yes, I would say this is working for some people and not others. It might be related to features that are turned on. It might be related to clients and usage patterns. In the end, I would not expect a router to crash for either of those things. We will keep trying to get to the bottom of it. Thanks for the advice.

Looking at your guides, I actually have been following the one for WPS reset and initialization you wrote; I did not actually put the name to what I had printed out. I found that guide at the beginning of my journey here. Great guide in what to do to get the firmware flashed to a clean state. Thanks for that too.

 

[L&LD]

There are many things that have changed in the 386.xx builds. Do not assume anything that was working before will work now. Everything is suspect, and everything needs testing/verifying before being considered part of the new 'defaults', for you.

Documentation | Asuswrt-Merlin

 

[seanwo]

There are many things that have changed in the 386.xx builds. Do not assume anything that was working before will work now. Everything is suspect, and everything needs testing/verifying before being considered part of the new 'defaults', for you.

Documentation | Asuswrt-Merlin

Totally I started by focussing on what changed when the testing showed the instability started https://github.com/RMerl/asuswrt-merlin.ng/compare/386.2_4...386.2_6. Time to go further down the rabbit hole when time permits me.

 

[seanwo]

The next test I will preform this weekend will basically be to use the latest stable 386 build (I don't want to introduce any problems by using the alpha) with the following configuration:

Full WPS reset as per @L&LD

lan:lan ip
ip address: 192.168.0.1

administration:system
router login name: [admin username]
new password: [new password]
retype password: [new password]
enable jffs custom scripts and configs: yes

lan:dhcp server
[add manual assignments] <-- I have to have a few in my house; no getting around that.

wireless:general
(2.4ghz)
network name: [network name]
wpa pre-shared key: [key]

(5ghz)
network name: [network name]
wpa pre-shared key: [key]

[Only enable ssh long enough to add a wan-start script to send an email to detect a reboot; then turn it off once the wan-start is created]:

create the jffs/scripts/wan-start

Anyone disagree this is a good minimal configuration to start with?

 

[m0rF1uS]

I would say after you do the reset just set up WiFi password and whatever manual IP addresses you need to do and leave it to run like that. No scrips, just the default settings for the firmware. After 7days you can log in and see if the router restarted or not …. Doesn’t really matter if it did on the first day or last. That will exclude most of the things L&LD was talking about as variables.
after this is complete and depending on the results you can continue troubleshooting accordingly.

 

[seanwo]

I would say after you do the reset just set up WiFi password and whatever manual IP addresses you need to do and leave it to run like that. No scrips, just the default settings for the firmware. After 7days you can log in and see if the router restarted or not …. Doesn’t really matter if it did on the first day or last. That will exclude most of the things L&LD was talking about as variables.
after this is complete and depending on the results you can continue troubleshooting accordingly.

OK. I will forgo running scripts and just check it every 24 hours to see what happened if I don't notice the reboot during daily use.

 

[JWoo]

Personally, I login once a day and look at the uptime on the General Log page. If you look after 7 days when the logfile is pretty large, it can be a science project to figure out what the uptime is and how many crashes have been experienced. I am only myself checking once a day to develop an understanding of the problem. If the router were stable, I would probably logon once a week or so. I am right now on 384.18 on my RT-AX56U and so far it is stable with a full configuration (no scripts or addons, but IPv6, 2 guest networks, 2 main networks, DoT). If it gets through the weekend with no crashes, I will probably switch back to the 386 base and start off with a reduced configuration to narrow down what function might be causing the problem. For me, I would probably take away Stubby first, then IPv6 if the router is still crashing and then Guest Networks to see which function is causing the crashes.

My opinion is that these crashes on the 386 base are widespread, but many users aren't checking frequently enough their uptime to know. As above, I login daily (which takes 5 minutes) and if my uptime on the General Log page is less than a day, then I know the router crashed in the last 24 hours and I save the crash log. If the uptime is more than a day, I know the router has not crashed.

 

[JWoo]

The other odd thing that I notice is that on the 384.18 build, I have 7 devices that pick up IPv6 addresses on the LAN side (in the IPv6 log). On any of the 386 builds (386.3-2, 386.2-2 and 386.4 alpha 1), the LAN side (IPv6 log) will have 0 devices or sometimes 1 device. And on the 386 builds, the wireless log (which shows the LAN side IP addresses including IPv6) never matches the IPv6 log. On the 384.18 build, the IPv6 addresses shown for the 7 devices on the wireless log match the 7 devices/IP addresses in the IPv6 log. So there is definitely something going on regarding IPv6 on the 386 builds.

 

[seanwo]

Personally, I login once a day and look at the uptime on the General Log page. If you look after 7 days when the logfile is pretty large, it can be a science project to figure out what the uptime is and how many crashes have been experienced. I am only myself checking once a day to develop an understanding of the problem. If the router were stable, I would probably logon once a week or so. I am right now on 384.18 on my RT-AX56U and so far it is stable with a full configuration (no scripts or addons, but IPv6, 2 guest networks, 2 main networks, DoT). If it gets through the weekend with no crashes, I will probably switch back to the 386 base and start off with a reduced configuration to narrow down what function might be causing the problem. For me, I would probably take away Stubby first, then IPv6 if the router is still crashing and then Guest Networks to see which function is causing the crashes.

My opinion is that these crashes on the 386 base are widespread, but many users aren't checking frequently enough their uptime to know. As above, I login daily (which takes 5 minutes) and if my uptime on the General Log page is less than a day, then I know the router crashed in the last 24 hours and I save the crash log. If the uptime is more than a day, I know the router has not crashed.

Agreed on most of the points above. I did not notice it for a while because the crash would happen overnight a lot. But when it started crashing in the middle of business meetings I started to get more serious figuring out what was going on. I truly believe it is the Broadcom driver update in 386.2_6 (to fix the fragattack) and that 386.2_4 is the last stable build; it took weeks of effort to narrow that down.

For me using wan-start gave me a way to audit the crashes since it would send me email every time it rebooted: https://github-wiki-see.page/m/RMerl/asuswrt-merlin.ng/wiki/Sending-Email Basically, if you get this email and the uptime is 0, it crashed. With that, I could now run firmware version experiments and be notified whether if failed. If I had not heard from the router in a week, I knew that version of the firmware was stable and moved on to the next one.

At this point, I am just going to go basically go with a default setup.. configuring my SSIDs and network keys to show that this has nothing to do with my setup. My money is on clients triggering this with those Broadcom drivers or some interaction with the guest network blowing it up. If it is the guest network, I should know with the next set of tests. I need to wait till the weekend to spin up more testing since we have day jobs too and actually need to use the internet. I have been stable for 13 days on 386.2_4 to get work done.

 

[JWoo]

That sounds like a good plan. For me, once I get through the weekend confirming stability on 384.18 or not, I will reduce my configuration on the 386 base by taking away Stubby and IPv6. If it is still not stable, I will remove the guest networks.

 

[Suzib6sw]

That sounds like a good plan. For me, once I get through the weekend confirming stability on 384.18 or not, I will reduce my configuration on the 386 base by taking away Stubby and IPv6. If it is still not stable, I will remove the guest networks.

Just to throw a wrench into that hypothesis, Im not running any scripts or IPV6. I do have Guest networks . I have been "up" for 2 days now since flashing Alpha 4 over itself on all nodes.

 

[seanwo]

Just to throw a wrench into that hypothesis, Im not running any scripts or IPV6. I do have Guest networks . I have been "up" for 2 days now since flashing Alpha 4 over itself on all nodes.

If you make it to 7 days I would be interested in a list of changes to defaults and whether that same configuration fails on the latest stable build.

 

[JWoo]

TEST 1
I just got to 5 days without any crashes on my RT-AX56U running firmware version 384.18. In my configuration I have:
- Two main networks (2.4 and 5.0 Ghz using WPA2 and fixed channels)
- Two guest networks (using 1 and 2 with WPA2 on both)
- Stubby DNS over TLS with 4 IPv4 and 4 IPv6 servers
- IPv6 native with prefix delegation / stateless

The 384.18 build is very stable with the functions I use. With any of the 386 builds I have tried (386.2-2, 386.3-2, 386.4 alpha1), the router would have crashed multiple times within the first couple of days. @john9527 it could be interesting to have a maintained 384 based build for the AX class routers as these builds are rock solid.

I noticed that on the 384.18 build that IPv6 works great and 1) I have 9 devices with LAN side IPv6 addresses and 2) the SYSTEM LOG -> WIRELESS matches the SYSTEM LOG -> IPv6. On the 386 builds using the same setup (with a clean flash), IPv6 has some faults, meaning 1) 1 - 2 devices get a LAN side IPv6 address and 2) the SYSTEM LOG -> WIRELESS shows the 1 IPv6 address while SYSTEM LOG -> IPv6 shows no IPv6 addresses at all.

TEST 2
My plan now is to go back to 386.3-2 with a clean flash and I will not use Stubby. The aim is to see if I get LAN side IPv6 addresses without Stubby in use and to see if the router also stabilizes on the 386 build. If the router stabilizes without Stubby, then very likely some interaction between IPv6 and Stubby is what is causing my crashes on the 386 builds.

 

[icodehead]

I have also had the crashing issue on the RT-AX58U for over 3 months. I finally went back to AsusWrt current version 3.0.0.4.386_45898 which is working fine with no crashes.
I can confirm that Merlin 386.2_6 was when I first noticed the crashing. At this point I'm staying with AsusWrt until this gets sorted out.
Thanks everyone for all your efforts.

 

[seanwo]

I have also had the crashing issue on the RT-AX58U for over 3 months. I finally went back to AsusWrt current version 3.0.0.4.386_45898 which is working fine with no crashes.
I can confirm that Merlin 386.2_6 was when I first noticed the crashing. At this point I'm staying with AsusWrt until this gets sorted out.
Thanks everyone for all your efforts.

Thanks for the data point. I am running some tests now to try to isolate the cause (whether it be router feature or client pattern). What features would you change from a stock reset of the RT-AX58U (ie. channels from auto->specific?, guest network(s)?, jffs scripts runnable?, etc?). Also, do you have any clients that match my list here: http://www.snbforums.com/threads/rt...even-new-replacement-router.74503/post-718355. Thanks!

 

[icodehead]

Thanks for the data point. I am running some tests now to try to isolate the cause (whether it be router feature or client pattern). What features would you change from a stock reset of the RT-AX58U (ie. channels from auto->specific?, guest network(s)?, jffs scripts runnable?, etc?). Also, do you have any clients that match my list here: http://www.snbforums.com/threads/rt...even-new-replacement-router.74503/post-718355. Thanks!

I have many devices but those on your list I have several Chomecasts including Google TV and Amazon Fire Tablet.
I minimized my setup from stock to use Guest 1 for IoT and the Guest2 had AP_Isolated enabled for my Chromecast devices.
I also configured WAN Dot using cloudflare.
Nothing else out of the ordinary.

 

[JWoo]

The issue may be with Guest Network 1: https://www.snbforums.com/threads/i...addresses-in-any-386-build-on-rt-ax56u.75301/ Try using only Guest Networks 2 and 3 and see if your problem goes away.