RT-AX86U reboot loop issue when adding AiMESH node

[IcecreamVille]

Hello,

Yesterday I posted two scenarios where my RT-AX86U goes into a reboot loop - post with details is here

Rather than continue that thread, I thought I'd start a new one, because I am completely perplexed on this.

My current setup has all devices on Merlin 386.7_2. The main router was factory reset after flashing Merlin, with a format of JFFS, and all setup done from scratch. It does have a USB drive (for media server) attached.

As I mentioned, I have an RT-AC86U that is an AiMesh node. My AiMesh backhall is presently all wireless. I was looking at the topology last night & noticed that one of my RT-AC86U nodes decided to become a backhaul to an RT-AC68U (RT-AC1900P, to be more specific). I decided to associate it with the main router. When this happened, the main router went into a reboot loop - watching it, the Internet light goes out, then comes back on with red, and goes to white for around 2 seconds & reboots again. Powering off the RT-AC86U stopped the RT-AX86U from the reboot loop.

This morning, I factory reset the RT-AC86U & decided to add it back into the AiMesh. I hooked up an Ethernet cable to the WAN of the RT-AC86U & the other end to Ethernet port 1 on the RT-AX86U. AiMesh node discovered & successfully added. Reboot loop started again.

Any clues what could be going on?

The entries from the system log from right before / during reboot cycle until it stopped are attached (unedited).

I'd rather not do a factory reset & start from scratch again, considering this was not a "dirty setup" - flashed to Merlin 386.7_2, factory reset, format JFFS, program router.

I do have another RT-AX86U as a node - I was thinking of pulling it, factory resetting, restoring my router's configuration to it, to see if the problem also exists on that one. I find it difficult to believe this may be a hardware issue, and even more curious why this is happening to begin with.

Last week when I set the router up, I was able to add the RT-AC86U without issue - the problem started when I changed the preferred connection to be that of my main router. I also have another RT-AC86U in my AiMesh setup, that works fine, so I don't think it is an incompatibility between the RT-AX86U & RT-AC86U.

 

[IcecreamVille]

It doesn't appear that my attachment is going thru - tried adding it again & don't see it. I added as a .TXT file. Is that filetype not valid for attaching?

Trying to include contents of logs in-line won't allow me to post (says to shorten the message).

Also don't know if this should have been posted in the Merlin section - a moderator can move if more appropriate there, as the firmware is a the merlin variant.

Thank you

 

[bbunge]

Hello,

Yesterday I posted two scenarios where my RT-AX86U goes into a reboot loop - post with details is here

Rather than continue that thread, I thought I'd start a new one, because I am completely perplexed on this.

My current setup has all devices on Merlin 386.7_2. The main router was factory reset after flashing Merlin, with a format of JFFS, and all setup done from scratch. It does have a USB drive (for media server) attached.

As I mentioned, I have an RT-AC86U that is an AiMESH node. My AiMESH backhall is presently all wireless. I was looking at the topology last night & noticed that one of my RT-AC86U nodes decided to become a backhaul to an RT-AC68U (RT-AC1900P, to be more specific). I decided to associate it with the main router. When this happened, the main router went into a reboot loop - watching it, the Internet light goes out, then comes back on with red, and goes to white for around 2 seconds & reboots again. Powering off the RT-AC86U stopped the RT-AC86U from the reboot loop.

This morning, I factory reset the RT-AC86U & decided to add it back into the AiMESH. I hooked up an Ethernet cable to the WAN of the RT-AC86U & the other end to Ethernet port 1 on the RT-AX86U. AiMESH node discovered & successfully added. Reboot loop started again.

Any clues what could be going on?

The entries from the system log from right before / during reboot cycle until it stopped are attached (unedited).

I'd rather not do a factory reset & start from scratch again, considering this was not a "dirty setup" - flashed to Merlin 386.7_2, factory reset, format JFFS, program router.

I do have another RT-AX86U as a node - I was thinking of pulling it, factory resetting, restoring my router's configuration to it, to see if the problem also exists on that one. I find it difficult to believe this may be a hardware issue, and even more curious why this is happening to begin with.

Last week when I set the router up, I was able to add the RT-AC86U without issue - the problem started when I changed the preferred connection to be that of my main router. I also have another RT-AC86U in my AiMESH setup, that works fine, so I don't think it is an incompatibility between the RT-AX86U & RT-AC86U.

Generally, the nodes should be on Asus factory firmware. There is no advantage on a node to use Merlin.

You may want to give the Asus 388 beta firmware a try on the main AX86U. It works well.

 

[IcecreamVille]

If anyone is interested, I thought I'd provide an update.

Because voltage is a continual theme of the RT-AX86U rebooting, I thought it wouldn't hurt to pull the USB drive (self-powered). I then tried plugging in the RT-AC86U that is causing my router to go into a continuous reboot loop. Issue persisted - not the USB drive.

I then restored my config to a backup that I had made on 9/3, right after I setup my router. Note at this point in time, the RT-AC86U that is giving me problems was in the AiMesh config & was working - the reboot issue began a few days ago when I tried adjusting the topology.

I powered on the RT-AC86U, figuring that it would boot & that my RT-AX86U would stay up, since my RT-AC86U node was reset & re-added to the mesh setup yesterday - and then I would attempt to re-add the node & see what happens. However, the same reboot loop happened as soon as the RT-AC86U booted.

What concerns me isn't so much the issue of the RT-AC86U causing the RT-AX86U to reboot as soon as it comes on-line - I had also mentioned an Ethernet cable plugged into an old RT-AC68U as a media bridge with no connection at the other end causing the same issue - what bothers me is if these 2 things can cause the router to go into a reboot loop, what if another device connects & has a setting that the router doesn't seem to like - will this cause a continuous reboot loop until the offending device is removed?

Another curious thing is if a node in AiMesh isn't detected, shouldn't it be in offline mode in the AiMesh section? Ideally I'd like to remove the node (even though its not showing up, even as off-line) & then try adding it again. If I'm able to SSH into the router, is there a command to list all associated AiMesh nodes (even those currently off-line) & a command to remove a node?

Next steps (as soon as I'm able to get everyone out of the house) are the following (until one of them resolves the issue):

• Factory reset RT-AX86U, including formatting JFFS & setup from scratch again.
• Swap my RT-AX86U AiMesh node with the router, to see if problem happens with the other RT-AX86U setup as the main router.
• Flash the problematic node with the ASUS firmware instead of Merlin's.
• Try the new ASUS 388 Beta software to see if the issue is resolved.

As I mentioned in another post, been lurking & using advice on this forum for many years - just haven't been posting. I'm hopeful I may be able to contribute in the future.

 

[OzarkEdge]

What concerns me isn't so much the issue of the RT-AC86U causing the RT-AC86U to reboot as soon as it comes on-line - I had also mentioned an Ethernet cable plugged into an old RT-AC68U as a media bridge with no connection at the other end causing the same issue - what bothers me is if these 2 things can cause the router to go into a reboot loop, what if another device connects & has a setting that the router doesn't seem to like - will this cause a continuous reboot loop until the offending device is removed?

I tend to attribute random rebooting to hardware issues, not settings. To troubleshoot, disconnect everything and rebuild one piece at a time. And don't overlook interconnects/cabling/connectors/switches. And the AC86U's history of burning out, which can typically begin with a period of strange behavior you won't be able to attribute to something straightforward like settings.

Another curious thing is if a node in AiMESH isn't detected, shouldn't it be in offline mode in the AiMESH section?

Only when everything is healthy and operating normally. That's not the case here.

Ideally I'd like to remove the node (even though its not showing up, even as off-line) & then try adding it again. If I'm able to SSH into the router, is there a command to list all associated AiMESH nodes (even those currently off-line) & a command to remove a node?

Get the router healthy and use the normal method to remove a node. Cutting corners under the covers is not going to fix a hard defect.

Next steps (as soon as I'm able to get everyone out of the house) are the following (until one of them resolves the issue):

• Factory reset RT-AX86U, including formatting JFFS & setup from scratch again.
• Swap my RT-AX86U AiMESH node with the router, to see if problem happens with the other RT-AX86U setup as the main router.
• Flash the problematic node with the ASUS firmware instead of Merlin's.

Without reviewing everything posted and what you have and have not done already (like that topology change), I would do what you say... recommission the network from scratch, one item at a time, allowing time in between for the issue to resurface. Don't add back your entire network until you are convinced that you have the basics running and healthy, a stable foundation.

If Asuswrt-Merlin is not required, use current Asuswrt (not beta), and at least use Asuswrt on the node(s) as is recommended. Hard Reset the router firmware to its default settings and configure from scratch:

Reset FAQ
Reset button/webUI Restore/node removal - clears settings in NVRAM; reboot restores fw defaults from CFE (fw defaults)
Hard Reset via WPS button/webUI Restore+Initialize - also clears data logged in /jffs partition (fw defaults+clear logs)

Watch for damaged/chewed Ethernet cables. And leave any router USB device out of the picture until last... I believe too much 'filing' can overwhelm/crash some routers. And be sure your AC power voltage is not sagging.... a UPS always helps here.

OE

 

[IcecreamVille]

Thank you OE for all your suggestions. System has been running stable since my last post - no reboots at all.

Figured I'd give another go at trying to add the RT-AC86U to my AiMesh - as I mentioned prior, I have another RT-AC86U in my AiMesh setup.

Prior to going "nuclear", I went thru all my settings again - when I originally setup the network, I chose one of the lower channels (36 or 42) & enabled the 160Mhz.

Once I had everything setup, checking my AiMesh signal quality, etc, it was much lower than it was prior to my setting up the RT-AX86U.

So I changed my channel to what the setup was prior to setting the RT-AX86U as the main router. I neglected, however, to untick the 160Mhz.

I then setup the RT-AC86U that is wreaking havoc with my network as a separate network, and did a factory reset thru the GUI, choosing the option to delete all data from logs, etc.

Hooked Ethernet cable up to WAN port on the RT-AC86U & hard-wired to my main node (RT-AX86U). After I added the node to my network, the RT-AX86U rebooted itself. But after the initial reboot, it didn't go into the reboot loop.

I then manually removed the RT-AC86U from my AiMesh network, using the "-" button in AiMesh, figuring it might clean-up any possible corruption - and then added it to the AiMesh again. Same as before, as soon as it added, the main node (RT-AX86U) rebooted once.

I left the system up in that fashion for around 20 minutes I then pulled the Ethernet cable from the RT-AX86U & then the RT-AC86U, and it switched to the 5Ghz backhaul. Left it up for another 10 minutes, no reboot. I then power-cycled the RT-AC86U. Once it came up, the RT-AX86U rebooted itself, and then seemed ok. Left it up for around an hour - no noticeable issues - until I changed the name of the RT-AC86U from "Home" (the default) to "Basement". Then the reboot loop started...

The RT-AC86U (2022 model) that is giving me issues has a small-form factor power supply. My other RT-AC86U (2018 model) (and RT-AC68U units) have a larger-form factor power supply. My main router had been an RT-AC86U, which just "burned out" (a few weeks ago, it just died) had the larger-form PSU. I checked the PSU specs of the two power supplies & they were identical (both ASUS brand). So I decided to try putting the larger form-factor PSU on the RT-AC86U that is causing my main node (RT-AX86U) to reboot. Once it booted, I was hopeful because the main node didn't reboot. However, after around 5 minutes, it did end up rebooting. So I pulled it from the network & things have been fine so far.

My next step is to swap the PSU from my other RT-AX86U to my main node, in the off-chance there is a voltage issue with the PSU on the main node. But because the system is stable without this particular RT-AC86U on the network, I really doubt this is the issue.

I'm then going to try swapping my RT-AX86U node that I'm planning on taking the PS from with my main node & restoring my config, to see if the issue follows the hardware.

I will then go "nuclear" & reconfigure from scratch (on stock firmware) adding a node a day (or so) - but as I mentioned, this was a brand new setup, with no firmware updates done after the setup, so I'm not holding out hope that this will fix it.

If that doesn't work, I may try opening an RMA with ASUS for the RT-AC86U that is causing the issue.

I just thought I'd post a follow-up & will continue to update after going "nuclear" (it may be a few weeks before I'm able to do this, though).

 

[IcecreamVille]

Another follow-up, in case anyone is interested.

I was under the assumption that this was likely an AiMesh-related issue.

I figured for the heck of it, I'd try setting up the RT-AC86U that causes the RT-AX86U to reboot, as a media bridge, to see if the problem only occurred when it was an AiMesh node.

I reset the RT-AC86U, re-downloaded firmware (in case the firmware was somehow corrupt - the CRC matched, but it may have gotten corrupted when uploading), & re-flashed the RT-AC86U, factory reset from the GUI & the setup the RT-AC86U as a Media Bridge.

Around a minute after I did this, I hear my wife yell from downstairs "The internet is down!"... so I quickly unplugged the RT-AC86U. Everything returned to normal...

I may try to exchange the RT-AC86U (I need to find the receipt & hope I'm within the return window), and if not, will likely open an RMA with ASUS (it is a 2022 model, so it is under warranty). If I do decide to go the RMA route, before I open it, I'm going to keep an eye on the network for the next week or so to assure there are no other "random reboots" that occur while this is not on the network.

 

[IcecreamVille]

Well...

I exchanged the RT-AC86U for another RT-AC86U. The one that I brought back was a 2022 model - the one I received in exchange was a 2021 model, so there was no chance they came from the same batch. Took it out of the shrink wrap & hooked it up to my primary router (RT-AX86U) - didn't do anything to the RT-AC86U prior to plugging it into the RT-AX86U (didn't do a factory reset, update firmware, etc - literally just out-of-the-box).

Exact same symptoms...

I will need to "go nuclear" on the main RT-AX86U or take the RT-AX86U that I have as a node, factory reset that, and try that as the primary router. Need to find time when no one is home for a few hours to attempt this.

The whole thing is very bizarre - it has been 3 days since my last post, and prior to trying to add the replacement RT-AC86U to my system, 3 days of system uptime.

I'll continue to post progress/results & hopefully find the underlying cause...

 

[IcecreamVille]

My apologies if I'm posting too much about this - I'm just trying to thoroughly document everything that is being done in efforts to isolate the issue, and more importantly, to help anyone else who may be experiencing similar issues.

Earlier, I had mentioned that to help eliminate the self-powered USB device, I had unplugged it when & then tried to add the RT-AC86U. What I didn't do, though, was turn off the media server. So I turned off the media server, unplugged the USB disk (which caused memory usage of the RT-AX86U to drop drastically), and tried adding the node. Same result. Left media server off for now.

I then tried turning off AiProtection - "Malicious Site Blocking" was already off, but "Two-Way IPS" & "Infected Device Prevention and Blocking" was on. So turned off AiProtection altogether. Same result. So turned AiProtection back on.

Something else that I neglected to mention in earlier posts - I let DHCP handle assigning addresses in most cases. But for specific devices (ie. printers, outdoor cameras, Media Bridges, etc), I add a reservation. I do the same for all my AiMesh nodes - I like to know what their IP address will be, so I can ping them every now & then to assure they are running, without needing to go into the router.

I then started going thru the whole situation again, thinking to exactly when the issue first surfaced. I had mentioned it was when I changed the topology of an RT-AC86U to point to the main router. I also changed the topology all-around to point to the main router. (My backhaul presently is wireless - at some point, I would like to hard-wire as many nodes as possible, but presently not possible).

And one of the names of the locations of an AiMesh node was "Custom", where I had typed in a name. I reverted this back to one of the preset values.

I then tried adding the RT-AC86U back into the AiMesh - added fine. Powered off the node, temporarily moved it to the next room & powered it back on. Node got recognized within a minute of powering on or so. System presently up & running for 45 minutes.

Going to leave it this way for a few days to assure network stability & no reboots.

At which point, I may attempt to remove the node from the AiMesh & setup as a media bridge (as I had previously reported that the issue also happened with that - but never tried on replacement RT-AC86U). And then I'll turn the Media Server back on & plug the USB drive back in.

So it is very possible that the issue pertained to forcing a specific network topology within the AiMesh, and then for whatever reason, when adding another RT-AC86U to the network (as either an AiMesh node or Media Bridge), it caused main RT-AX86U to crash & go into reboot loop.

I may still "go nuclear" at some point - but as has been referenced in other threads - don't break a working network. If in fact the topology change in the AiMesh was causing the issue & the network is running fine, without rebooting, then I'm considering the issue resolved & my network working properly.

I will post back in a few days - sooner if issues still arise.

 

[OzarkEdge]

My apologies if I'm posting too much about this - I'm just trying to thoroughly document everything that is being done in efforts to isolate the issue, and more importantly, to help anyone else who may be experiencing similar issues.

No problem with thinking it through outloud. Once you resolve the issue, review this thread to maybe discover and learn from any missed turns.

OE

 

[IcecreamVille]

Another update...

System running for ~3 hours no issues.

I have an AiMesh node (RT-AC1900P) in the yard (which I technically do not need anymore, because the Wi-Fi coverage in the yard seems to be pretty good, since I put in the RT-AX86U units.

My wife was on a very important call in the yard & she was saying to the person on the other end that the internet is running very slow (which it would be, since the yard was now associated with another node in the AiMesh & not the home router). So I pulled the plug on the yard temporarily.

She was able to resume what she needed to do.

Once she was off the call, I plugged the Yard back in & logged into the main router & figured I'd try just associating that one node with the main router... as soon as I did that, the reboot loop began again.

So I pulled that node (as I mentioned, I likely don't need it for the rest of the season)... so I'm getting more convinced that the issue is somehow topology related within the AiMesh setup (overriding the "Auto" in the AiMesh -> <node> -> Management -> Preferrred Wifi Uplink AP).

If I need to use the yard node at some point this season, I'll try running an Ethernet cable thru a window & link it to the nearest node.

Will keep you posted

 

[IcecreamVille]

What I'm writing may be met with criticism - even though I'm a new contributor here, I've been following the forums for years, when I first bought my first RT-AC68P in 2015 - and have been using @RMerlin firmware since I switched to ASUS routers.

When I attempted to add my "Yard" RT-AC1900P back to my AiMesh, I was met with the same symptoms - continuous reboots of the main RT-AX86U router. Turning off he RT-AC1900P stopped the reboot loop.

I then started taking screenshots & documenting all my (important) changes, knowing that going "nuclear" was inevitable.

At the same time, I removed a guest network on the 2.4Ghz network that I'm no longer using - as I suspected, this had no impact on the issue.

I then came across this thread - and figured that enabling IPv6 in Passthru mode would be worth a shot. Doing so resulted in a reboot when adding the AC-1900P back to the AiMesh - but not continuous. When I then tried to power on the node when it wasn't directly connected to the RT-AX86U, the reboot loop started again - so this wasn't my issue.

Figuring I had nothing else to lose (and because I wouldn't be able to go nuclear for a few days), because of that thread & other threads that mentioned issues with 386.7_2, I figured I had nothing to lose by downgrading to 386.5_2 & just see what happens... I figured the issue would persist (and possibly be worse, because of possible configuration corruption) - to my surprise, that seemed to resolve the issue. The network has been running stable for almost 24-hours, GUI interface seems "zippier", and there have been no network hiccups, reboots, or household complaints.

I then downgraded my RT-AX86U node from 386.7_2 to 386.5_2 - to have both RT-AX86U units on the same firmware - so any potential hiccups can not be blamed on the RT-RX86U node running 386.7_2.

If the system stays up & running - and stable - for a week, I'll then add the media server back in. I'm planning on travelling on business the following week, so that'll give me close to a week to monitor things after adding the media server back. Don't want the wife calling when I'm away saying there is an issue with the Internet...

After I get back, I may start playing the AiMesh topology - since this was when the issue first reared its ugly head. And I also may still "go nuclear", just to clear out anything in the config that may have become corrupted with 386.7_2.

Considering other threads on RT-AX86U reboots & other issues on 386.7_2, I think there is something in there that is causing issues & is affecting different users in different ways. Mine just happened to crop up in the AiMesh area.

Also - just wondering exactly what is stored in the JFFS as far as network setup. Outside of the media server, I have a fairly "vanilla" setup - I presently am not using OpenVPN, scripts, or other functionality. I'm considering checking the box to "Format JFFS partition on next boot" to clean things up a bit without "going nuclear".

If problems do arise, I definitely will "go nuclear" - just don't know if I will do so on the 386.5_2 or official ASUS firmware. But as things stand now, I'm happy with the results of downgrading to 386.5_2. And my network appears to be working as expected.

Thanks to everyone who replied, gave suggestions, etc. I'm someone whose real-world job is in IT & try to determine underlying causes. "Going nuclear" without attempting a downgrade goes against my troubleshooting instincts - and figured that I at least needed to try it.

I'll post back as soon as things become unstable - or in a week or so if things still seem to be running as expected.

This also helped assure me that my issue isn't because of faulty hardware. Hopefully it'll help others.

Thanks

 

[IcecreamVille]

Just a follow-up.

After around 5 or 6 days, I plugged the USB drive back in & turned the media server back on.

Ran fine for a week. Then I went away. Got back a few days ago & just checked the system uptime. Over 21 days no reboots. See attached

So something in 386.7_2 (or perhaps even in 386.7 - didn't downgrade to that) was causing my issue - 386.5_2 appears to be rock-solid.