AiMesh unstable with third ethernet-backhaul RT-AC68U node

[Krusty]

I have an RT-AC5300 as the AiMesh router, and then multiple RT-AC68U AiMesh nodes.

F/W: 3.0.0.4.384_32799
All nodes were factory reset and installed per instructions, with wired backhaul
the 2.4Ghz band has Roaming Assistant, Airtime Fairness, Multi-User MIMO, Universal Beamforming off

Nearly all of my 2.4Ghz wireless devices are bulbs/locks/switches/cameras etc that tend to be stationary. Most other useful (TV, AppleTV etc, desktops, printers) things are hardwired anyway, with the two 5Ghz bands intended for laptop, phones etc to roam in the AiMesh.

I have been having ongoing issues with HomeKit (!) and LIFX bulbs (and other smart devices) on the network. things had become unusable (even taking HomeKit out of the mix, some bulbs were not able to connect). I also have occasionally had wired network dropouts.

After some recent playing around with the WiFi settings, trying out different Roaming RSSI levels etc, with wholly unpredictable results, I decided to start back with the Main Router only (which I power-cycled). Things were OK, but not enough signal for remote cameras. I slowly added back (turned power on) the other nodes... Things were stable with 2 nodes.

After adding the third one, pretty much all devices (at least according to the router GUI) dropped off their corresponding nodes... and didn't reappear on the main one. Some would pop in and out.

This may be an issue with the third node itself (I will try a complete reset/re-add of that node after I've seen how long the network is stable as is), but this behavior is a bit similar to what I saw when I accidentally connected a node via LAN not WAN port during my initial setup...

I recall seeing something (that I can't find now) from Asus about a maximum number of wired nodes, or how many could be in a line or something. My topology is all AiMesh nodes as siblings connected via their WAN ports (and then some 1GHz unmanaged switches) to LAN ports of the main router.

Wondering if anyone has seen something similar... whether it is topology related, AC68U related or what, or if perhaps LAN switch topology is a factor, though that seems unlikely.

 

[wesbez]

Just a quick one. Are you using standard FW? have you downloaded the app. It’ll show you what’s connected to the AImesh and can be used to set up the mesh automatically if you decided to redo it. It’s also good for a quick visual.

I see you are also using cameras. How is the CPU holding up with the IP cameras and the two nodes? Is it maxing out causing drops.

 

[Krusty]

Just a quick one. Are you using standard FW? have you downloaded the app. It’ll show you what’s connected to the AImesh and can be used to set up the mesh automatically if you decided to redo it. It’s also good for a quick visual.

I see you are also using cameras. How is the CPU holding up with the IP cameras and the two nodes? Is it maxing out causing drops.

I am using the standard firmware. Only SD on the cameras, and the image is pretty constant... very little CPU usage on the AC-5300.

I wish the GUI gave any sort of half decent status on the nodes, since they redirect you from the GUI... I will try logging into the nodes to get CPU etc and report back. Any useful logs I should check on the nodes?

Note... I am seeing occasional drops with the two nodes, so that's why I'm looking at that. Jury is still out on why the 3rd node hoses everything completely.

 

[HuskyHerder]

Running 4 (the 5th and final, will be added this weekend) nodes now, no issue (knocks wood). However I did have issues when I had purchased a couple of refurb units from new egg. I sent them back and have been purchasing used/new ones as I get a good deal. So far so good.
Those referb units would take down my net, they would crash and die slowly till rebooted. Then they would work again for a day or two.

Home kit is still an issue, around here as well, not so much for my iDevice plugs etc, but it poses an issue with my hue system. Home kits comes and goes in spurts. Ill have a good day and then it goes wonky. I think it's the hue system and not really home kit, I plan on changing its control frequency this weekend.

As for my setup the 5300 is ran to 2 switches that feed/supply the node, they are 1GB Netgear unmanaged as well. Basically my 5300 is the hub and each node is a spoke.

but this behavior is a bit similar to what I saw when I accidentally connected a node via LAN not WAN port during my initial setup...

I really think you have hit on a great point here. I had much the same results till I learned to always move it to WAN, this last setup I did was pretty flawless and I always remembered the cable. This is also the best my system has ever ran. EVER. I really think something happens when the cable gets left in the wrong spot. I almost thought it a non issue till I see you mentioned it.

I setup everything from scratch this past go round, nodes, router etc, and have been adding nodes as I acquire them with no issues.

I have gone to rebooting my whole system once, I add a node and it seems to also help. Could just be me but it seems beneficial to me.

 

[Krusty]

Running 4 (the 5th will be added this weekend) nodes now, no issue (knocks wood). However I did have issues when I had purchased a couple of refurb units from new egg. I sent them back and have been purchasing used/new ones as I get a good deal. So far so good.
Those referb units would take down my net, they would crash and die slowly till rebooted. Then they would work again for a day or two.

Home kit is still an issue, around here as well, not so much for my iDevice plugs etc, but it poses an issue with my hue system. Home kits comes and goes in spurts. Ill have a good day and then it goes wonky. I think it's the hue system and not really home kit, I plan on changing its control frequency this weekend.

As for my setup the 5300 is ran to 2 switches that feed/supply the node, they are 1GB Netgear unmanaged as well. Basically my 5300 is the hub and each node is a spoke.




I really think you have hit on a great point here. I had much the same results till I learned to always move it to WAN, this last setup I did was pretty flawless and I always remembered the cable. This is also the best my system has ever ran. EVER. I really think something happens when the cable gets left in the wrong spot. I almost thought it a non issue till I see you mentioned it.

Excellent ... thanks for the info... my devices are also hub/spoke, though each spokes may join at an (perhaps different) un-managed hub before reaching LAN port of 5300.
With two nodes, HomeKit seems pretty stable (before it was getting stuck indefinitely whilst bulbs themselves were working via UDP over wireless - given that HomeKit needs tunnels to the devices which presumably stick around for a bit being expensive-ish to set up, I am totally guessing that perhaps the wired network weirdness was leaving IP not actually functioning, but not necessarily appearing broken from either end).

I haven't had a chance to try, but will try 2 different nodes (disconnect A, keep B, reconnect C) before I reset C to see if the number is important.

 

[HuskyHerder]

Your also in the AC section of the site, you may wish to also keep an eye on the https://www.snbforums.com/forums/asuswrt-official.51/ section as a few of us have been chatting over there about Mesh.

 

[Krusty]

.. I will try logging into the nodes to get CPU etc and report back. Any useful logs I should check on the nodes?...

I logged into the 2 nodes + router and there weren't any CPU problems there (at least with the network functioning). If I have problems when I add the third back in, I'll take a look again as the world burns.

 

[Krusty]

Your also in the AC section of the site, you may wish to also keep an eye on the https://www.snbforums.com/forums/asuswrt-official.51/ section as a few of us have been chatting over there about Mesh.

I am checking it out, will jump over there for new questions...

@HuskyHerder since you have a functioning system, a couple of quick questions/notes

1) The web GUI list of clients is very flaky for me (at least with large numbers of connected clients)... count changes a lot even though the devices aren't really coming/going. certainly I think it can fail to retrieve data from some nodes during a poll, but the "wl assoclist" on each node seems much more stable
2) Speaking of web GUI (and suckiness of current roll up of info from nodes to router), I did confirm that log shown is only for the router node.
3) I turned roaming back on, and it seems to work fine - note when I turned this on again, all clients disconnected and actually did connect to the strongest signal node. The GUI says "network SSID/security has changed", will try to figure out what this action is equivalent to (e.g. if down/up on the interface(s) has the same effect)
4) Interestingly, my network seems better if I turn of the 2.4Ghz on the main router only (via wl down)... wondering what the deal is with the 5300 vs all others being 68U... it may be that the rssi numbers just are differently sacaled for the 5300 such that a better connection on the 5300 looks worse. This presumably screws with roaming.

 

[HuskyHerder]

@Krusty

re:
#1 yeah the GUI has had issues for a long time now. It's a long sought after, item to be fixed as far as users go.
#3 Roaming works wonders for me, well, coupled with turning my power level down to balanced, and tweaking my RSSI levels to my particular environment. It's working pretty well across all nodes. Hand off is pretty seamless for me.
The GUI "changed" page comes up from time to time for me. Sometimes, I see it, an at other times when I expect to see it, I don't. I expect it comes up when you change something in settings and it must be written to the node, and then everything goes down and back up to make sure its all on the same page so to speak. But it also happens when its router only specific changes. But then again I don't know how the code is linked together. It could be 100% necessary for the settings I am monkeying with.

I tweaked a few settings last night, and noticed it killed the wireless on the router and the node closest to me. Very likely the other ones too, but they are in different areas of the home and not easily monitored. I expected the router to change but not the node. But whatever,

I have different RSSI roaming levels for the 2.4 G band vs my 5 G, and as I said I turned down the transmit levels to balanced.

 

[Krusty]

@Krusty


#3 Roaming works wonders for me, well, coupled with turning my power level down to balanced, and tweaking my RSSI levels to my particular environment. It's working pretty well across all nodes. Hand off is pretty seamless for me.

Cool; i'll try balanced (I had already tweaked my RSSI levels)

The GUI "changed" page comes up from time to time for me. Sometimes, I see it, an at other times when I expect to see it, I don't. I expect it comes up when you change something in settings and it must be written to the node, and then everything goes down and back up to make sure its all on the same page so to speak. But it also happens when its router only specific changes. But then again I don't know how the code is linked together. It could be 100% necessary for the settings I am monkeying with.

Yeah, I'm gonna dig around a little in the code to see what triggers this - it seems to have a better chance of forcing clients onto new nodes

I tweaked a few settings last night, and noticed it killed the wireless on the router and the node closest to me. Very likely the other ones too, but they are in different areas of the home and not easily monitored. I expected the router to change but not the node. But whatever,

Agreed; I'll try to hack together some scripts to run on the router (probably will require adding a ssh key to autologin to the nodes) to get aimesh information in a more handy form...

 

[Krusty]

Note, that I tried roaming on (and balanced)... the former definitely causes HomeKit issues for me (devices that are not reachable according to HomeKit)