Possible solution to AiMesh wired backhaul network instability

[Krusty]

I have an AC5300 and 3x AC86U (only two plugged in atm because 3 was unstable, but going to try 3 again shortly with this new possible fix).

I have about 50 bulbs and other home automation/security devices on 2.4Ghz band. These will somewhat randomly drop on/off wifi - sometimes everything works well, sometimes a large part of the network goes south.

When the network is unstable, HomeKit really has issues (devices stuck "updating", devices which can report status but not be controlled or vice versa).

I myself have roaming off (along with MIMO, fairness and universal beamforming) on 2.4GHz and I have backhaul set to "ethernet", but have seen reports of (and seen myself when I had roaming on), roamast considering or actually disconnecting AiMesh nodes off WiFi.

Whilst playing around, I noticed things were much more stable if I used "wl down" to bring down the 2.4Ghz interface on the AiMesh main router. This got me thinking:

1) Perhaps there was still WiFi traffic going from the nodes to the router
2) Perhaps this is a routing bug in AiMesh with a wired connection (especially I can imagine that the network setup requiring WAN->LAN links between routers, might be unhappy if traffic is taking another path which doesn't perhaps honor whatever network partitioning they have implemented).
3) HomeKit could plausibly act exactly the way it was if traffic on some of its (presumably more persistent) secure tunnels was getting lost in one direction or the other. It might believe the connections were good but be unable to communicate.

Anyway, I am continuing to see how this works out. Based on my guess, I have allowed the wireless interface back up, but have simply added the AiMesh nodes' 2.4GHz MAC addresses to the MAC filters for 2.4Ghz on the GUI, since there is no reason for any 2.4Ghz traffic between any nodes or the router in this setup.

Fingers crossed... please report back if you give this a try.

P.S. if you bring the 2.4GHz interface down on the main router, the device list in the GUI is more useless than ever, just randomly changing every few seconds. Clearly they have a bit of a state merge bug there - though likely GUI only, since assoclists around the network are stable.

 

[Krusty]

P.S. if you bring the 2.4GHz interface down on the main router, the device list in the GUI is more useless than ever, just randomly changing every few seconds. Clearly they have a bit of a state merge bug there - though likely GUI only, since assoclists around the network are stable.

Actually I guess the network map client list is just plain broken anyway.

 

[Grisu]

did you try last 45149 firmwares?

 

[Krusty]

did you try last 45149 firmwares?

Ah hah, I hadn't yet. Thanks for pointing out the new firmware! Will report back

 

[Krusty]

Ah hah, I hadn't yet. Thanks for pointing out the new firmware! Will report back

I upgraded the firmware, but left the MAC filtering in place... all was OK - a day later, I turned off the MAC filtering, and the problems came back. Therefore it seems this workaround is still needed. Next thing to try is to see if the MAC filter fixes the problem with my currently disabled (because it breaks the network) 3rd node.

Note, on reflection, that there is no particular reason that you shouldn't have to filter out the 5Ghz MACs also. Again perhaps this is the issue with this particular problematic 3rd node - it is close enough to the main router for a good 5Ghz signal.

 

[Krusty]

I upgraded the firmware, but left the MAC filtering in place... all was OK - a day later, I turned off the MAC filtering, and the problems came back. Therefore it seems this workaround is still needed. Next thing to try is to see if the MAC filter fixes the problem with my currently disabled (because it breaks the network) 3rd node.

Note, on reflection, that there is no particular reason that you shouldn't have to filter out the 5Ghz MACs also. Again perhaps this is the issue with this particular problematic 3rd node - it is close enough to the main router for a good 5Ghz signal.

Actually I'm not 100% for sure that MAC filtering is required, it seems that whenever I change MAC filters in any way, i may need to let/help the network sort itself out again. I will risk again in the future trying it without MAC filtering, and giving it more of a chance to right itself afterwards. In the meanwhile, if anyone else tries this either way, please LMK.

 

[Krusty]

Well it is Boxing day, the traditional day of rebuilding the house wireless networks after xmas. As such I have risked all, and added the 3rd AiMesh node back. So far so good (fingers crossed).

I did make one significant change which was to adjust wiring so that all 3 nodes are now connected via a single wired hub (to which they and the router are the only connections) all other wired connections on the network go thru other hubs and into other ports on the router. Not obviously clear why this should make a difference, but it is one less thing.

 

[DummyPLUG]

Guest what, I had use the same work around but with both 2.4G and 5G filtered on the 68U

 

[Krusty]

Guest what, I had use the same work around but with both 2.4G and 5G filtered on the 68U

Interesting... yeah it seems like you should have to filter both. Just to be clear, you are filtering which MACs where? I guess from the GUI is fine.

 

[DummyPLUG]

Interesting... yeah it seems like you should have to filter both. Just to be clear, you are filtering which MACs where? I guess from the GUI is fine.

I tried 1. just the node 2. all the wifi's mac in the web GUI, it seems no different between them. I am using the 68U as AP with a CCR1009 as router, I start to block the wifi's mac because I saw loopback warning for the node. I didn't try the latest firmware yet

 

[OzarkEdge]

Well it is Boxing day, the traditional day of rebuilding the house wireless networks after xmas. As such I have risked all, and added the 3rd AiMesh node back. So far so good (fingers crossed).

I did make one significant change which was to adjust wiring so that all 3 nodes are now connected via a single wired hub (to which they and the router are the only connections) all other wired connections on the network go thru other hubs and into other ports on the router. Not obviously clear why this should make a difference, but it is one less thing.

So, are you currently not filtering node MACs?

Have you always routed your wired backhauls through your many hubs?

Have you ever tried directly wired backhauls to eliminate your hubs to see how your AiMesh then behaves?

OE

 

[DummyPLUG]

So, are you currently not filtering node MACs?

Have you always routed your wired backhauls through your many hubs?

Have you ever tried directly wired backhauls to eliminate your hubs to see how your AiMesh then behaves?

OE

No No No, as of 384_32799 the MAC list will sync to the node, so all I need is put all the 68U 2.4G and 5G MACs in the web GUI then it will sync them to the node and seems prevent each node connect to others by wifi.
I always use wired backhaul through 3~4 switch, some are cisco, some are HP, and only 1 68U on each swtich, with nothing else connect to the 68U ethernet port (that means the 68U only connect to those swtich).
I can't directly connect them right now as they are in different floor, but I did try just use the 68U to setup a netowrk for testing when AImesh go offical, they at least work for a few hours but I didn't let them keep running more then that.

 

[OzarkEdge]

No No No, as of 384_32799 the MAC list will sync to the node, so all I need is put all the 68U 2.4G and 5G MACs in the web GUI then it will sync them to the node and seems prevent each node connect to others by wifi.
I always use wired backhaul through 3~4 switch, some are cisco, some are HP, and only 1 68U on each swtich, with nothing else connect to the 68U ethernet port (that means the 68U only connect to those swtich).
I can't directly connect them right now as they are in different floor, but I did try just use the 68U to setup a netowrk for testing when AImesh go offical, they at least work for a few hours but I didn't let them keep running more then that.

In the spirit of 'questioning everything', I'm skeptical of this need to block node-to-node WiFi to force AiMesh to respect wired backhauls to make AiMesh work. It just seems too simple a defect on ASUS's part.

A common factor here appears to be wired backhauls through additional equipment. Maybe Aimesh is seeing a wired backhaul failure and is trying to self-heal over a very poor wireless backhaul, which then breaks client communications, with lessor clients then falling down, not knowing how to get back up again.

We already know wired backhauls have trouble auto-sensing with some intervening equipment. Did you have to force AiMesh backhaul type to Ethernet, or did it auto sense?

OE

 

[DummyPLUG]

In the spirit of 'questioning everything', I'm skeptical of this need to block node-to-node WiFi to force AiMesh to respect wired backhauls to make AiMesh work. It just seems too simple a defect on ASUS's part.

A common factor here appears to be wired backhauls through additional equipment. Maybe Aimesh is seeing a wired backhaul failure and is trying to self-heal over a very poor wireless backhaul, which then breaks client communications, with lessor clients then falling down, not knowing how to get back up again.

We already know wired backhauls have trouble auto-sensing with some intervening equipment. Did you have to force AiMesh backhaul type to Ethernet, or did it auto sense?

OE

I always forced to ethernet but I see some loopback warning if I didn't block the node to node wifi,

 

[OzarkEdge]

I always forced to ethernet but I see some loopback warning if I didn't block the node to node wifi,

Sounds like AiMesh may have trouble through your wired equipment. Your layout may not permit it, but a test could be a purely wired backhaul.

OE

 

[DummyPLUG]

Sounds like AiMesh may have trouble through your wired equipment. Your layout may not permit it, but a test could be a purely wired backhaul.

OE

It will be great if there is a list dos and don'ts, or how aimesh decide go for wifi or ethernet. I had tried different combination of setup, also try setup the switch as a dump switch. Block the node to node wifi is the last resort, and at least it work, and I am not going to replace those switch (Actually I hate to test each SFP+ again and again).

 

[OzarkEdge]

It will be great if there is a list dos and don'ts, or how aimesh decide go for wifi or ethernet. I had tried different combination of setup, also try setup the switch as a dump switch. Block the node to node wifi is the last resort, and at least it work, and I am not going to replace those switch (Actually I hate to test each SFP+ again and again).

Maybe your equipment is too enterprise class for AiMesh.

OE

 

[DummyPLUG]

Maybe your equipment is too enterprise class for AiMesh.

OE

LOL

 

[Krusty]

So, are you currently not filtering node MACs?'

I am not currently MAC filtering. I have 2.4GHz down on the main router (which is roughly equivalent to the MAC filter on that freq), but that is more because many of my bulbs seem to want to connect there over the nodes (which have better signal) instead of network stability issues.

It may be that the latest firmware has helped - if anyone in the know can say whether anything to do with traffic leaking over wireless (or bad route setup/whatever) was fixed that'd be helpful - my network could sometimes be stable for a while (and also unstable for a while even with a good change) so it takes me a long time to make and assess small changes one at a time.

Have you always routed your wired backhauls through your many hubs?

As of AiMesh up until now, yes, I had just gone thru the existing wired networking. This has about 12 wired spokes (with the nodes at the end of 3 of them, other devices and possible wired hubs elsewhere). Because I didn't have a 12+ port switch, I just used a couple of 6/8 ports I had lying around. I have consolidated the 3 wire spokes onto their own single 4 port hub which connects to the main router.

Have you ever tried directly wired backhauls to eliminate your hubs to see how your AiMesh then behaves?

Thought about this, but had some network things running that I didn't want to stop; i figured all on one hub, and one wire from that would be an improvement and a good compromise.

 

[Krusty]

In the spirit of 'questioning everything', I'm skeptical of this need to block node-to-node WiFi to force AiMesh to respect wired backhauls to make AiMesh work. It just seems too simple a defect on ASUS's part.

A common factor here appears to be wired backhauls through additional equipment. Maybe Aimesh is seeing a wired backhaul failure and is trying to self-heal over a very poor wireless backhaul, which then breaks client communications, with lessor clients then falling down, not knowing how to get back up again.

We already know wired backhauls have trouble auto-sensing with some intervening equipment. Did you have to force AiMesh backhaul type to Ethernet, or did it auto sense?

OE

I actually have it forced to ethernet (this may in fact be part of the problem if there is/was (prior to the latest firmware) a bug in the routeset up)... though it was auto-detecting ethernet anyway.

My guess (I'm not well versed enough to fully evaluate the routing rules to spot errors) was that some non backhaul traffic was going over wifi by mistake in ethernet only mode. And was thus causing all kinds of WAN/LAN confusion.