ASUS AiMesh - Have I exceeded max clients for 6 Nodes? Lots of unresponsive clients.

[Techlifter]

Long story short, we have a lot of smart home and wireless devices (~165) and I’ve been struggling with wireless clients (usually lights or speakers) dropping out and becoming unresponsive for long periods (45min-2hrs), only to reconnect on their own if I haven’t physically intervened by restarting the routers for an immediate fix.

I’ve paid two network professionals to come in over the last 18 months to try to troubleshoot/resolve the issue with no improvement, but neither had personal familiarity with ASUS AiMesh.

All 6 of the mesh nodes we have in service are connected via Cat5e Ethernet-only backhaul (3 story house was wired by a non-profit that owned/operated from it for the last 20+ years) but I'll try to share all of the relevant basics.

AiMesh (Ethernet backhaul):

• 4 x AX92u
• 1 x AC1900
• 1 x AC5300
• SmartConnect turned on (default config)

~140-175 network clients throughout the year

• fluctuates as we add 20-25 additional wifi smart plugs during the holidays to control decorations.
• Most are IoT devices, connecting to the 2.4Ghz
  • 90 Wiz smart lights (no hub)
  • 32 Hue smart bulbs (hub)
  • Appliances (fridge, oven and hood)
  • 10-30 smart plugs (seasonal)
  • 7 Lorex WiFi cameras
  • 7 Echo devices (full size and 2nd Gen echo dots)
• The rest are 5Ghz devices
  • 3 Mac laptops
  • 3 iPhones
  • 1 iPad
  • 1 PC laptop
  • 16 Sonos speakers & soundbars (various models)
  • 5 Samsung smart TVs
• And finally, the hardwired (Cat5e Ethernet) components
  • 2 HTPCs
  • 2 NAS boxes
  • 1 managed switch (TP-Link SG1024DE)
  • 2 unmanaged switches (TP-Link SG1024)

As far as usage, there are just two people in the house who rarely stream video separately but we have music streaming (semi-successfully) on the Sonos speakers during all waking hours. All of the lights are on schedules and we have several routines for controlling the the TVs and appliances. Naturally, as things drop off, those routines become more frustrating to use because some clients simply don't respond.

Maybe we have something misconfigured? Perhaps we’ve reached the limit of consumer grade hardware and need to jump to Ubiquity or something else? We generally sit at <35% CPU and ~76% RAM (screenshots attached) for resource utilization. Memory usage does tend to creep up into the low to mid 90% over time, but returns to 76% after a restart of the mesh network. Once everything settles, we tend to get about 25-35 clients distributed per node and have moved the nodes around to try and avoid a single unit shouldering all of the burden.

I've shared screenshots of the ASUS Router app on my phone to give a general idea of what we're seeing, but I can share more details if I know what's relevant.

If this is a situation where ASUS AiMesh just can't handle that many clients, I'd welcome being pointed in the right direction towards hardware than can do so in such a way that we don't have to think about it and could potentially add more.

Link to Screenshots:

https://imgur.com/a/Dy75qMS

 

[Techlifter]

I forgot to add that we're on the latest firmware (3.0.0.4.388_22525) and not running any of the AiProtection services which tend to push the RAM usage even higher.

 

[Tech9]

I’ve paid two network professionals to come in over the last 18 months to try to troubleshoot/resolve the issue with no improvement, but neither had personal familiarity with ASUS AiMesh.

No wonder. No professional will take your approach. I personally call a setup like this AiMess. Tri-band home routers with wired backhaul (waste of AIO routers used as APs only), some >10 years old in tech (AC1900), perhaps way too many, SC enabled and not supported by all (AC1900), perhaps IoTs mixed with main devices on the same network... my only sure fix will be everything out and replaced by the number of needed only business class APs with PoE, a new switch with PoE and the number of ports needed with some extra for expansion, firewall with VLANs and proper network segmentation for IoTs, cameras, NAS boxes, main devices, guest network, etc. I'm sure you are going to receive AiMesh optimization ideas, but... good luck with it.

 

[heysoundude]

I'd wager you're running into DHCP addressing issues. You can only get 255 addresses on 192.168.1.x, and you may be at or above the actual limit for the router you have (which often ISN'T 255 in reality). (This is why IPv6 was invented...but we'll save that for another discussion)
There's also the possibility that you're saturating your Internet connection with all that streaming potential of client devices, not to mention the "smart" devices calling home.
If you want to stay with Asus stuff (which you have an investment in already), I'd recommend upgrading the AC routers to at least one of the GT-AX routers, because that can get you into 2.5Gbps wired infrastructure. Further, I would strongly suggest you flash Merlin to it, and use some of the scripts. (Do some research on the Asuswrt-Merlin forum, and dig into the Add-ons forum too BEFORE you buy a new GT-AX router to make sure it's supported)

 

[Tech9]

I'd recommend upgrading the AC routers to at least one of the GT-AX routers

This is going to be even more money thrown into already wrong approach. A new router with 3rd party firmware and custom scripts won't solve the Wi-Fi mess. If he calls back the professionals they won't even bother to come. Too busy, no time. Tell the professionals what you need and listen to what they say is the right thing to do. They know what's needed. It's not AiMesh with 6 routers.

 

[tgl]

How large is the house physically (you mentioned three stories, but no more detail)? What are the interior walls made of? Six APs is one heckuva lot of wifi, so unless the place is just massive or very wifi-unfriendly, it's likely that the APs are interfering with each other more than helping.

I share @heysoundude's suspicion that you may have more clients than AIMesh is able to handle, too.

As a low-cost test I'd try reducing the number of APs to the absolute minimum that can cover your space with adequate signal (a wifi scanner app can help you with determining that). If it's still flaky then you need to toss the ASUS gear overboard and invest in SMB-grade APs. It'd still be the case that less is more, so this might not be a huge investment. Don't overspend by acquiring a bunch of router-grade units: you just need one router, and the rest should be wifi access points only.

 

[drinkingbird]

I forgot to add that we're on the latest firmware (3.0.0.4.388_22525) and not running any of the AiProtection services which tend to push the RAM usage even higher.

This isn't going to be a CPU or RAM issue. This setup just wasn't intended to scale that far. Too many APs on 2.4Ghz, too many clients (especially the bulbs and plugs which can't handle all the network noise created by that many devices), etc.

It could probably be made to work, but you'll need to reduce the number of APs (or at the very least do some sort of site survey and map out the channels for each AP properly) and find ways to isolate things, i.e. moving the bulbs and plugs over to Guest Wireless 1 with Access Intranet disabled (just bear in mind those devices will not be able to contact anything on the main LAN so you probably need the hub and anything else that needs to communicate with them on that GW also). Isolating stuff is the only way you're going to have any chance of getting it working on your existing setup. I would say what you have warrants 2 or 3 more SSIDs but with AiMesh only GW #1 is propagated to the nodes, so you're basically limited to 2 isolated networks/SSIDs. Even if you got rid of aimesh and configured them all manually GW2 and 3 would not be isolated at the nodes so that won't help.

The only other option would be to run each in router mode, ditch aimesh totally. All your nodes become double NAT'd routers (or you could disable NAT on the nodes and do some static routing but that would get somewhat complex to set up and won't necessarily buy you anything, other than some easier inter-communication between the networks, for say the hub to be able to reach all the bulbs more easily). You still need to map out the channels to try and reduce overlap and interference especially on 2.4, but this will completely distribute both the wifi and routing load amongst all the devices and significantly reduce network chatter. Given how many devices you have, this might actually be the best solution, actually offers quite a bit of flexibility over what can/can't talk to what. You need to assign a unique subnet to each node, and ideally a unique SSID also, though it could be made to work with the same SSID. You would probably need to do some port forwarding for the hub to be able to reach all the bulbs (assuming they aren't all on the same router) and have the hub hanging off the main router, probably a few other things like that to tweak/take into account on this setup too.

But all that being said, if the house is large enough to warrant that many bulbs, I think it is large enough to invest in higher grade gear and a proper site survey and channel setup from a pro. If nothing else, inconspicuous ceiling or flush wall mounted APs running to a router (with separate interface for each AP) hidden in a closet somewhere is much more aesthetically pleasing in what I'm assuming is a pretty nice/expensive house.

 

[Yakkosmurf]

I'm not expert enough to have suggestions that you already haven't gotten. But, I have seen some of your issues on my network with regards to 2.4G smart devices not responding at times or going offline for a while and reconnecting. I only have about 60 total clients on my network, with about half of them wired, and about a dozen smart devices. The smart devices are assigned to Guest Network 1. I have a 4 node AI Mesh system all connected with a wired backhaul.

I used to have the guest network only broadcast from the router that's located in the center of the house. But, as I added smart outlets outside and in parts of the house far from the router, I extended the guest network to the nodes which are each positioned at an extreme corner of my house, with its rectangular footprint. Two things improved my situation and has pretty much eliminated my clients not responding. First, I tweaked the roaming assistant value in the Wireless/Professional screen of the web GUI to -60dbm. This improved things some, but I noticed the day after my weekly reboot, random smart devices wouldn't connect to the network until I rebooted smart device. So, I've eliminated the weekly reboot, and I haven't had a disconnect, or non-responsive smart device in the three weeks since I made this change.

I expect my issue is part of what was mentioned before in that with so many radios transmitting, the 2.4G band is getting saturated and the 2.4G clients are "seeing" too many of the radios at once. My wife and I work from home frequently, and I have some 5G outdoor cameras, which has necessitated more nodes to fix some dead zones with the 5G coverage. I'd be curious to know what your roaming assistant is set to for the 2.4G band and if the disconnects would continue if you didn't reboot your system as often.

 

[heysoundude]

@Yakkosmurf raises some interesting considerations, @Techlifter : Roaming Assistant can be a help, and to further fine tune things, you can adjust the broadcast strength of the radios downward so they don't overlap too much...and if you get deep into tweaking this balance, you've thrown too much AP hardware at your network as @drinkingbird noted.

But still, @Tech9 is absolutely correct - trying to use consumer grade home gear for a network with as much traffic as yours seems to have will eventually have you trying to use a bucket on your own to try to stop Titanic from sinking. (which is why you're here, I think - you have failed at leveraging the magic of internet/wifi yourself to serve your own purposes)

I'd go so far as telling you that you need to look at some home automation to get all those smart bulbs and outlets off your WAN connection - they need a LAN of their own, period.
Same for your video surveillance. (Maybe these are what your AC routers can be used for- subnet? vLAN?) (is the video surveillance going to one of the NAS devices? music/photos/videos/"private cloud" to the other?)
(then you can Mesh the AX routers for your internet/streaming and maybe everything will play nicely together)

whichever way you choose to go, @Techlifter you'll need to roll up your sleeves or open the wallet, but probably some of both.

 

[sfx2000]

This is going to be even more money thrown into already wrong approach. A new router with 3rd party firmware and custom scripts won't solve the Wi-Fi mess. If he calls back the professionals they won't even bother to come. Too busy, no time. Tell the professionals what you need and listen to what they say is the right thing to do. They know what's needed. It's not AiMesh with 6 routers.

Yep - and this calls for a more measured solution...

Organic growth like this happens - we've seen it before, not perhaps to this level, but it does happen.

This is way beyond the scope/capability of AIMesh (or pretty much any other home based Mesh routing platform)

Omada would be a likely good start, but it does also require some advanced planning... but it would be worth the investment time, and cash out, competitive with a 6 mode Asus Mesh platform with much better coverage and performance.

Side Note - for the NAT'ed side of the network, if this hasn't already happened, move the dhcp scope from 192.168.0.0/16 (which on most home networks ends up being a /24 by default) over to something with a bit more - 172.16.0.0/12 which allows for a lot more DHCP assignments - with close to 200 devices, you can start running out of addresses...

 

[drinkingbird]

Yep - and this calls for a more measured solution...

Organic growth like this happens - we've seen it before, not perhaps to this level, but it does happen.

This is way beyond the scope/capability of AIMesh (or pretty much any other home based Mesh routing platform)

Omada would be a likely good start, but it does also require some advanced planning... but it would be worth the investment time, and cash out, competitive with a 6 mode Asus Mesh platform with much better coverage and performance.

Side Note - for the NAT'ed side of the network, if this hasn't already happened, move the dhcp scope from 192.168.0.0/16 (which on most home networks ends up being a /24 by default) over to something with a bit more - 172.16.0.0/12 which allows for a lot more DHCP assignments - with close to 200 devices, you can start running out of addresses...

I actually tried just out of curiosity any my AC68U allows 10.0.0.0/8 and a huge DHCP scope. Whether it will actually assign more than 250 or so IPs, not sure. But either way, a broadcast domain that big is likely going to be an issue no matter what, especially with smart home/IOT stuff.

I'm actually starting to lean toward my idea of converting all the nodes (or maybe 4 of them, seems like they have too many APs) to routers, will provide great segmentation. Can even have an IOT guest network on each router. Though depending on how those bulbs want to interact it may not work, especially if there is a hub involved. Some all want to be on the same SSID and/or subnet, others don't care.

Ideally for IOT devices that don't move around having separate SSIDs you can assign them to (one on each AP/router) can really help things. But again, some won't allow that.

 

[sfx2000]

'm actually starting to lean toward my idea of converting all the nodes (or maybe 4 of them, seems like they have too many APs) to routers, will provide great segmentation. Can even have an IOT guest network on each router. Though depending on how those bulbs want to interact it may not work, especially if there is a hub involved. Some all want to be on the same SSID and/or subnet, others don't care.

I think we are all aligned in that AiMesh is not fit for purpose here...

I don't agree with converting the mesh nodes back to routers and/or AP's either - this network is already running hot, and consumer gear isn't made for this.

Continuing down that path, the WLAN will always be fragile and could fall apart at any moment.

I do agree that with careful planning and segmentation, a forklift upgrade into a prosumer set of gear (see my comment about Omada, also there is the uBiquity gear) is a worthwhile investment that will last for a long time, and having central management will make things easier in the long term.

 

[Tech9]

172.16.0.0/12 which allows for a lot more DHCP assignments

Last time I did some experiments with different than default IP range Asuswrt Client List went South.

I'm actually starting to lean toward my idea of converting all the nodes

All the nodes converted to e-waste or sold on eBay and start over with proper equipment - this is the solution.

 

[drinkingbird]

Last time I did some experiments with different than default IP range Asuswrt Client List went South.



All the nodes converted to e-waste or sold on eBay and start over with proper equipment - this is the solution.

I use 10.0 and client list works fine.

Sure the best solution as I mentioned is higher end gear, but should they want to attempt to use what they have, there are options they can try.

Even with the best equipment out there, without proper isolation those smart bulbs are probably going to have problems regardless.

 

[sfx2000]

There's a lot of upside to a replan/rebuild/replace...

ArsTechnica had a great article (actually series of articles) on Prosumer geat (UniFI in their case) - lessons learned that could apply to OP's situation

What I’ve learned from nearly three years of enterprise Wi-Fi at home

The ups and downs of software-defined networking—and having too many access points.

arstechnica.com

 

[Tech9]

those smart bulbs

I better say nothing on 120 "smart bulbs" and 10-30 "smart plugs" subject.

 

[sfx2000]

I better say nothing on 120 "smart bulbs" and 10-30 "smart plugs" subject.

That is a bit much...

But hey, go big or go home

Would be interesting to get some wireless PCAP's on that WLAN

 

[drinkingbird]

That is a bit much...

But hey, go big or go home

Would be interesting to get some wireless PCAP's on that WLAN

100Mbps of MDNS, ARP, and Broadcast.

 

[Techlifter]

I wasn't sure of the preferred method to respond to questions (respond to each poster in a separate reply or respond to all questions in a single post), so I'll err on the side of brevity (Ha!) and go with the latter.

@Tech9 : Thank you for the assessment and warning about going further down this path. As @sfx2000 suggested, the current state of the network evolved organically over time; we started out with a smaller mesh network and way fewer clients and added over time as the client number(s) grew. Critically, the first network professional we hired said that our ~255 available IPs (minus the routers themselves, of course) was sufficient but that we didn't have enough nodes to support our total clients and that we just needed to add more. So, we did and have just been winging it ever since. I understand that theoretical maxes (clients, throughput, etc.) and reality are not likely to meet but, until now, I was unaware this was a dead end for our purposes.

Based on the general consensus from the group that has responded to my question, it sounds like we've got two things working against us:
1.) We've way too many clients and things going on in our network to be supported by ASUS consumer hardware. Throwing more ASUS nodes at the problem won't help and, in fact, we already have too many and the additional network noise is likely to make things worse.
2.) We should be segmenting the network (VLANs or completely separate networks) to keep IoT and end user devices in their own respective lanes. Right now, having every client clamoring for space under a single (W)LAN is just too chaotic.

That works for me. I have no interest in throwing good money after bad, so if we need to clear everything out and start again, that's just the way it needs to be. Given this expensive lesson and the time lost fighting to understand the network's failings, all I can try to do is make smarter choices from here on out.

@tgl : I'm happy to answer your request for details on the house. My initial post felt pretty long in my efforts to provide a clear picture and I was concerned about going overboard with a TL;DR for my troubles.

The house is 4-sided brick, about 4500sq/ft in total, with a large brick chimney in the center of the building that begins in the basement and opens on 3 fireplaces on 2 floors. Concern about the chimney blocking or otherwise reflecting routers' transmissions was one of the reasons we placed the nodes where we did on each floor. Along those lines, I'm including some screenshots of an exploded rendering of the house and a web link to the full 3D model of the home hosted on Floorplanner. When viewing the 2D floors, you can see the current placement of each of the routers indicated by the traditional Radiation symbol (I couldn't find a WiFi or router icon that had similar visibility). When viewing the 3D version, you can rotate the house around to see rooms from all sides.

https://floorplanner.com/projects/75681588/viewer

Again, we added more nodes to try to shoulder the client burden as well as get around problems of obstruction when we first tried to do this entirely wirelessly (without the ethernet backhaul = disaster). Some of the walls are plaster (built in 1956) and the 1st network professional said the metal lathe was likely causing interference. I know the rendering isn't exactly photo-realitic, but it does a decent job at conveying materials and orientation in a way that would be cumbersome to do in words.

@drinkingbird , you also hit the nail on the head with the headaches that would come with trying to shoehorn in separate networks/SSIDs with the current gear: part of the aim is to be able to control any of the lights/devices in the house by either speaking aloud to the Amazon Echoes or using your phone; access to other clients across the different SSIDs becomes a headache and local access can go entirely out the window if the Internet connectivity goes down. I'm happy to pay a professional to do the work, including upping the hardware game, but our luck hasn't been great in the Atlanta area. Consequently, I've just tried to educate myself, with all the flailing first steps you'd expect.

We have 4-6 Wiz downlights (Ex. https://www.amazon.com/dp/B08PJFR7D8/?tag=snbforums-20) in each room, which is part of the reason there are so many. It doesn't feel like too many overhead lights for a house, but that seems to be the general feeling from the people who have responded to the post.
¯\_(ツ)_/¯

You're also right that I would prefer the gear to be as hidden as is reasonably possible without impacting effectiveness.

@Yakkosmurf / @heysoundude : thank you very much for the suggestions. We've gone so far as to try to manually set channel selections to avoid overlap, where possible. Other houses are generally far enough away that they've been discounted as sources of significant interference. As much as the other responders have said it's hopeless, I can at least give that a try while we wait for another professional to come in and set up new gear from the ground up. As requested, I've included a screenshot of our Roaming Assistant settings. I'm absolutely ready to put down the bailing bucket and just open the wallet.

@sfx2000 : Thank you for recognizing that this is basically the result of growth and attempts at problem solving over time. I'm happy to look into Omada and/or Ubiquity as more appropriate hardware options and accept that we'll likely want to change the CIDR to something like 172.16.0.0/12 for more addressable IP space. I want to add smart lights and speakers to the outdoor landscaping plan, so client count is likely to increase from here. I will hop on reading the article (https://arstechnica.com/information...flecting-on-almost-three-years-with-pro-gear/) you provided and appreciate you sharing it.

Lastly, based on your mention of PCAPs, I found the following tutorial for grabbing one using my MacBook, but I'm not sure if targeting a particular channel (there are several in use) would be preferred or helpful.

Using OS X for Wireless Packet Capture: A MetaGeek Tutorial

It might have been more of a rhetorical mention to highlight the apparent trainwreck we currently have, but I'm hat-in-hand looking for help so I'm taking all info requests at face value.

Let me know and I'll do my level best to provide any additional details.

Fingers crossed I've responded appropriately.

 

[John Fitzgerald]

Asus indicates 5 nodes max, perhaps that‘s the problem.
Master plus 4 nodes.

[AiMesh] How to setup AiMesh system (Web GUI)? | Official Support | ASUS USA