AiMesh Node going OFFLINE after a few weeks

[Daddio]

I am running an AiMesh network that has occasionally experienced offline nodes. By occasionally, I have experienced offline nodes twice since setting up the AiMesh network in October. So, my problem is sporadic, and my posting is to add information on this problem.

There are 3 wired nodes in my network, and all went offline at the same time last week. I resolved the problem by rebooting the main router through its GUI. The nodes came back online. I detected that the nodes were offline when I logged into the router and seeing their offline designation. Here is peculiarity, my computer was wired directly into an offline node, but I had access to the router through the offline node. The devices wired into the nodes had internet connectivity. I did not test whether the offline nodes had wireless capability. My question is when a node is offline, what features are lost. Is it wireless access?

My AiMesh network has a RT-AC88U as the main router, and three RT-AC68U nodes. All nodes are wired directly into the router using cat 5e wire. The router indicates that three nodes are connected at 1 Gbps, and the router indicates that the connection quality is ‘Great’. I have achieved transfer speeds greater than 100 Mbps through a node. There is also wired access point wired to the router, and it never has given me any problem.

The RT-AC88U router has the Asus firmware 3.0.0.4.386_41800 installed, and the RT-AC68U nodes have 3.0.0.4.386_41634. Last week’s offline occurred when the nodes were running the prior Asus firmware 3.0.0.4.386.40558; I updated nodes after the offline occurrence. The first offline occurred in October 2020. and the router and nodes were running the latest Merlin firmware at that time.

 

[OzarkEdge]

I am running an AiMesh network that has occasionally experienced offline nodes. By occasionally, I have experienced offline nodes twice since setting up the AiMesh network in October. So, my problem is sporadic, and my posting is to add information on this problem.

There are 3 wired nodes in my network, and all went offline at the same time last week. I resolved the problem by rebooting the main router through its GUI. The nodes came back online. I detected that the nodes were offline when I logged into the router and seeing their offline designation. Here is peculiarity, my computer was wired directly into an offline node, but I had access to the router through the offline node. The devices wired into the nodes had internet connectivity. I did not test whether the offline nodes had wireless capability. My question is when a node is offline, what features are lost. Is it wireless access?

My AiMesh network has a RT-AC88U as the main router, and three RT-AC68U nodes. All nodes are wired directly into the router using cat 5e wire. The router indicates that three nodes are connected at 1 Gbps, and the router indicates that the connection quality is ‘Great’. I have achieved transfer speeds greater than 100 Mbps through a node. There is also wired access point wired to the router, and it never has given me any problem.

The RT-AC88U router has the Asus firmware 3.0.0.4.386_41800 installed, and the RT-AC68U nodes have 3.0.0.4.386_41634. Last week’s offline occurred when the nodes were running the prior Asus firmware 3.0.0.4.386.40558; I updated nodes after the offline occurrence. The first offline occurred in October 2020. and the router and nodes were running the latest Merlin firmware at that time.

Which AC88U LAN ports are your nodes wired to?

OE

 

[Daddio]

My nodes are wired into lan ports 1 thru 3.

 

[SignedAdam]

Ethernet still has lower latency than wifi, even if ax is now faster (in speed), it’s not when it comes to latency, power lines retain Ethernet latency where as wifi doesn’t

I unfortunately don’t have another AiMesh node. I only have an gt-ax11000 and a
RT-ac86u

The reason why the rt-ac86u is up in the attic is because we have found when it’s up there it does the best job of serving the garden

I hope with AiMesh 2.0 the zombie nodes have gone, I won’t be able to tell you till a month from now, non the less a bit of unstableness shouldn’t be causing zombie nodes

again having a reset feature in the nodes firmware would benefit everyone, including beta testers. I can see scenarios where zombie nodes might come walking in on beta firmware so I don’t see how you can’t see what I’m on about? shortsighted?

I was being sarcastic. We're back to debating how much network instability is to be tolerated. My position is still "none"... lossy Ethernet is unacceptable here.

Do you have another router you can swap into the attic to see how it performs there and how the current attic router performs elsewhere?

OE

 

[SignedAdam]

Unlike you @Daddio my offline nodes stay offline even after a reboot of the main node (router) ever experienced that yet ?

My nodes are wired into lan ports 1 thru 3.

 

[OzarkEdge]

Ethernet still has lower latency than wifi, even if ax is now faster (in speed), it’s not when it comes to latency, power lines retain Ethernet latency where as wifi doesn’t

I unfortunately don’t have another AiMesh node. I only have an gt-ax11000 and a
RT-ac86u

The reason why the rt-ac86u is up in the attic is because we have found when it’s up there it does the best job of serving the garden

I hope with AiMesh 2.0 the zombie nodes have gone, I won’t be able to tell you till a month from now, non the less a bit of unstableness shouldn’t be causing zombie nodes

again having a reset feature in the nodes firmware would benefit everyone, including beta testers. I can see scenarios where zombie nodes might come walking in on beta firmware so I don’t see how you can’t see what I’m on about? shortsighted?

AiMesh 2.0 here offers a System Reset and a Node Reboot, but I suppose neither will work if the node is "locked up" or a "zombie"... but toggling the circuit breaker/power should wake it up.

OE

 

[SignedAdam]

As I have said previously, removing power, turning the socket off then back on does nothing for me, it stays in zombie state till I factory reset it. but this was on AiMesh 1.0 not 2.0

Again I can access the nodes IP address when it gos in to this zombie state, so if the option to reset the node to defaults was offered at the nodes IP address, then yes I would be able to reset the thing to defaults. it makes a lot of sense to me to offer this option with in the nodes IP address, just like the ability to upload firmware to it.

AiMesh 2.0 here offers a System Reset and a Node Reboot, but I suppose neither will work if the node is "locked up" or a "zombie"... but toggling the circuit breaker/power should wake it up.

OE

 

[OzarkEdge]

As I have said previously, removing power, turning the socket off then back on does nothing for me, it stays in zombie state till I factory reset it. but this was on AiMesh 1.0 not 2.0

Again I can access the nodes IP address when it gos in to this zombie state, so if the option to reset the node to defaults was offered at the nodes IP address, then yes I would be able to reset the thing to defaults. it makes a lot of sense to me to offer this option with in the nodes IP address, just like the ability to upload firmware to it.

How old is your AC86U... it could be dying?

Have you tried using AiMesh 2.0 Reconnect Node?

OE

 

[SignedAdam]

The rt-ac86u is barely 2 years old

What’s AiMesh 2.0 reconnect?

How old is your AC86U... it could be dying?

Have you tried using AiMesh 2.0 Reconnect Node?

OE

 

[OzarkEdge]

The rt-ac86u is barely 2 years old

What’s AiMesh 2.0 reconnect?

2-year old AC86U... hmm.

AiMesh 2.0 has a Reconnect Node command under AiMesh\node\Management\Reconnect Node.

OE

 

[SignedAdam]

I’ll be sure to test the new “reconnect node” feature out when and if time requires it, thank you for sharing the information of the new feature with me

2-year old AC86U... hmm.

AiMesh 2.0 has a Reconnect Node command under AiMesh\node\Management\Reconnect Node.

View attachment 29823

OE

 

[Thw0rted]

@SignedAdam did you try SSH yet? Administration -> System, Enable SSH -> LAN only, disable password login, paste your public key in Authorized Keys.

If you're on Mac or Linux, you should already have SSH and associated utilities.

If you're on Windows and need a client, PuTTY is my go-to, and these docs do a pretty good job of explaining how to make and use a key.

 

[Thw0rted]

@Daddio I have had the same experience as you. The way I understand it, the network ports act as a mostly-unmanaged switch, effectively (with an asterisk for the WAN port in certain modes). When a node is "offline", as described in this thread, it's not because the device isn't running, but because it wasn't able to negotiate membership in the mesh for whatever reason. If you think about it, though, "the mesh" is really just a bunch of wireless settings (SSID, auth, block-lists, some signal metadata to facilitate handoff, etc) along with some very basic top-level settings (I'm guessing NTP, network gateway, web interface login, LED control?) that shouldn't have any impact on wired connections.

That's why I keep suggesting SSH -- a node that can't sync to the mesh controller should "coast" on its last known settings, which should include basic network data and accounts. If we knew where to look, we might even be able to find something in the logs to tell us why it's failing to (re-)join.

 

[SignedAdam]

You don’t need a client on Windows 10 anymore, enable subsystem for Linux, with in “windows add ons” you’ll find that under add and remove programs.

after you’ve enabled subsystem for Linux under windows 10 you’ll have a bash application, just type bash after enabling the add on, then you can use bash under windows 10 to ssh in to the router or nodes “anything”

I know all about ssh, however it isn’t as easy as going to a node IP address in the browser, having a nice login page then a reset to defaults option.. if this isn’t available then I suggest @ASUSWRT_2020 enables the option to - reset a node remotely

@SignedAdam did you try SSH yet? Administration -> System, Enable SSH -> LAN only, disable password login, paste your public key in Authorized Keys.

If you're on Mac or Linux, you should already have SSH and associated utilities.

If you're on Windows and need a client, PuTTY is my go-to, and these docs do a pretty good job of explaining how to make and use a key.

 

[Thw0rted]

I have to imagine that this would be pretty far down their todo list, because if your node is in a state where you can view its web interface but it's not showing up on the main router's AiMesh page, something has gone very wrong already. (Also, if it's any indication of likelihood, the one and only web page I've been able to load from a node is the one to remotely upload firmware, linked from the main Firmware page -- any other URL redirects to the main router's front page.)

Put another way, I think a web interface for this would be nicer than fixing it by SSH, but while you wait for that, setting up SSH is a lot nicer than crawling around in wads of fiberglass.

 

[OzarkEdge]

You don’t need a client on Windows 10 anymore, enable subsystem for Linux, with in “windows add ons” you’ll find that under add and remove programs.

after you’ve enabled subsystem for Linux under windows 10 you’ll have a bash application, just type bash after enabling the add on, then you can use bash under windows 10 to ssh in to the router or nodes “anything”

I know all about ssh

You can also use Win10's built-in OpenSSH Client from PowerShell:

PS C:\Users\OE> ssh OE@192.168.1.1

OE

 

[Thw0rted]

You can also use Win10's built-in OpenSSH Client from PowerShell:

PS C:\Users\OE> ssh OE@192.168.1.1

OE

Boy, Win10 contains multitudes, doesn't it? I've used WSL bash, but that's a big feature to suggest adding in a general-purpose forum like this one. Everybody's got PowerShell these days, though. I didn't realize it had its own SSH client, which is neat. (FYSA, apparently you have to use `ssh-keygen`, where I was more used to `openssl genkey`.)

 

[SignedAdam]

I can confirm it is indeed built in to windows 10 powershell. Nice find @OzarkEdge

you can also download openssh for windows 8.1 and integrate it using https://github.com/PowerShell/Win32-OpenSSH

Takes a lot of work

You can also use Win10's built-in OpenSSH Client from PowerShell:

PS C:\Users\OE> ssh OE@192.168.1.1

OE

 

[OzarkEdge]

I can confirm it is indeed built in to windows 10 powershell. Nice find @OzarkEdge

It was nothing... I never install a program without first checking to see if Windows can do it well enough. Saves time and trouble.

OE

 

[Thw0rted]

Now be fair, I've been using PuTTY for the thick end of two decades and they only shipped a built-in SSH client in 2018, if 10 seconds of googling got that right. (Otherwise, absolutely, don't install an extra program if there's a decent built-in solution!)