AiMesh Node going OFFLINE after a few weeks

[LimJK]

Hi,

I last reported this issue about 3 weeks ago over GUI Feedback when one of my AiMesh Node (Bedroom) gone OFFLINE. I rebooted everything and all seems to be fine and I continue to observe. This morning suddenly my other AiMesh Node (StudyRoom) got OFFLINE status.

I am posting here, hoping to get Asus attention, and I will point to this thread in my GUI feedback.

I only start to have this kind of problem with the current firmware.
(My configuration in the Signature below)

 

[VANT]

Same with me. Yesterday my 86u not start 5GHz band and node (68u) was offline..... Both routers have latest fw

 

[LimJK]

Same with me. Yesterday my 86u not start 5GHz band and node (68u) was offline..... Both routers have latest fw

Additional Input for Asus:

• I just checked that my WiFi device (iPhoneX) is able to roam to this node and everything is working as normal.
• So it is just the GUI report that AiMesh Node is OFFLINE that is wrong!

 

[LimJK]

On more additional information: I was looking at the log and I noticed that there is a continuous stream of DHCPREQUEST every 30 seconds for the last 24 hours. The following is just a section of the log:

Nov 14 22:59:09 dnsmasq-dhcp[342]: DHCPREQUEST(br0) 192.168.nnn.253 10:7b:44:13:xx:xx
Nov 14 22:59:09 dnsmasq-dhcp[342]: DHCPACK(br0) 192.168.nnn.253 10:7b:44:13:xx:xx
Nov 14 22:59:39 dnsmasq-dhcp[342]: DHCPREQUEST(br0) 192.168.nnn.253 10:7b:44:13:xx:xx
Nov 14 22:59:39 dnsmasq-dhcp[342]: DHCPACK(br0) 192.168.nnn.253 10:7b:44:13:xx:xx
Nov 14 23:00:09 dnsmasq-dhcp[342]: DHCPREQUEST(br0) 192.168.nnn.253 10:7b:44:13:xx:xx
Nov 14 23:00:09 dnsmasq-dhcp[342]: DHCPACK(br0) 192.168.nnn.253 10:7b:44:13:xx:xx
Nov 14 23:00:39 dnsmasq-dhcp[342]: DHCPREQUEST(br0) 192.168.nnn.253 10:7b:44:13:xx:xx
Nov 14 23:00:39 dnsmasq-dhcp[342]: DHCPACK(br0) 192.168.nnn.253 10:7b:44:13:xx:xx
Nov 14 23:01:09 dnsmasq-dhcp[342]: DHCPREQUEST(br0) 192.168.nnn.253 10:7b:44:13:xx:xx
Nov 14 23:01:09 dnsmasq-dhcp[342]: DHCPACK(br0) 192.168.nnn.253 10:7b:44:13:xx:xx
Nov 14 23:01:39 dnsmasq-dhcp[342]: DHCPREQUEST(br0) 192.168.nnn.253 10:7b:44:13:xx:xx
Nov 14 23:01:39 dnsmasq-dhcp[342]: DHCPACK(br0) 192.168.nnn.253 10:7b:44:13:xx:xx

Edit: Oops, forgot to mentioned that the DHCPREQUEST(br0) 192.168.nnn.253 10:7b:44:13:xx:xx comes from the offending AiMesh Node (Studyroom)

 

[OzarkEdge]

Additional Input for Asus:

• I just checked that my WiFi device (iPhoneX) is able to roam to this node and everything is working as normal.
• So it is just the GUI report that AiMesh Node is OFFLINE that is wrong!

I think there is a bunch of cosmetic stuff ASUS has not been able to get to yet.

OE

 

[LimJK]

I think there is a bunch of cosmetic stuff ASUS has not been able to get to yet.

OE

OzarkEdge,
Initially, I thought that it is just cosmetic too, however, in my subsequent post, it is obviously a problem ... from the log; there is a continuous stream of DHCPREQUEST(br0) 192.168.nnn.253 10:7b:44:13:xx:xx at about a 30 seconds interval for about 24 hours. Hoping to get Arthur's attention.

I rebooted all the AiMesh Router and Nodes ... it is OK now. I am watching to see when the same issue is going to happen for the 3rd time; I think I may have to wait for another about 3 weeks

Other than that my day to day usage has been excellent; all my wireless devices are Apple (MacBookpro(s) & iPhone(s)) are 5.0GHz capable, so I hardly have a chance to use 2.4GHz where I see some issues has been discussed here.

 

[HuskyHerder]

@LimJK

Another user @Markboz and myself have also experienced the dropping mesh node. Rebooting for me also brought the node back online. Ours was a private conversation nothing posted to the forums.

I am also on 3.0.0.4.384_32799 on a 5300 and on 2 nodes rt-ac68u.

All my devices are 5G Macs, iOS etc much the same as yours with the exception of 4 IOT devices.

Edit : Spectrum (ISP) has been doing work in the area. I have been having to reboot the 5300 the last few days to regain connectivity along with my modem. So I doubt I can be much use as far as logs go.

 

[OzarkEdge]

OzarkEdge,
Initially, I thought that it is just cosmetic too, however, in my subsequent post, it is obviously a problem ... from the log; there is a continuous stream of DHCPREQUEST(br0) 192.168.nnn.253 10:7b:44:13:xx:xx at about a 30 seconds interval for about 24 hours. Hoping to get Arthur's attention.

I rebooted all the AiMesh Router and Nodes ... it is OK now. I am watching to see when the same issue is going to happen for the 3rd time; I think I may have to wait for another about 3 weeks

Other than that my day to day usage has been excellent; all my wireless devices are Apple (MacBookpro(s) & iPhone(s)) are 5.0GHz capable, so I hardly have a chance to use 2.4GHz where I see some issues has been discussed here.

I see what you mean. Maybe the wired backhaul/switch are causing some oddness... no particular reason for that thought!

OE

 

[LimJK]

I see what you mean. Maybe the wired backhaul/switch are causing some oddness... no particular reason for that thought!

OE

OzarkEdge,

• Backhaul: Yes this issue only happens with the recent Ethernet Onboarding for wired backhaul in 32799
• Switch: I used to have some AiMesh Node OFFLINE issue for WIRED backhaul with a very old Cisco gigabit Switch, however, since then I have changed to a newer Linksys GigaBit Switch, and I have not got any issue yet
• The challenge is this issue does not happen right away, so quite difficult for ASUS to reproduce to fix
• I went back to check my records
  • 1st occured on 1st Nov 2018
  • 2nd occur on 14th Nov 2018
  • Earlier, between 14 Sep (32799 released) to 1st Nov - I suppose, I have done multiple factory reset or reconfigure or reboot, etc. So AiMesh configuration was not in a stable state long enough for issue to be discovered

I will continue to monitor again to see if issue occurs the 3rd time in 2-3 weeks, however, hoping that Asus Engineer would have resolve it in the next firmware version.

 

[LimJK]

I will continue to monitor again to see if issue occurs the 3rd time in 2-3 weeks, however, hoping that Asus Engineer would have resolve it in the next firmware version.

Oops Now my other AiMesh Node (Bedroom) shows OFFLINE after I rebooted all AiMesh Router and Nodes about 2-3 days ago. OK I am going to do a complete factory reset this time and see when it is going to go OFFLINE again.

Hoping that someone in ASUS can help see what is the problem here, clearly repeatable.

 

[abracadabra11]

On more additional information: I was looking at the log and I noticed that there is a continuous stream of DHCPREQUEST every 30 seconds for the last 24 hours. The following is just a section of the log:

Nov 14 22:59:09 dnsmasq-dhcp[342]: DHCPREQUEST(br0) 192.168.nnn.253 10:7b:44:13:xx:xx
Nov 14 22:59:09 dnsmasq-dhcp[342]: DHCPACK(br0) 192.168.nnn.253 10:7b:44:13:xx:xx
Nov 14 22:59:39 dnsmasq-dhcp[342]: DHCPREQUEST(br0) 192.168.nnn.253 10:7b:44:13:xx:xx
Nov 14 22:59:39 dnsmasq-dhcp[342]: DHCPACK(br0) 192.168.nnn.253 10:7b:44:13:xx:xx
Nov 14 23:00:09 dnsmasq-dhcp[342]: DHCPREQUEST(br0) 192.168.nnn.253 10:7b:44:13:xx:xx
Nov 14 23:00:09 dnsmasq-dhcp[342]: DHCPACK(br0) 192.168.nnn.253 10:7b:44:13:xx:xx
Nov 14 23:00:39 dnsmasq-dhcp[342]: DHCPREQUEST(br0) 192.168.nnn.253 10:7b:44:13:xx:xx
Nov 14 23:00:39 dnsmasq-dhcp[342]: DHCPACK(br0) 192.168.nnn.253 10:7b:44:13:xx:xx
Nov 14 23:01:09 dnsmasq-dhcp[342]: DHCPREQUEST(br0) 192.168.nnn.253 10:7b:44:13:xx:xx
Nov 14 23:01:09 dnsmasq-dhcp[342]: DHCPACK(br0) 192.168.nnn.253 10:7b:44:13:xx:xx
Nov 14 23:01:39 dnsmasq-dhcp[342]: DHCPREQUEST(br0) 192.168.nnn.253 10:7b:44:13:xx:xx
Nov 14 23:01:39 dnsmasq-dhcp[342]: DHCPACK(br0) 192.168.nnn.253 10:7b:44:13:xx:xx

Edit: Oops, forgot to mentioned that the DHCPREQUEST(br0) 192.168.nnn.253 10:7b:44:13:xx:xx comes from the offending AiMesh Node (Studyroom)

Seeing same behavior in my system. Was rock solid for a few weeks and dropped offline and now spamming DHCPREQ and DHCPACK messages.

Tried rebooting mesh node, but still same behavior.

Has anyone found a solution or root cause for this?

 

[Thw0rted]

I think I posted this elsewhere, Asus really should let the main mesh AP / router assign a static IP to each participating node, when its own IP is static. It makes no sense to assign a static IP to the main AP but require operational DHCP to allow nodes to join the network.

 

[SignedAdam]

The problem I have is :

GT-AX11000 main node router
RT-AC86U node
Both are connected by Ethernet (power lines) tp-link 2000
The RT-AC86U after a few weeks or even days in service can suddenly lock up? how do I know it’s locked up? well when I use the web interface of GT-AX11000 and tell it to reboot RT-AC86U, the RT-AC86U ignores the command, if I turn the RT-AC86U off and on the RT-AC86U still ignores reboots commands, some times I can turn it on and it shows offline, yet when I go in to device list using the GT-AX11000 it’s listed there just sitting on the network... if I go to the ip I’m told to revisit the main node.

The only way of fixing the issue when it happens is to factory default the RT-AC86U
And add it back as a node, as the RT-AC86U is up in the attic this can be dangerous.... I shouldn’t have to hardware reset my node when it becomes unresponsive, thankfully it’s been happening less with aimesh 2.0 but it still can happen

I would like to see a back up option, where in the event a situation like this happens again I can go to the nodes IP address input a user name and password and press a reset to defaults button in the web ui of the node, that way I don’t have to go up in the attic holding buttons...

both my router node and nodes are on official firmware not beta

 

[OzarkEdge]

The problem I have is :

GT-AX11000 main node router
RT-AC86U node
Both are connected by Ethernet (power lines) tp-link 2000
The RT-AC86U after a few weeks or even days in service can suddenly lock up? how do I know it’s locked up? well when I use the web interface of GT-AX11000 and tell it to reboot RT-AC86U, the RT-AC86U ignores the command, if I turn the RT-AC86U off and on the RT-AC86U still ignores reboots commands, some times I can turn it on and it shows offline, yet when I go in to device list using the GT-AX11000 it’s listed there just sitting on the network... if I go to the ip I’m told to revisit the main node.

The only way of fixing the issue when it happens is to factory default the RT-AC86U
And add it back as a node, as the RT-AC86U is up in the attic this can be dangerous.... I shouldn’t have to hardware reset my node when it becomes unresponsive, thankfully it’s been happening less with aimesh 2.0 but it still can happen

I would like to see a back up option, where in the event a situation like this happens again I can go to the nodes IP address input a user name and password and press a reset to defaults button in the web ui of the node, that way I don’t have to go up in the attic holding buttons...

both my router node and nodes are on official firmware not beta

A node "locking up" is unusual.

Have you tried using a normal wired or wireless backhaul to see if the issue persists? If the node behaves normally without the powerline backhaul, then I would try to eliminate the powerline backhaul.

OE

 

[SignedAdam]

@OzarkEdge
it’s the same problem others are having so I thought I’d add to it, if I switch to AiMesh with wifi the signal to the node won’t be strong, which is the whole reason I’m using power lines, as it happens over a period of days, I’d suggest it’s something to do with AiMesh, because before any of the AiMesh stuff came out I used the attic as a “access point” instead of a “AiMesh node never had an issue” which again suggests it’s something to do with AiMesh. Wouldn’t be such a problem if the nodes had a back door way of resetting them by their IP address. Which is totally possible but for what ever reason asus hasn’t implemented that yet, I’ve noticed that they have implemented a way to access the usb of the node and use it as a few other things, by way of accessing the nodes IP address would be nice if they could just add that option and then i have an extra way of resetting the thing when it does malfunction

 

[Thw0rted]

@SignedAdam if you click through to Administration -> Firmware Upgrade, then look at the list of nodes, you'll see an "Upload" link for each node, under the current version. This opens a new window into the web UI for that particular node, at {node IP}/AiMesh_Node_FirmwareUpgrade.asp. Alternately, I think the AiMesh management page might also list IPs for nodes?

Once you have the IP for the node, if you've already enabled SSH access (key-based, ideally) you should be able to SSH into the node. This would at least let you reboot the node using standard commands. I found this page which says that "flash default" resets the NVRAM but that could be out of date, I thought the modern OS used commands that start with "nvram". I've had a heck of a time finding a good reference...

 

[OzarkEdge]

@SignedAdam if you click through to Administration -> Firmware Upgrade, then look at the list of nodes, you'll see an "Upload" link for each node, under the current version. This opens a new window into the web UI for that particular node, at {node IP}/AiMesh_Node_FirmwareUpgrade.asp. Alternately, I think the AiMesh management page might also list IPs for nodes?

Once you have the IP for the node, if you've already enabled SSH access (key-based, ideally) you should be able to SSH into the node. This would at least let you reboot the node using standard commands. I found this page which says that "flash default" resets the NVRAM but that could be out of date, I thought the modern OS used commands that start with "nvram". I've had a heck of a time finding a good reference...

He said his node in the attic was "locked up". When something is locked up, it doesn't respond. Maybe it's not locked up but just hates being stuck in the attic.

OE

 

[Thw0rted]

I don't think he explained exactly what "locked up" looks like. Does it stop responding to pings? If SSH were enabled, could he still log in that way? What about the web interface (e.g. that "node firmware upload" mini-page)?

One of my least favorite things about mesh is the amount of magic involved -- when it works, it just works, but when the node won't join... well, why won't it join? Do the logs tell you? No, just format everything and hope it works after that.

 

[OzarkEdge]

I don't think he explained exactly what "locked up" looks like. Does it stop responding to pings? If SSH were enabled, could he still log in that way? What about the web interface (e.g. that "node firmware upload" mini-page)?

One of my least favorite things about mesh is the amount of magic involved -- when it works, it just works, but when the node won't join... well, why won't it join? Do the logs tell you? No, just format everything and hope it works after that.

And he implied that it works wireless without the powerline link.

OE

 

[SignedAdam]

Should be a page at {node IP}/ that has a button to reset the node to defaults, shouldn’t have to ssh in just to reset a node... something like {node IP}/AiMesh_Node_Reset.asp ? with a button to do just that...

Are you guys seriously telling me that you’ve never had a node not show up in the main node interface? Locking up means becoming unresponsive, I’d say this behaviour is locking up