RT-AX88U messed up DHCP on wired devices (beta 1 and 3)

[werkstrom]

A month ago (or so) I upgraded from an AC68U that have served me well to a AX88U (Black Friday sale ). Never ran it on stock FW as I have run Merlin on the AC68U “forever”. Running the beta (starting with 1 and updating to 3 today). I am experiencing issues with the clients on the wired network losing their connection to the LAN. After a bit of troubleshooting (the normal restarts of the router and such) I ended up checking the IP config and it turned out that for some reason the IP had been set to a completely different net on all the wired devices. The DNSes and gateway was correct including the local router IP (using 1.1.1.1, 8.8.8.8 and the local 192.168.1.1) but the assigned IP gotten from the DHCP was 164.something.something.something (cant exactly remember the last three and lost the screenshot).

This issue affected all wired devices, including My Win10 PC, a Win2019 Server, the wired Chromecast Ultra, The wired Nvidia Shield Pro, The Sony Android TV… To resolve I rebooted the AX88 and everything went back to the correct Ips (after some time and/or rebooting/resetting the affected devices)

Since I have used the same ISP for well over two years using the AC68U and Merlin I have a bit of a hard time thinking there is something changed on the ISP side that would coincide perfectly with me changing router AND only affecting wired (but not wireless) devices…

One would suspect a second DHCP sever on the LAN but I can’t find one and neither can Fing. Since everything goes back to normal on a reboot one would perhaps suspect the DHCP on the router somehow provides IP’s from the wrong range. The fact that DNS and gateway addresses are correct even when the IP is wrong would also perhaps point in this direction. Thirdly the fact that wireless devices are unaffected would perhaps also point towards this.

There could of course be an upstream DHCP but my router gets an IP of 192.36.X.X from ISP so that makes no sense as my devices gets 164.X.X.X. Also, I am guessing Fing would possibly detect the second DHCP and it does not.



As I have set up A LOT of things now, I would very much like not to have to revert to Stock (or stable Merlin) but rather help in finding out if this is a bug and if so how to fix it. If possible, that is of course.

I have very little experience with using the logging features so my first question would be if there indeed is a possibility to log DHCP requests and assignments. If so, I guess that would be a place to start. Or (even better of course, some one having a brilliant idea on how to resolve the issue right of the bat )

As always, any input is greatly appreciated. Thanks in advance.

 

[ColinTaylor]

As you say, this make no sense at all. 164.x.x.x is a public IP address range not a private one so the router should never issue (or be configured to issue) that range. It would make slightly more sense if the addresses were 169.254.x.x as they are link local addresses.

I don't recall anybody else reporting this problem so unless you can provide screenshots and the syslog I don't think we'd ever know what happened.

 

[werkstrom]

Thanks for the input Colin,

You might very well be right about the 164 vs 169 thing. I had to take that of the top of my head. When/if there is an issue again I'll make sure to write it down. The 169.x.x.x does perhaps make some sense given my new findings in the Windows log of my PC.

Later in the afternoon I started my stationary PC and got this same error. All wired devices (including the Chrome Cast Ultra my son was using) did not work either. If I now check my Windows Event log I have two errors @ 14:40 where Windows was unable to obtain an IP from the DHCP resulting in a windows error:

Your computer was not assigned an address from the network (by the DHCP Server) for the Network Card with network address 0xF4B7E22D8721. The following error occurred: 0x79. Your computer will continue to try and obtain an address on its own from the network address (DHCP) server.

Here's where I think your input on 169.x.x.x might be right as it appears Windows defaults to (among others) 169.x.x.x if DHCP fails.

Now, that would, to me indicate something is stopping all traffic on the Wired LAN, or the DHCP not working properly. Since this affects all devices I would rule out device settings. Also I would rule out anything on the LAN itself (switches etc) as one affected PC is connected directly to one of the AI Mesh nodes and that in turn is directly connected to the AX88U without any switches or alike in between. That leaves the router and upstream.

Since wireless devices are not affected that would effectively rule out anything upstream... Leaving us with the router. Making a huge assumption that the DHCP on the router stops responding on the wired interface, for whatever reason. This would be expected behavior, right...

 

[ColinTaylor]

I can't think of a situation where a device connected directly to the router by Ethernet couldn't get a DHCP address but a device connected by WiFi could (although the reverse might be possible). Regardless, if you upload the complete syslog that includes the time period when the problem occurred we can take a look at it for any clues (for example, the router may have run out of memory).

 

[werkstrom]

Thanks again Collin, I really appreciate your help and input. One such scenario would probably be the LAN interface stopping, while the WiFi and WAN are still running... Not saying that is the case here but if that were the case, that would explain why all LANconnected devices stops working at the same time... And why they cannot reach the DHCP. If it was only the DHCP that was down, devices that is already up with an IP would still work, right. Any way. I'll Upload the logs for today. Would a PM be alright? Not entirely comfy with posting logs with all of this info onto an open forum...

Also like to clarify, heres how affected devices are connected to the network.

AndroidTV --> SubSwitch A --> Main Switch --> Main Router (AX88)
ChromecastUltra --> SubSwitch B --> Main Switch --> Main Router (AX88)
Nvidia Shield --> SubSwitch B --> Main Switch --> Main Router (AX88)
PC 1 --> MeshRouter (AX92) --> Main Router (AX88) - Note: This does not use the main switch
PC 2 --> Main Switch --> Main Router (AX88)

All of these are affected at the same time. None of the WiFi connected devices are affected, ever...

So the first common denominator for all affected devices is the AX88
The only (I can think of) common difference between affected and non affected devices is the interface (WiFi vs LAN)

If this happens again I'll try to set up one of the PCs with a manual IP and see if I can access the router, if not... Well then the IF is dead, for whatever reason. Otherwise the IF is up but the DHCP is not able to provide an IP. Also it would be fun to see if I could check the status of the LAN IFs by logging on to the router using WiFi, not sure how to do it though. Also if there was some way of restarting the LAN IFs without rebooting the entire router that would be interesting.

Again, I so appreciate your time and effort and am very grateful for all help I can get. So thanks once again.

 

[adrenalize]

I know you're not posting your log, but would be interested to see if it has lots of dnsmasq DHCPDISCOVER and DHCPOFFER but the device doesn't seem to take the IP and then Windows devices will default to giving themselves a 169.x.x.x address and will say no internet. Usually happens when turning on or when renewing a lease. As this happens at different times for different clients some may to continue to work until they renew. I've experienced this on and off since around 384.14 and occasionally others do too, full factory reset doesn't fix it, upgrading has mixed results on fixing it too. Can go days or months before it recurs. Not been able to put a finger on what exactly is happening. Not very scientific I know and hard to fault find as unpredictable. I have been on 384.19 for a while now and it's only happened a couple of times.

 

[ColinTaylor]

Sure, send me a PM and I'll take a look. If it's easier you can upload the log to pastebin.com with a password and expiry period and then PM me the details.

 

[werkstrom]

@adrenalize There are some dnsmasq, but only for "Ignoring domain my_employers_domain.com for DHCP host name my_corporate_laptop_name"

 

[werkstrom]

Quick update (and reason for me not posting a reply before).

Noticed that one of my two AX92U-AI-Mesh nodes always was disconnected when this issue appeared. Also noted that clients connected to my second AX92 Node worked just fine (working Internet connection f.ex). So I've been testing and right now I've removed the one AX92U that always disconnected and the net has been up without problem for almost a whole day. Will keep you posted as I find out more. Again thanks to everyone who is trying to help.

 

[werkstrom]

Alright, here's an update

Turned out I got the error just a few hours after posting the last post. I then disconnected the second mesh router as well, This time only unregistering it in the GUI but leaving it physically connected (not in use at all, just standing there on the shelf powered on) and... Lo and behold! The network have been working just fine ever since.

Not sure if I should continue this thread or start a new one but there appears to be some issue with AImesh.

I have had this setup (same AX92 nodes on same node firmware) setup with a AC68U for some time and that just worked fine. The AC68U did run Merlin as well, but the 384version, on the AX I'm running the 386 beta. So I can only think of three things that has changed

1. The physical router. If there is some sort of hardware issue with the AX. Kind of think this is the least probable cause since everything is physically still connected in exactly the same fashion as when I get the error
2. The AX88-FW in general. I have no idea if the AC68U and AX88U shares code and if so to what extent so I cannot determine if this is probable/likely
3. The 386 Merlin vs the 384.

Any of the above in combination with my particular AI-mesh setup (i.e. AX92 on stock FW)

Any input on how to further trouble shoot this would be greatly appreciated.

Heres what I'm thinking I could do:

1: Connect one of the AX92s again, but this time the one I disconnected first, i.e. eliminating AX92 no 2 as the root cause.
2: Flash a different FW on the AX88. Stock or 384 stable
3. Add the old AC68U as a mesh node

But I'd really appreciate some guidance to make sure my findings are useful and narrows down the root cause effectively, and again thanks for helping.

/Henrik

 

[werkstrom]

Ok, A bit of an update. I factory reset my (very) old AC68U and set it up as the lone AI-node. Has been running for the whole week now, without any issues... It should be noted I'm running the AC68U on the Merlin 384_19 firmware, that I know is not recommended (from what I understand it is preffered to run Asus FW on the nodes) but it still works without issues, for me. I will leav it a few days more to make sure it is still stable. And then move on with troubleshooting with an AX92.

One thought is that from my understanding the AX92 is using a quite different architecture compared to the AX88 and Ac68 as there is no Merlin FW for the AX92. Wonder if there might actually be some sort of compatibility issue.

And counting, with no issues so far.

 

[ColinTaylor]

I would be most suspicious of the change from firmware based on 384/385 to 386. Release 386 introduced AiMesh 2.0 which has some significant changes behind the scenes.

 

[werkstrom]

Thanks for the input @CollinTaylor. I am actually running the 386 FW (Merlin of course) on the "main" router which is the AX88U. It does appear to work just fine with the AC68U with old FW. I cannot see what version of FW the AX92U were running, as they are now disconnected for testing/troubleshooting. But I'll try to upgrade them to latest once I've run my network with the AC68U only connected to the AX88 for a bit more just to validate it's working. If the AX92Us then introduces issues again, I'll try to downgrade them to an older FW. Thanks again for the input.

And counting. No issues so far.