WAN was exceptionally disconnected

[Kanji-San]

I debugged the problem further to find out that Internet connectivity was still at the ONT and that my router sporadically dropped WAN connectivity and issued the dreaded ISP's DHCP did not function properly. (That was still on 384.19 which never had this problem before my gigabit upgrade)

I bit the bullet and nuclear reset the router and all nodes and upgraded to the latest 386.2_2 and painstakingly entered all settings manually. And it just happened again, no Internet connectivity although the Internet is still working at the ONT.

No matter what I do on the router, toggle WAN, ifconfig down/up, put a switch between the router and the ONT, the only way to recover is to reboot the router (RT-AX88U).

Does anyone have any ideas what else to do?

 

[ColinTaylor]

And it just happened again, no Internet connectivity although the Internet is still working at the ONT.

How are you determining that the internet is still working? If you're unplugging the cable from the ONT and plugging it into a PC that's the same as you rebooting the router so it doesn't really confirm that the internet is still working.

Did you get anywhere with your tcpdump? Does the DHCP error message and loss of internet happen at the moment the lease expires?

 

[Kanji-San]

How are you determining that the internet is still working? If you're unplugging the cable from the ONT and plugging it into a PC that's the same as you rebooting the router so it doesn't really confirm that the internet is still working.

Did you get anywhere with your tcpdump? Does the DHCP error message and loss of internet happen at the moment the lease expires?

Thank you for your responses. I really appreciate them.

When the WAN connection goes down at the router, I disconnect it from the ONT and connect a laptop with the cloned MAC address from the router to the ONT. I immediately get the same IP address I always get and Internet connectivity is there. It is really the router dropping the WAN connection.

Unfortunately, unplugging the cable from the ONT is not the same thing as rebooting the router. When I plug it back into the router, WAN is still down. Even a switch between the ONT and router does not make a difference. It looks like there is a long history of this WAN disconnect problem with Asus firmware. I have been reading many threads, some 9 years old.

tcpdump revealed that the router or udhcpc does exactly what it is supposed to do. When the WAN goes down, it sends Discover packets according to the selected DHCP query frequency setting. There are never any incoming packets. It is not related to an expiring lease. It's completely random.

These WAN ISP's DHCP does not function properly errors started happening when my ISP upgraded me to gigabit service, replacing a media converter with an ONT, replacing optical splitters, and giving me a totally different IP address.

 

[ColinTaylor]

OK, so physically unplugging the Ethernet cable from the ONT, waiting a few seconds and plugging it back in doesn't solve the problem. The only scenario that works is rebooting the router. Correct?

What is the router's WAN connection type? Automatic IP, PPPoE, etc. Has this changed from before?

 

[Kanji-San]

OK, so physically unplugging the Ethernet cable from the ONT, waiting a few seconds and plugging it back in doesn't solve the problem. The only scenario that works is rebooting the router. Correct?

What is the router's WAN connection type? Automatic IP, PPPoE, etc. Has this changed from before?

Yes, only reboot works.

It's Automatic IP. No change, no dual WAN.

 

[ColinTaylor]

Well I'm out of ideas. But then I've never had this problem, other than when there genuinely was a problem with my ISP's DHCP.

I'm assuming that this problem is initially caused because there really is a loss of internet connectivity, albeit briefly. The router's Network Monitoring detects this and tries to restart the WAN connection.

It might be worth turning off Network Monitoring (ping and DNS) under Administration - System. That might reduce the frequency of this problem happening. Other than that you could create the /jffs/scripts/dhcpc-event script below to see if udhcpc is doing something unexpected.

#!/bin/sh
logger -t "dhcpc-event" "Parms =" $1 $2

 

[Kanji-San]

Well I'm out of ideas. But then I've never had this problem, other than when there genuinely was a problem with my ISP's DHCP.

I'm assuming that this problem is initially caused because there really is a loss of internet connectivity, albeit briefly. The router's Network Monitoring detects this and tries to restart the WAN connection.

It might be worth turning off Network Monitoring (ping and DNS) under Administration - System. That might reduce the frequency of this problem happening. Other than that you could create the /jffs/scripts/dhcpc-event script below to see if udhcpc is doing something unexpected.

#!/bin/sh
logger -t "dhcpc-event" "Parms:" $1 $2

Good ideas, but I already tried them both.

I started looking into the wanduck code and it seems there is an nvram variable that offers some debug output:

nvram set wanduck_debug=1
nvram commit
service restart_wan

However I don't see where it goes. Still hunting down _dprintf which might or might not write to /dev/console.
Anyone with some experience?

 

[KevTech]

As a test, have you tried running one of the AC68 hooked up to the ONT instead of the AX88?

 

[Kanji-San]

As a test, have you tried running one of the AC68 hooked up to the ONT instead of the AX88?

I have not tried this, yet. Probably, the next thing I will do.

 

[netware5]

Just one remark: The same thing happened once on my AX88U since I upgraded to 386.2. But I am with FIXED WAN IP address! So no DHCP interaction with my ISP! As it happened only once I am not able to investigate it more. I have a theory - maybe something wrong happened after the WAN went off and on again and the router's real hardware MAC address has been displayed instead the manually entered MAC address in Menu>WAN>Internet Connection>MAC Address (my ISP has a "MAC address protection" activated, so I shall use a particular MAC address). Then the router reboot fixed the issue displaying the required MAC address ...

 

[Kanji-San]

Just one remark: The same thing happened once on my AX88U since I upgraded to 386.2. But I am with FIXED WAN IP address! So no DHCP interaction with my ISP! As it happened only once I am not able to investigate it more. I have a theory - maybe something wrong happened after the WAN went off and on again and the router's real hardware MAC address has been displayed instead the manually entered MAC address in Menu>WAN>Internet Connection>MAC Address (my ISP has a "MAC address protection" activated, so I shall use a particular MAC address). Then the router reboot fixed the issue displaying the required MAC address ...

This is an interesting idea that the ONT blocks because of a MAC address mismatch. My ISP enforces one MAC address. The ONT enforces the one that it sees after the power-up.
One argument against this idea is that I power-cycled the ONT when the router dropped the WAN connection which had no effect.

 

[Kanji-San]

In the meantime I experimented with some other configurations. I got a MikroTik router and connected it to the ONT and put my Asus RT-AX88U in a double NAT configuration behind it. Surprisingly, my Asus router still went down (ISP's DHCP did not function properly) while the MikroTik still had Internet/WAN connectivity. This is a clear sign, that my Asus router is the culprit.

The last thing I tried was completely wiping my Asus router and installing the latest stock firmware 386.42820 which has some WAN fixes (42819). Again, after one day it went down with the usual message and only a reboot fixed it.

Now, the RT-AX88U goes back to Asus for warranty repair. In the meantime, I am running my old RT-AC68U with Merlin 386.2_4. Let's see how this goes...

Btw, my ISP replaced the original ONT with a different model, put me on a different optical splitter, and measured the fiber light levels. All seems good on the ISP side.

 

[Carlos Irigaray]

In the meantime I experimented with some other configurations. I got a MikroTik router and connected it to the ONT and put my Asus RT-AX88U in a double NAT configuration behind it. Surprisingly, my Asus router still went down (ISP's DHCP did not function properly) while the MikroTik still had Internet/WAN connectivity. This is a clear sign, that my Asus router is the culprit.

The last thing I tried was completely wiping my Asus router and installing the latest stock firmware 386.42820 which has some WAN fixes (42819). Again, after one day it went down with the usual message and only a reboot fixed it.

Now, the RT-AX88U goes back to Asus for warranty repair. In the meantime, I am running my old RT-AC68U with Merlin 386.2_4. Let's see how this goes...

Btw, my ISP replaced the original ONT with a different model, put me on a different optical splitter, and measured the fiber light levels. All seems good on the ISP side.

I have been experiencing this same problem since I replaced my AC5300 with an AX88U. I would get the WAN error every few days or even a few times a day. After trying everything (reinstalled firmware after wiping out all settings and reconfiguring everything manually) same problem. 2 weeks ago I broke the AiMesh and only kept the AX88U connected to the ONT and it has been working without any issues since. This morning I setup AiMesh again and a few hours later I got the WAN error. It doesn't seem to make any sense but, in my experience, it looks like this happens only when AiMesh is active.

 

[Kanji-San]

I have been experiencing this same problem since I replaced my AC5300 with an AX88U. I would get the WAN error every few days or even a few times a day. After trying everything (reinstalled firmware after wiping out all settings and reconfiguring everything manually) same problem. 2 weeks ago I broke the AiMesh and only kept the AX88U connected to the ONT and it has been working without any issues since. This morning I setup AiMesh again and a few hours later I got the WAN error. It doesn't seem to make any sense but, in my experience, it looks like this happens only when AiMesh is active.

This is really interesting. I do use AiMesh. I would try this but I sent in my AX88U to Asus for repair.

However, I had been running the AX88U with AC68U as an AiMesh node for around eight months on 384.19 without any problems. WAN drops started when my Internet service was upgraded from 100 to 1000Mbit/s (the media converter was replaced by an ONT). WAN drops continued after a firmware upgrade to Merlin 386.2_4. Even on the latest stock firmware.

It might be a combination of factors that play a role here.

I am currently running the AC68U on 386.2_4 with another AC68U as an AiMesh node without any WAN drops. Ergo, the AX88U is the culprit. I cannot say if this is hardware or firmware-related.

 

[Carlos Irigaray]

This is really interesting. I do use AiMesh. I would try this but I sent in my AX88U to Asus for repair.

However, I had been running the AX88U with AC68U as an AiMesh node for around eight months on 384.19 without any problems. WAN drops started when my Internet service was upgraded from 100 to 1000Mbit/s (the media converter was replaced by an ONT). WAN drops continued after a firmware upgrade to Merlin 386.2_4. Even on the latest stock firmware.

It might be a combination of factors that play a role here.

I am currently running the AC68U on 386.2_4 with another AC68U as an AiMesh node without any WAN drops. Ergo, the AX88U is the culprit. I cannot say if this is hardware or firmware-related.

Problem started for me after I did the following:
- Connected AX88U to WAN
- Setup AC5300 as AIMesh Node (connected in ethernet backhaul mode)
- Upgrade both to latest 386.2 firmware.

As you said, it is probably a combination of these factors: AiMesh, AX88U and firmware are in common on both setups. I want to keep the AX88U connected to the WAN as it has 8 ports for the LAN and the AC5300 only has 4. Will try to setup the AC5300 as a regular AP node.

Will try a few different setups and report later.

 

[Kanji-San]

Problem started for me after I did the following:
- Connected AX88U to WAN
- Setup AC5300 as AIMesh Node (connected in ethernet backhaul mode)
- Upgrade both to latest 386.2 firmware.

As you said, it is probably a combination of these factors: AiMesh, AX88U and firmware are in common on both setups. I want to keep the AX88U connected to the WAN as it has 8 ports for the LAN and the AC5300 only has 4. Will try to setup the AC5300 as a regular AP node.

Will try a few different setups and report later.

I suspect the switch from 100Mbit/s to 1Gbit/s on the WAN side played a role because this was really the only thing, besides turning off QoS to achieve the full throughput, that changed when the drops started for me.

 

[Kanji-San]

Here is my final update of my journey that started almost three months ago when my Fiber connection was upgraded from 100Mbit/s to 1Gbit/s service. All of the equipment on the ISP side was replaced. My RT-AX88U started dropping the WAN connection randomly immediately after the upgrade and after a lot of debugging, I finally determined that the router was the root cause. I sent in the RT-AX88U to Asus for warranty repair which unfortunately was not possible due to a lack of replacement parts (COVID!). Asus offered me a different (Merlin incompatible!) model which I declined. Finally, Asus offered me a buyback since no other models for replacements were available.

I purchased a new RT-AX86U, running Merlin of course, and everything has been stable ever since.

My conclusions: Although my RT-AX88U did not change when my FIOS services was upgraded, no piece of equipment can be ruled out until positively proven functional for the specific setup. I do not believe the AX88U simply broke exactly the day of the upgrade. I think there is an underlying incompatibility / issue between the RT-AX88U and my ISP's new equipment (GPON) or setup. I had no problems prior to the Gbit upgrade with my RT-AX88U on the 100Mbit/s service.

The WAN dropping was very intermittent, somewhere between one and five days which makes any kind of troubleshooting a very long process. I used tcpdump to record all WAN traffic and saw that the inbound traffic just randomly stopped. It was not DHCP lease related. A switch at the WAN port did not help. This even happened in a double-NAT configuration. Changing many different router settings as well as a complete factory reset and using the latest ASUS stock firmware did not make any difference.

I tried three different routers instead of my RT-AX88U and all of them maintained the WAN connection without problems. One was a Mikrotik and the other two were Asus (RT-AC68U, RT-AX86U) with stock and Merlin firmware.

I am still curious what problem really was but it must be buried deep in the RT-AX88U's hardware or firmware and probably only Asus could debug this.

It was a reasonable assumption at first that the equipment that was changed should be the root cause of any troubles. However, eventually a systematic step by step elimination process must be executed. Intermittent problems are hard to find.

I hope this tale helps someone else.

 

[ah1750]

Well I'm out of ideas. But then I've never had this problem, other than when there genuinely was a problem with my ISP's DHCP.

I'm assuming that this problem is initially caused because there really is a loss of internet connectivity, albeit briefly. The router's Network Monitoring detects this and tries to restart the WAN connection.

It might be worth turning off Network Monitoring (ping and DNS) under Administration - System. That might reduce the frequency of this problem happening. Other than that you could create the /jffs/scripts/dhcpc-event script below to see if udhcpc is doing something unexpected.

#!/bin/sh
logger -t "dhcpc-event" "Parms =" $1 $2

Just wanted to send a thanks for posting this suggestion. I've got a new RT-AX86U running the current 386_44130 and it was dropping the WAN regularly. I noticed, like others posted, that a full reboot would bring it back. I did have the Network Monitoring turned on via Ping -- and after shutting that off there's been no WAN loss. I also haven't gotten the "WAN Connection: WAN was exceptionally disconnected" in the logs any more. It seems like others may have different issues, but to me there's clearly a bug here that's causing the WAN to not reconnect.

 

[sfortis]

Same issue with with 386.3_2 on my AC86U. Router is getting an ip (pppoe) and after a few seconds is going to status "disconnected".
Rolled back to 386_3 and problem is gone.

 

[hawkman]

Just wanted to send a thanks for posting this suggestion. I've got a new RT-AX86U running the current 386_44130 and it was dropping the WAN regularly. I noticed, like others posted, that a full reboot would bring it back. I did have the Network Monitoring turned on via Ping -- and after shutting that off there's been no WAN loss. I also haven't gotten the "WAN Connection: WAN was exceptionally disconnected" in the logs any more. It seems like others may have different issues, but to me there's clearly a bug here that's causing the WAN to not reconnect.

Joined to say that turning off network monitoring via ping worked for me too, to resolve exactly this problem that was happening extremely often. So pleased I saw your post! I switched to monitoring via DNS lookup and that doesn’t cause the same problem for me no disconnects for days. Running 386_45934 on an AX82U, and using AiMesh.