AX88U Merlin 386.x and 388.1 random WAN disconnects mostly without relevant log entries

[JohnDeere]

The idea is to set the router's WAN MTU to be lower than that of the modem. So if the modem defaults to 1500, set the router to 1400.

OK, I tried with modem 1400 MTU and router 1500 MTU. Result is the same, rendom disconnects. Seems like I have to do a clean setup...

 

[JohnDeere]

I didn't even pick up on that you're using an LTE modem until @ColinTaylor mentioned this. I'm going to have to guess that a wireless signal like this will always run into more connection issues than a hardline. Here are a few more plausible explanations in your case for why you see it working most times, but not other times:

1. Cell tower is overloaded/latency issues
2. Wireless interference from some other outside source
3. Signal quality or low signal issues
4. Electromagnetic interference / sunspots.

Yes, you're right but I'm able to monitor my LTE connection in details. I locked my modem to the strongest LTE cell tower, SNR is perfect, I have external antennas for both uplink and downlink, so this part is fine. On 20MHz I have download speeds up to 140Mbit, which is outstanding.
Connecting my LAN to the modem directly works without noticeable errors.

Used to be without errors with router for years. Problems started to occur since one of the latest FW versions.

 

[JohnDeere]

There are many, many staples that hold the whole thing together.
Networking is not for average Joe, I'm afraid.
To many points can fail solo or in cascade.

But consumer products are made for average Joe (maybe I'm wrong)
My original problem had the same effects, but no LTE modem.
My connection to the internet is a 1,5mtr long CAT7 cable.

To be precise, my LTE modem is built into a Mikrotik LTE router, which acts as a WAN passthrough bridge, so basically I have more ways to connect my AX88U behind this LTE router. I will do some experiments if I will have some time.

 

[Viktor Jaep]

To be precise, my LTE modem is built into a Mikrotik LTE router, which acts as a WAN passthrough bridge, so basically I have more ways to connect my AX88U behind this LTE router. I will do some experiments if I will have some time.

Sounds like a serious piece of equipment. I really hope you find a solution! Other than a wipe, possible firmware downgrade, or rip&replace equipment, I'm about fresh out of ideas. :/

 

[ColinTaylor]

I also get this warning on every ovpn connection. I thought it may have some relation with disconnections.

Dec 07 14:34:28 RT-AX88U ovpn-server1 WARNING: 'link-mtu' is used inconsistently, local='link-mtu 1601', remote='link-mtu 1585'

It's unrelated to the disconnections. That message always appears and is a known issue with OpenVPN and can be ignored.

 

[DocUmibozu]

Hello everyone,

I have a WAN disconnection problem since 386.x and 388.1. If anyone has any idea, please let me know. The symptom is:

Dec 05 18:29:23 RT-AX88U WAN(0)_Connection WAN(0) link down.
Dec 05 18:29:28 RT-AX88U WAN(0)_Connection WAN was restored.

My WAN connection gets disconnected randomly. There are days when no disconnection occurs at all. And there are days when WAN is disconnected a dozen of times.

Hi, this week also started happening to me after upgrade to 388.1
I've an ax68u which was super stable with 388 betas.
Now I see a random wan disconnect/reconnect every 12/14 hours.
Wan is served to the asus by an ER-X which load balances two connections (one fttc and one lte).

Asus logs are these:

Dec 8 09:39:19 kernel: eth0 (Int switch port: 3) (Logical Port: 3) (phyId: c) Link DOWN.
Dec 8 09:39:19 kernel: ===> Activate Deep Green Mode
Dec 8 09:39:19 kernel: bcmswlpbk0 (Ext switch port: 8) (Logical Port: 8) Virtual link DOWN
Dec 8 09:39:21 kernel: <=== Deactivate Deep Green Mode
Dec 8 09:39:21 kernel: bcmswlpbk0 (Ext switch port: 8) (Logical Port: 8) Virtual link UP


ER-X logs are these:

Dec 8 08:39:21	ubnt kernel: ESW: Link Status Changed - Port2 Link UP
Dec 8 08:39:18	ubnt kernel: ESW: Link Status Changed - Port2 Link Down

 

[JohnDeere]

Sounds like a serious piece of equipment. I really hope you find a solution! Other than a wipe, possible firmware downgrade, or rip&replace equipment, I'm about fresh out of ideas. :/

I also hope so, thanks! (by the way, the WAN device is a Mikrotik Chateau)

 

[JohnDeere]

It's unrelated to the disconnections. That message always appears and is a known issue with OpenVPN and can be ignored.

OK, thanks a lot for clarifying this!

 

[JohnDeere]

Hi, this week also started happening to me after upgrade to 388.1
I've an ax68u which was super stable with 388 betas.
Now I see a random wan disconnect/reconnect every 12/14 hours.
Wan is served to the asus by an ER-X which load balances two connections (one fttc and one lte).

Asus logs are these:

Dec 8 09:39:19 kernel: eth0 (Int switch port: 3) (Logical Port: 3) (phyId: c) Link DOWN.
Dec 8 09:39:19 kernel: ===> Activate Deep Green Mode
Dec 8 09:39:19 kernel: bcmswlpbk0 (Ext switch port: 8) (Logical Port: 8) Virtual link DOWN
Dec 8 09:39:21 kernel: <=== Deactivate Deep Green Mode
Dec 8 09:39:21 kernel: bcmswlpbk0 (Ext switch port: 8) (Logical Port: 8) Virtual link UP


ER-X logs are these:

Dec 8 08:39:21	ubnt kernel: ESW: Link Status Changed - Port2 Link UP
Dec 8 08:39:18	ubnt kernel: ESW: Link Status Changed - Port2 Link Down

Somewhat similar to what I'm experiencing with this 388.1 release... Today I've had a dozen of disconnections so far. Some of them are:

Dec 08 07:35:21 RT-AX88U WAN(0)_Connection WAN(0) link down.
Dec 08 07:35:42 RT-AX88U WAN(0)_Connection WAN was restored.

Dec 08 08:20:26 RT-AX88U WAN(0)_Connection WAN(0) link down.
Dec 08 08:29:41 RT-AX88U WAN(0)_Connection WAN was restored.

Dec 08 08:36:14 RT-AX88U WAN(0)_Connection WAN(0) link down.
Dec 08 08:36:21 RT-AX88U WAN(0)_Connection WAN was restored.

Dec 08 09:05:35 RT-AX88U WAN(0)_Connection WAN(0) link down.
Dec 08 09:05:46 RT-AX88U WAN(0)_Connection WAN was restored.

etc etc. Not always the same time difference between events.

Did you do a "dirty" upgrade to 388.1 ?

 

[ColinTaylor]

Hi, this week also started happening to me after upgrade to 388.1
I've an ax68u which was super stable with 388 betas.
Now I see a random wan disconnect/reconnect every 12/14 hours.

I suspect the firmware update is just coincidental. There was only one minor cosmetic change between beta4 and the release version. Go back to beta4 to confirm whether or not the problem goes away.

 

[DocUmibozu]

Did you do a "dirty" upgrade to 388.1 ?

Yes I did.
And the only change I made to the network in the last week was the firmware upgrade.
I've started to collect the logs to have a clearer picture of the problem.

 

[JohnDeere]

I suspect the firmware update is just coincidental. There was only one minor cosmetic change between beta4 and the release version. Go back to beta4 to confirm whether or not the problem goes away.

Actually I did not use betas at all, and this problem existed in 386.8 too, but only rarely. That's why I did not post it here. It wasn't too serious. But now it is. I will try to go back to 386.7 if I find my saved settings for that build.
I may start all over from scratch with this build... I'll see.

 

[JohnDeere]

Yes I did.
And the only change I made to the network in the last week was the firmware upgrade.
I've started to collect the logs to have a clearer picture of the problem.

Hmm... Interesting. Same here. Unfortunately I don't see anything related to the cause of problem in my logs. Do you plan to setup your router from scratch with this build?

 

[ColinTaylor]

Actually I did not use betas at all..

I was replying to @DocUmibozu.

 

[DocUmibozu]

Hmm... Interesting. Same here. Unfortunately I don't see anything related to the cause of problem in my logs. Do you plan to setup your router from scratch with this build?

Well, setting up for scratch is a real pain for me... For now I'll monitor it to see if is a real problem or only an occasional glitch

 

[JohnDeere]

I was replying to @DocUmibozu.

Oh, sorry!

 

[JohnDeere]

Well, setting up for scratch is a real pain for me... For now I'll monitor it to see if is a real problem or only an occasional glitch

I manually forced this error to occur and this is what I got in my AX88U log. By the way: can you also force it to happen? I do this by downloading huge test files and wait for a few minutes and it will surely disconnect.

Dec 08 19:52:40 RT-AX88U WAN(0)_Connection WAN(0) link down.
Dec 08 19:53:02 RT-AX88U hostapd eth6: STA 40:9f:38:31:68:d2 WPA: group key handshake completed (RSN)
Dec 08 19:53:02 RT-AX88U hostapd eth6: STA 54:55:d5:8e:79:3d WPA: group key handshake completed (RSN)
Dec 08 19:53:22 RT-AX88U rc_service wanduck 1032:notify_rc restart_wan_if 0
Dec 08 19:53:22 RT-AX88U custom_script Running /jffs/scripts/service-event (args: restart wan_if)
Dec 08 19:53:22 RT-AX88U ovpn-server1 event_wait : Interrupted system call (code=4)
Dec 08 19:53:22 RT-AX88U ovpn-server1 Closing TUN/TAP interface
Dec 08 19:53:22 RT-AX88U ovpn-server1 /usr/sbin/ip addr del dev tun21 10.8.0.1/24
Dec 08 19:53:22 RT-AX88U lldpd removal request for address of 10.8.0.1%25, but no knowledge of it
Dec 08 19:53:22 RT-AX88U ovpn-server1 ovpn-down 1 server tun21 1500 1621 10.8.0.1 255.255.255.0 init
Dec 08 19:53:22 RT-AX88U ovpn-server1 PLUGIN_CLOSE: /usr/lib/openvpn-plugin-auth-pam.so
Dec 08 19:53:22 RT-AX88U ovpn-server1 PLUGIN AUTH-PAM: Error signaling background process to exit: Connection refused (errno=111)
Dec 08 19:53:22 RT-AX88U ovpn-server1 SIGTERM[hard,] received, process exiting
Dec 08 19:53:22 RT-AX88U ovpn-server2 event_wait : Interrupted system call (code=4)
Dec 08 19:53:22 RT-AX88U ovpn-server2 Closing TUN/TAP interface
Dec 08 19:53:22 RT-AX88U ovpn-server2 /usr/sbin/ip addr del dev tun22 10.16.0.1/24
Dec 08 19:53:22 RT-AX88U lldpd removal request for address of 10.16.0.1%26, but no knowledge of it
Dec 08 19:53:22 RT-AX88U ovpn-server2 ovpn-down 2 server tun22 1500 1621 10.16.0.1 255.255.255.0 init
Dec 08 19:53:22 RT-AX88U ovpn-server2 PLUGIN_CLOSE: /usr/lib/openvpn-plugin-auth-pam.so
Dec 08 19:53:22 RT-AX88U ovpn-server2 PLUGIN AUTH-PAM: Error signaling background process to exit: Connection refused (errno=111)
Dec 08 19:53:22 RT-AX88U ovpn-server2 SIGTERM[hard,] received, process exiting
Dec 08 19:53:22 RT-AX88U lldpd removal request for address of 84.xxx.xxx.177%12, but no knowledge of it
Dec 08 19:53:25 RT-AX88U wan finish adding multi routes
Dec 08 19:53:25 RT-AX88U Skynet Startup Initiated... ( skynetloc=/tmp/mnt/ext4/skynet )
Dec 08 19:53:25 RT-AX88U rc_service udhcpc_wan 13263:notify_rc start_vpnserver1
Dec 08 19:53:25 RT-AX88U rc_service udhcpc_wan 13263:notify_rc start_vpnserver2
Dec 08 19:53:25 RT-AX88U rc_service waitting "start_vpnserver1" via udhcpc_wan ...
Dec 08 19:53:25 RT-AX88U custom_script Running /jffs/scripts/service-event (args: start vpnserver1)
Dec 08 19:53:25 RT-AX88U ovpn-server1 OpenVPN 2.5.7 arm-buildroot-linux-gnueabi [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [MH/PKTINFO] [AEAD] built on Dec 3 2022
Dec 08 19:53:25 RT-AX88U ovpn-server1 library versions: OpenSSL 1.1.1s 1 Nov 2022, LZO 2.08
Dec 08 19:53:25 RT-AX88U ovpn-server1 NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Dec 08 19:53:25 RT-AX88U ovpn-server1 PLUGIN AUTH-PAM: initialization succeeded (fg)
Dec 08 19:53:25 RT-AX88U ovpn-server1 PLUGIN_INIT: POST /usr/lib/openvpn-plugin-auth-pam.so '[/usr/lib/openvpn-plugin-auth-pam.so] [openvpn]' intercepted=PLUGIN_AUTH_USER_PASS_VERIFY
Dec 08 19:53:25 RT-AX88U ovpn-server1 Diffie-Hellman initialized with 2048 bit key
Dec 08 19:53:25 RT-AX88U ovpn-server1 Outgoing Control Channel Authentication: Using 512 bit message hash 'SHA512' for HMAC authentication
Dec 08 19:53:25 RT-AX88U ovpn-server1 Incoming Control Channel Authentication: Using 512 bit message hash 'SHA512' for HMAC authentication
Dec 08 19:53:25 RT-AX88U ovpn-server1 TUN/TAP device tun21 opened
Dec 08 19:53:25 RT-AX88U ovpn-server1 TUN/TAP TX queue length set to 1000
Dec 08 19:53:25 RT-AX88U ovpn-server1 /usr/sbin/ip link set dev tun21 up mtu 1500
Dec 08 19:53:25 RT-AX88U vpnserver1 PLUGIN AUTH-PAM: BACKGROUND: initialization succeeded
Dec 08 19:53:25 RT-AX88U ovpn-server1 /usr/sbin/ip link set dev tun21 up
Dec 08 19:53:25 RT-AX88U ovpn-server1 /usr/sbin/ip addr add dev tun21 10.8.0.1/24
Dec 08 19:53:25 RT-AX88U ovpn-server1 ovpn-up 1 server tun21 1500 1621 10.8.0.1 255.255.255.0 init
Dec 08 19:53:25 RT-AX88U ovpn-server1 Socket Buffers: R=[524288->524288] S=[524288->524288]
Dec 08 19:53:25 RT-AX88U ovpn-server1 UDPv4 link local (bound): [AF_INET][undef]:xxxxxxxxx
Dec 08 19:53:25 RT-AX88U ovpn-server1 UDPv4 link remote: [AF_UNSPEC]
Dec 08 19:53:25 RT-AX88U ovpn-server1 MULTI: multi_init called, r=256 v=256
Dec 08 19:53:25 RT-AX88U ovpn-server1 IFCONFIG POOL IPv4: base=10.8.0.2 size=253
Dec 08 19:53:25 RT-AX88U ovpn-server1 Initialization Sequence Completed
Dec 08 19:53:26 RT-AX88U rc_service wanduck 1032:notify_rc restart_wan_line 1
Dec 08 19:53:26 RT-AX88U custom_script Running /jffs/scripts/service-event (args: restart wan_line)
Dec 08 19:53:26 RT-AX88U custom_script Running /jffs/scripts/firewall-start (args: eth4)
Dec 08 19:53:26 RT-AX88U wan finish adding multi routes
Dec 08 19:53:26 RT-AX88U Skynet [*] Lock File Detected (start skynetloc=/tmp/mnt/ext4/skynet) (pid=13270) - Exiting (cpid=13646)
Dec 08 19:53:26 RT-AX88U Diversion added type 65 blocking rules in iptables and ip6tables
Dec 08 19:53:26 RT-AX88U ovpn-server1 event_wait : Interrupted system call (code=4)
Dec 08 19:53:26 RT-AX88U ovpn-server1 Closing TUN/TAP interface
Dec 08 19:53:26 RT-AX88U ovpn-server1 /usr/sbin/ip addr del dev tun21 10.8.0.1/24
Dec 08 19:53:26 RT-AX88U lldpd removal request for address of 10.8.0.1%29, but no knowledge of it
Dec 08 19:53:26 RT-AX88U ovpn-server1 ovpn-down 1 server tun21 1500 1621 10.8.0.1 255.255.255.0 init
Dec 08 19:53:26 RT-AX88U ovpn-server1 PLUGIN_CLOSE: /usr/lib/openvpn-plugin-auth-pam.so
Dec 08 19:53:26 RT-AX88U ovpn-server1 PLUGIN AUTH-PAM: Error signaling background process to exit: Connection refused (errno=111)
Dec 08 19:53:26 RT-AX88U ovpn-server1 SIGTERM[hard,] received, process exiting

 

[BigScream]

I've also been having this issue as well, I actually made an account to report this.

The issue was very intermittent on 386.4, but on 388.1 its happening once an hour sometimes. The router I'm using is a AX88U. Ill keep monitoring the logs to see what I find.

 

[Viktor Jaep]

Question for those in the know... how does your standard router (or AX88U) determine if the WAN link is down? Is there actually a link up/down kind of alarm that sets it off, or is it sending some kind of regular heartbeat signal out, and if it doesn't get a response, goes down? Just wondering how this truly functions?

 

[BigScream]

I've also been having this issue as well, I actually made an account to report this.

The issue was very intermittent on 386.4, but on 388.1 its happening once an hour sometimes. The router I'm using is a AX88U. Ill keep monitoring the logs to see what I find.

Just wanted to report ever since downgrading the issue still persists.

It happens on the hour EXACTLY. When it happens the router's admin page is also no longer responsive, nor will it load.