Starting OpenVPN with TAP Client seems to crash RT-AX86U

[AndrewL733]

I just purchased an RT-AX86U PRO to replace an aging RT-AC68U and running an OpenVPN client with a TAP bridge seems to crash the RT-AX86U PRO. Both routers are on the latest versions of ASUSWRT-MERLIN -- 388.2_2 and 386.11, respectively.

Previously, the RT-AC68U was set up in SPAIN with at TAP bridge to my house in the USA (where I have an RT-AC86U -- note, not AX). This setup has been working flawlessly for several years and has allowed me to use things like Apple TimeMachine, DNLA media servers, and security system monitoring that requires broadcast traffic to be transmitted between the two locations. (The overhead isn't too bad -- a 5-20 KB/sec of broadcast traffic fairly constantly.)

However, when I try to start the OpenVPN client on the new RT-AX86U PRO, the router crashes and reboots. I am using the exact same OVPN config file on the old RT-AC68U and on the new RT-AX86U PRO. It looks to me that there might be an issue between the TAP driver and the new Hardware on the RT-AX86U PRO. While monitoring the syslog over ssh while starting openvpn client, I managed to capture the beginning of the kernel crash. I tried reverting my AX86U PRO to the first Asuswrt-Merlin version that supported this router and got the same crash.

PLEASE SEE THE ATTACHED LOGS FOR THE RT-AX86U PRO AND THE RT-AC68U. I have removed my USA IP address and substituted with with [MY USA IP ADDRESS]

Here's the kernel crash:

May 21 03:09:25 kernel: Unable to handle kernel NULL pointer dereference at virtual address 00000000000002b8
May 21 03:09:25 kernel: Mem abort info:
May 21 03:09:25 kernel: ESR = 0x96000007
May 21 03:09:25 kernel: Exception class = DABT (current EL), IL = 32 bits
May 21 03:09:25 kernel: SET = 0, FnV = 0
May 21 03:09:25 kernel: EA = 0, S1PTW = 0
May 21 03:09:25 kernel: Data abort info:
May 21 03:09:25 kernel: ISV = 0, ISS = 0x00000007
May 21 03:09:25 kernel: CM = 0, WnR = 0
May 21 03:09:25 kernel: user pgtable: 4k pages, 39-bit VAs, pgdp = 0000000000e273ba
May 21 03:09:25 kernel: [00000000000002b8] pgd=0000000025027003, pud=0000000025027003, pmd=000000002534c003, pte=0000000000000000

One difference I note is that on the RT-AC68U, there are NOT any lines in the syslog such as
May 21 03:09:19 kernel: br0: port 8(tap11) entered blocking state
May 21 03:09:19 kernel: br0: port 8(tap11) entered disabled state
May 21 03:09:19 kernel: device tap11 entered promiscuous mode

This may have to do with the logging level set. On the RT-AX86U PRO I set logging to capture absolutely everything to try to debug this problem.

Does anybody have a clue what's going wrong with the RT-AX86U PRO? Is it just broken for OpenVPN TAP bridges?

Regards,
Andrew

 

[AndrewL733]

For what it's worth, the RT-AX86U Pro running the stock ASUS firmware also crashes when starting up the VPN. It takes a bit longer for the crash to occur, but after about 60 seconds the router reboots. Before rebooting, I can see in the VPN FUSION GUI (the only place to configure a VPN Client) a message that says "routing conflict". This doesn't make sense, as the VPN configuration works absolutely fine on the RT-AC68U. So, the problem isn't specifically related to ASUSWRT-MERLIN. It seems to be a problem with the ASUS hardware.

 

[octopus]

For what it's worth, the RT-AX86U Pro running the stock ASUS firmware also crashes when starting up the VPN. It takes a bit longer for the crash to occur, but after about 60 seconds the router reboots. Before rebooting, I can see in the VPN FUSION GUI (the only place to configure a VPN Client) a message that says "routing conflict". This doesn't make sense, as the VPN configuration works absolutely fine on the RT-AC68U. So, the problem isn't specifically related to ASUSWRT-MERLIN. It seems to be a problem with the ASUS hardware.

Look at your log, that hint you.

 

[AndrewL733]

@octopus, what do you see in the log that explains the issue?

 

[octopus]

@octopus, what do you see in the log that explains the issue?

That hint you what problem is with your vpn connection.

 

[AndrewL733]

@octupus, please allow me to be more precise. Can you please say which line in the log do you believe explains the problem? And do you not see that same line in the log for the RT-AC68U, which works fine with the same openvpn client configuration.

 

[AndrewL733]

Also, to be clear. The OpenVPN Server and Client configurations were created by the Asuswrt-Merlin OpenVPN user interface running on the my RT-AC86U. So if there is something wrong with the configuration, the Asuswrt-Merlin openvpn management may be the source of the issue.

 

[RMerlin]

So if there is something wrong with the configuration, the Asuswrt-Merlin openvpn management may be the source of the issue.

No. If something is wrong with the configuration, then it`s because the user pasted something invalid in the Custom field. Which lately is always the issue, people importing config files that contain keywords that are no longer valid with OpenVPN 2.6.

 

[AndrewL733]

the user pasted something invalid in the Custom field. Which lately is always the issue, people importing config files that contain keywords that are no longer valid with OpenVPN 2.6.

I’m not an expert here but as you can see from the logs, both routers are running the same version of OpenVPN -- 2.6.3. The configuration works on the RT-AC68U. The RT-AX86U Pro crashes. Also, the latest stock ASUS firmware for the RT-AX86U Pro includes a much older version of OpenVPN -- 2.4.12 -- and when running that firmware, the router also crashes upon starting up the VPN client. So, it doesn't really make sense when you suggest that the crashing has something to do with the configuration file. What makes more sense is that the TAP kernel driver doesn't work with the RT-AX86U and that's why it's causing a kernel oops.

I'm wondering what these lines in the syslog mean on the RT-AX86U Pro when the vpn client is starting up:

May 21 03:09:19 kernel: tun: Universal TUN/TAP device driver, 1.6
May 21 03:09:19 kernel: br0: port 8(tap11) entered blocking state
May 21 03:09:19 kernel: br0: port 8(tap11) entered disabled state
May 21 03:09:19 kernel: device tap11 entered promiscuous mode
May 21 03:09:22 kernel: br0: port 8(tap11) entered learning state
May 21 03:09:24 kernel: br0: port 8(tap11) entered forwarding state
May 21 03:09:24 kernel: br0: topology change detected, sending tcn bpdu

Those occur right before the crash. I don't see those same lines in the syslog of the RT-AC68U

Andy

 

[AndrewL733]

Hi @RMerlin. Thanks for chiming in above.

No. If something is wrong with the configuration, then it`s because the user pasted something invalid in the Custom field. Which lately is always the issue, people importing config files that contain keywords that are no longer valid with OpenVPN 2.6.

I realize you are all volunteering here but I'm a little disappointed in the responses I've gotten.

Perhaps you missed my comment earlier in this thread. My OpenVPN Client Configuration works fine on an RT-AC68U that has been upgraded to the latest Merlin FW. With that update, both the AC68U and the AX86U Pro have the exact same version of OpenVPN -- 2.6.3. The fact that my configuration works on the AC68U and crashes the AX86U (May 21 03:09:25 kernel: Unable to handle kernel NULL pointer dereference at virtual address 00000000000002b8) suggests the issue is unrelated to the version of OpenVPN and keyword compatibility, because if that were the case I would expect the AC68U to crash as well. Do you disagree with that statement? If so, I'd like to understand why. Also, just so you know, I haven't pasted anything in the custom field. There are a few lines there, but they were specified by the opvn file created by the Server router.

Also, the fact that the same configuration crashes the AX86U Pro when it's running stock ASUS FW (which has OpenVPN 2.4.12) makes me think the issue is a hardware/driver interaction. I feel like the router is crashing shortly after loading the TAP driver.

Unfortunately, the technical support folks at ASUS have not been very helpful -- I think this kind of issue is just way beyond what any ASUS support person can offer on the phone. The best they can suggest is to swap routers and see if the problem still occurs with a replacement one. I'm sort of skeptical that swapping routers is going to solve the problem, because everything else on the AX86U Pro seemed to work beautifully. But, I'm going to give it a try because I'm at a loss for what else to do.

Anyway, thanks again for any assistance you can provide.

 

[ColinTaylor]

Can you confirm the details of the firmware version on the server router, the RT-AC86U. Also, the OpenVPN version it uses.

You mentioned an error message related to routing. When you changed to the RT-AX86U Pro as the VPN client did you also use its default subnet (192.168.50.x) instead of the RT-AC68U's default subnet (192.68.1.x)? Is the RT-AC86U also using 192.168.50.x?

 

[AndrewL733]

Hi Colin,

Thanks also for chiming in. The “server” router is running 386.10, so that has OpenVPN 2.6.0. I’m going to update it tomorrow to 386.11 once my kids finish with their remote work for the day. But, it’s already on 2.6.x so I don’t think that should matter.

My VPN setup is unusual. I’m using “Ethernet layer 2 Bridging” — tap — because I need broadcast traffic to go over the vpn. The overhead is acceptable.

Both sides are on the same subnet. 192.168.15.x. Side A has 100-179 for dhcp and Side B has 180-254. There are also
a number of static addresses in the 1-99 range. No duplicates. I have a script that prevents each side from seeing the other side’s dhcp server. It works very well with my existing routers. It’s also very easy to direct devices (such as my AppleTVs) to use one side or the other simple by changing the gateway between 15.1 to 15.2. Given that one side is in the USA and the other in Spain, it makes for a seamless connection between two worlds (and security cameras and media servers and a Linux box providing Apple TimeMachine backups).

 

[AndrewL733]

Hello again @ColinTaylor @RMerlin,

I received a replacement RT-AX86U Pro this morning and set it up with the default ASUS FW (including OpenVPN 2.4.12). Once again, all the basic functions work, but when I import my OVPN file and enable the VPN, I was able to observe the same kernel crash again. I tailed the syslog over ssh (otherwise I would miss the kernel oops).

May 27 03:49:06 wlceventd: wlceventd_proc_event(530): eth7: Auth 40:5D:82:28:90:57, status: Successful (0), rssi:-65
May 27 03:49:06 wlceventd: wlceventd_proc_event(559): eth7: Assoc 40:5D:82:28:90:57, status: Successful (0), rssi:-65
May 27 03:49:11 wlceventd: wlceventd_proc_event(530): wl0.1: Auth C87:78:0F:8D:13, status: Successful (0), rssi:0
May 27 03:49:11 wlceventd: wlceventd_proc_event(559): wl0.1: Assoc C87:78:0F:8D:13, status: Successful (0), rssi:-57
May 27 03:49:40 rc_service: conn_diag 7401:notify_rc restart_amas_portstatus
May 27 03:50:38 wlceventd: wlceventd_proc_event(494): eth6: Deauth_ind 5C:E5:0C:C3:35:8C, status: 0, reason: Deauthenticated because sending station is leaving (or has left) IBSS or ESS (3), rssi:0
May 27 03:50:38 wlceventd: wlceventd_proc_event(511): eth6: Disassoc 5C:E5:0C:C3:35:8C, status: 0, reason: Disassociated because sending station is leaving (or has left) BSS (8), rssi:0
May 27 03:50:46 wlceventd: wlceventd_proc_event(530): eth6: Auth 5C:E5:0C:C3:35:8C, status: Successful (0), rssi:0
May 27 03:50:46 wlceventd: wlceventd_proc_event(559): eth6: Assoc 5C:E5:0C:C3:35:8C, status: Successful (0), rssi:-52
May 27 03:51:14 dropbear[8243]: Password auth succeeded for 'admin' from 192.168.15.211:50428
May 27 03:52:31 rc_service: httpd 7358:notify_rc restart_vpnc;restart_default_wan;
May 27 03:52:33 vpnclient5[8521]: OpenVPN 2.4.12 arm-buildroot-linux-gnueabi [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [MH/PKTINFO] [AEAD] built on Oct 12 2022
May 27 03:52:33 vpnclient5[8521]: library versions: OpenSSL 1.1.1n 15 Mar 2022, LZO 2.10
May 27 03:52:33 vpnclient5[8522]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
May 27 03:52:33 vpnclient5[8522]: TCP/UDP: Preserving recently used remote address: [AF_INET](My remote IP address -- replace by text for security):1194
May 27 03:52:33 vpnclient5[8522]: Socket Buffers: R=[524288->524288] S=[524288->524288]
May 27 03:52:33 vpnclient5[8522]: UDP link local: (not bound)
May 27 03:52:33 vpnclient5[8522]: UDP link remote: [AF_INET](My remote IP address -- replace by text for security):1194
May 27 03:52:33 vpnclient5[8522]: TLS: Initial packet from [AF_INET](My remote IP address -- replace by text for security):1194, sid=141f2619 fa8bcb51
May 27 03:52:33 vpnclient5[8522]: VERIFY OK: depth=1, C=TW, ST=TW, L=Taipei, O=ASUS, CN=RT-AC86U, emailAddress=me@myhost.mydomain
May 27 03:52:33 vpnclient5[8522]: VERIFY KU OK
May 27 03:52:33 vpnclient5[8522]: Validating certificate extended key usage
May 27 03:52:33 vpnclient5[8522]: ++ Certificate has EKU (str) TLS Web Server Authentication, expects TLS Web Server Authentication
May 27 03:52:33 vpnclient5[8522]: VERIFY EKU OK
May 27 03:52:33 vpnclient5[8522]: VERIFY OK: depth=0, C=TW, ST=TW, L=Taipei, O=ASUS, CN=RT-AC86U, emailAddress=me@myhost.mydomain
May 27 03:52:34 vpnclient5[8522]: WARNING: 'link-mtu' is used inconsistently, local='link-mtu 1586', remote='link-mtu 1602'
May 27 03:52:34 vpnclient5[8522]: WARNING: 'tun-mtu' is used inconsistently, local='tun-mtu 1532', remote='tun-mtu 1500'
May 27 03:52:34 vpnclient5[8522]: WARNING: 'cipher' is used inconsistently, local='cipher BF-CBC', remote='cipher AES-128-CBC'
May 27 03:52:34 vpnclient5[8522]: Control Channel: TLSv1.3, cipher TLSv1.3 TLS_AES_256_GCM_SHA384, 1024 bit RSA
May 27 03:52:34 vpnclient5[8522]: [RT-AC86U] Peer Connection Initiated with [AF_INET](My remote IP address -- replace by text for security):1194
May 27 03:52:35 vpnclient5[8522]: SENT CONTROL [RT-AC86U]: 'PUSH_REQUEST' (status=1)
May 27 03:52:35 vpnclient5[8522]: PUSH: Received control message: 'PUSH_REPLY,route 0.0.0.0 255.255.255.255 net_gateway,route-gateway dhcp,ping 15,ping-restart 60,peer-id 0,cipher AES-256-GCM'
May 27 03:52:35 vpnclient5[8522]: OPTIONS IMPORT: timers and/or timeouts modified
May 27 03:52:35 vpnclient5[8522]: OPTIONS IMPORT: route options modified
May 27 03:52:35 vpnclient5[8522]: OPTIONS IMPORT: route-related options modified
May 27 03:52:35 vpnclient5[8522]: OPTIONS IMPORT: peer-id set
May 27 03:52:35 vpnclient5[8522]: OPTIONS IMPORT: adjusting link_mtu to 1657
May 27 03:52:35 vpnclient5[8522]: OPTIONS IMPORT: data channel crypto options modified
May 27 03:52:35 vpnclient5[8522]: Data Channel: using negotiated cipher 'AES-256-GCM'
May 27 03:52:35 vpnclient5[8522]: Outgoing Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key
May 27 03:52:35 vpnclient5[8522]: Incoming Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key
May 27 03:52:35 vpnclient5[8522]: TUN/TAP device tap15 opened
May 27 03:52:35 vpnclient5[8522]: TUN/TAP TX queue length set to 100
May 27 03:52:35 vpnclient5[8522]: /etc/openvpn/ovpnc-up 5 tap15 1500 1585 init
May 27 03:52:35 vpnclient5: WARNING: Replace default vpn gateway by using 0.0.0.0/1 and 128.0.0.0/1
May 27 03:52:35 vpnclient5[8522]: WARNING: this configuration may cache passwords in memory -- use the auth-nocache option to prevent this
May 27 03:52:35 vpnclient5[8522]: Initialization Sequence Completed
May 27 03:52:47 kernel: Unable to handle kernel NULL pointer dereference at virtual address 00000000000002b8
May 27 03:52:47 kernel: Mem abort info:
May 27 03:52:47 kernel: ESR = 0x96000007
May 27 03:52:47 kernel: Exception class = DABT (current EL), IL = 32 bits
May 27 03:52:47 kernel: SET = 0, FnV = 0
May 27 03:52:47 kernel: EA = 0, S1PTW = 0
May 27 03:52:47 kernel: Data abort info:
May 27 03:52:47 kernel: ISV = 0, ISS = 0x00000007
May 27 03:52:47 kernel: CM = 0, WnR = 0
May 27 03:52:47 kernel: user pgtable: 4k pages, 39-bit VAs, pgdp = 000000000963c35a
May 27 03:52:47 kernel: [00000000000002b8] pgd=000000003390f003, pud=000000003390f003, pmd=000000002171a003, pte=0000000000000000

Again, this is with the stock ASUS FW, but I saw the exact same thing with asuswrt-merlin on the original router. I do not get this crash on the RT-AC68U even running the latest 386.11 asuswrt-merlin firmware. As I have said multiple times in this thread, I believe the issue is the tap driver causing a kernel crash -- and that the crash has nothing to do with the client configuration file.

The ovpn client configuration file was generated by the RT-AC86U on the other side of the Atlantic, and I didn't add any custom options to it, I'm not sure what options to remove from the ovpn client configuration file to see if they are causing issues. But here is the client ovpn file (without the certificates, of course).

# Config generated by Asuswrt-Merlin 386.10, requires OpenVPN 2.4.0 or newer.client

dev tap
# Windows needs the TAP-Win32 adapter name
# from the Network Connections panel
# if you have more than one.
;dev-node MyTap
proto udp
remote (My remote DYNDNS Host Name, deleted for privacy) 1194
resolv-retry infinite
nobind
float
ncp-ciphers AES-256-GCM:AES-128-GCM:AES-256-CBC:AES-128-CBC
auth SHA256
comp-lzo adaptive
keepalive 15 60
remote-cert-tls server

 

[AndrewL733]

Okay, so I tried the following experiment. After disabling the OpenVPN server on the Boston/USA side, I updated my new RT-AX86U Pro router on the Spain side to asuswrt-merlin 388.2 and enabled OpenVPN server on it and exported the client OVPN config file. Then I imported the OVPN on the Boston/USA side (on the RT-AC86U to set up the client there). As soon as I tried to connect to Spain, the RT-AX86U Pro (the new router in Spain) crashed with the same kernel panic I saw when I was trying to set it up as the client for the Boston server. So in other words, the RT-AC86U Pro crashes whether it is the server or the client in the bridge.

So, I think this lends evidence to the idea that USING the tap adapter on the RT-AX86U Pro causes the kernel panic.

Here is the complete log from the Spain RT-AX86U Pro when it is the bridge server rather than the bridge client:

admin@RT-AX86U_Pro:/tmp/home/root# tail -f /jffs/syslog.log
May 27 13:15:54 dropbear[5150]: Password auth succeeded for 'admin' from 192.168.15.211:56178
May 27 13:16:12 rc_service: httpd 2448:notify_rc restart_chpass;restart_vpnserver1
May 27 13:16:13 ovpn-server1[5264]: OpenVPN 2.6.3 arm-buildroot-linux-gnueabi [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [MH/PKTINFO] [AEAD]
May 27 13:16:13 ovpn-server1[5264]: library versions: OpenSSL 1.1.1t 7 Feb 2023, LZO 2.10
May 27 13:16:13 ovpn-server1[5265]: NOTE: when bridging your LAN adapter with the TAP adapter, note that the new bridge adapter will often take on its own IP address that is different from what the LAN adapter was previously set to
May 27 13:16:13 ovpn-server1[5265]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
May 27 13:16:13 ovpn-server1[5265]: Diffie-Hellman initialized with 2048 bit key
May 27 13:16:13 ovpn-server1[5265]: TUN/TAP device tap21 opened
May 27 13:16:13 ovpn-server1[5265]: TUN/TAP TX queue length set to 1000
May 27 13:16:13 ovpn-server1[5265]: ovpn-up 1 server tap21 1500 0 init
May 27 13:16:13 ovpn-server1[5265]: Socket Buffers: R=[524288->524288] S=[524288->524288]
May 27 13:16:13 ovpn-server1[5265]: UDPv4 link local (bound): [AF_INET][undef]:1194
May 27 13:16:13 ovpn-server1[5265]: UDPv4 link remote: [AF_UNSPEC]
May 27 13:16:13 ovpn-server1[5265]: MULTI: multi_init called, r=256 v=256
May 27 13:16:13 ovpn-server1[5265]: Initialization Sequence Completed
May 27 13:23:27 wlceventd: wlceventd_proc_event(530): eth6: Auth 84:F3:EB:86:88:FE, status: Successful (0), rssi:0
May 27 13:23:27 wlceventd: wlceventd_proc_event(559): eth6: Assoc 84:F3:EB:86:88:FE, status: Successful (0), rssi:-45
May 27 13:23:32 wlceventd: wlceventd_proc_event(511): eth6: Disassoc 84:F3:EB:86:88:FE, status: 0, reason: Disassociated because sending station is leaving (or has left) BSS (8), rssi:0
May 27 13:23:32 wlceventd: wlceventd_proc_event(494): eth6: Deauth_ind 84:F3:EB:86:88:FE, status: 0, reason: Deauthenticated because sending station is leaving (or has left) IBSS or ESS (3), rssi:0
May 27 13:24:26 ovpn-server1[5265]: (Boston client IP -- removed for privacy):38229 VERIFY OK: depth=1, C=TW, ST=TW, L=Taipei, O=ASUS, CN=RT-AX86U_Pro, emailAddress=me@myhost.mydomain
May 27 13:24:26 ovpn-server1[5265]: (Boston client IP -- removed for privacy):38229 VERIFY OK: depth=0, C=TW, ST=TW, L=Taipei, O=ASUS, CN=client, emailAddress=me@myhost.mydomain
May 27 13:24:26 ovpn-server1[5265]: (Boston client IP -- removed for privacy):38229 peer info: IV_VER=2.6.3
May 27 13:24:26 ovpn-server1[5265]: (Boston client IP -- removed for privacy):38229 peer info: IV_PLAT=linux
May 27 13:24:26 ovpn-server1[5265]: (Boston client IP -- removed for privacy):38229 peer info: IV_TCPNL=1
May 27 13:24:26 ovpn-server1[5265]: (Boston client IP -- removed for privacy):38229 peer info: IV_MTU=1600
May 27 13:24:26 ovpn-server1[5265]: (Boston client IP -- removed for privacy):38229 peer info: IV_NCP=2
May 27 13:24:26 ovpn-server1[5265]: (Boston client IP -- removed for privacy):38229 peer info: IV_CIPHERS=AES-256-GCM:AES-128-GCM:AES-256-CBC:AES-128-CBC:CHACHA20-POLY1305
May 27 13:24:26 ovpn-server1[5265]: (Boston client IP -- removed for privacy):38229 peer info: IV_PROTO=990
May 27 13:24:26 ovpn-server1[5265]: (Boston client IP -- removed for privacy):38229 peer info: IV_LZO_STUB=1
May 27 13:24:26 ovpn-server1[5265]: (Boston client IP -- removed for privacy):38229 peer info: IV_COMP_STUB=1
May 27 13:24:26 ovpn-server1[5265]: (Boston client IP -- removed for privacy):38229 peer info: IV_COMP_STUBv2=1
May 27 13:24:26 ovpn-server1[5265]: (Boston client IP -- removed for privacy):38229 TLS: move_session: dest=TM_ACTIVE src=TM_INITIAL reinit_src=1
May 27 13:24:26 ovpn-server1[5265]: (Boston client IP -- removed for privacy):38229 TLS: tls_multi_process: initial untrusted session promoted to trusted
May 27 13:24:26 ovpn-server1[5265]: (Boston client IP -- removed for privacy):38229 Control Channel: TLSv1.3, cipher TLSv1.3 TLS_AES_256_GCM_SHA384, peer certificate: 1024 bit RSA, signature: RSA-SHA256
May 27 13:24:26 ovpn-server1[5265]: (Boston client IP -- removed for privacy):38229 [client] Peer Connection Initiated with [AF_INET](Boston client IP -- removed for privacy):38229 (via [AF_INET]88.1.204.132%ppp0)
May 27 13:24:26 ovpn-server1[5265]: client/(Boston client IP -- removed for privacy):38229 MULTI: no dynamic or static remote--ifconfig address is available for client/(Boston client IP -- removed for privacy):38229
May 27 13:24:26 ovpn-server1[5265]: client/(Boston client IP -- removed for privacy):38229 SENT CONTROL [client]: 'PUSH_REPLY,route 0.0.0.0 255.255.255.255 net_gateway,route-gateway dhcp,ping 15,ping-restart 60,peer-id 0,cipher AES-256-GCM,protocol-flags cc-exit tls-ekm dyn-tls-crypt,tun-mtu 1500' (status=1)
May 27 13:24:26 ovpn-server1[5265]: client/(Boston client IP -- removed for privacy):38229 MULTI: Learn: 4a:b1:c3:00:a3:5f@0 -> client/(Boston client IP -- removed for privacy):38229
May 27 13:24:27 ovpn-server1[5265]: client/(Boston client IP -- removed for privacy):38229 Data Channel: cipher 'AES-256-GCM', peer-id: 0
May 27 13:24:27 ovpn-server1[5265]: client/(Boston client IP -- removed for privacy):38229 Timers: ping 15, ping-restart 120
May 27 13:24:27 ovpn-server1[5265]: client/(Boston client IP -- removed for privacy):38229 Protocol options: protocol-flags cc-exit tls-ekm dyn-tls-crypt
May 27 13:24:31 ovpn-server1[5265]: client/(Boston client IP -- removed for privacy):38229 MULTI: Learn: 3c:ec:ef:d9:23:15@0 -> client/(Boston client IP -- removed for privacy):38229
May 27 13:24:31 ovpn-server1[5265]: client/(Boston client IP -- removed for privacy):38229 MULTI: Learn: 4c:ed:fb:8f:dd:28@0 -> client/(Boston client IP -- removed for privacy):38229
May 27 13:24:31 ovpn-server1[5265]: client/(Boston client IP -- removed for privacy):38229 MULTI: Learn: 54:e6:fc:f9:56:d0@0 -> client/(Boston client IP -- removed for privacy):38229
May 27 13:24:31 ovpn-server1[5265]: client/(Boston client IP -- removed for privacy):38229 MULTI: Learn: 00:a0fd:ad:41@0 -> client/(Boston client IP -- removed for privacy):38229
May 27 13:24:34 kernel: Unable to handle kernel NULL pointer dereference at virtual address 00000000000002b8
May 27 13:24:34 kernel: Mem abort info:
May 27 13:24:34 kernel: ESR = 0x96000007
May 27 13:24:34 kernel: Exception class = DABT (current EL), IL = 32 bits
May 27 13:24:34 kernel: SET = 0, FnV = 0
May 27 13:24:34 kernel: EA = 0, S1PTW = 0
May 27 13:24:34 kernel: Data abort info:
May 27 13:24:34 kernel: ISV = 0, ISS = 0x00000007
May 27 13:24:34 kernel: CM = 0, WnR = 0
May 27 13:24:34 kernel: user pgtable: 4k pages, 39-bit VAs, pgdp = 000000005aa2c669
May 27 13:24:34 kernel: [00000000000002b8] pgd=000000002535b003, pud=000000002535b003, pmd=0000000025556003, pte=0000000000000000

 

[seabass]

Okay, so I tried the following experiment. After disabling the OpenVPN server on the Boston/USA side, I updated my new RT-AX86U Pro router on the Spain side to asuswrt-merlin 388.2 and enabled OpenVPN server on it and exported the client OVPN config file. Then I imported the OVPN on the Boston/USA side (on the RT-AC86U to set up the client there). As soon as I tried to connect to Spain, the RT-AX86U Pro (the new router in Spain) crashed with the same kernel panic I saw when I was trying to set it up as the client for the Boston server. So in other words, the RT-AC86U Pro crashes whether it is the server or the client in the bridge.

So, I think this lends evidence to the idea that USING the tap adapter on the RT-AX86U Pro causes the kernel panic.

Here is the complete log from the Spain RT-AX86U Pro when it is the bridge server rather than the bridge client:

admin@RT-AX86U_Pro:/tmp/home/root# tail -f /jffs/syslog.log
May 27 13:15:54 dropbear[5150]: Password auth succeeded for 'admin' from 192.168.15.211:56178
May 27 13:16:12 rc_service: httpd 2448:notify_rc restart_chpass;restart_vpnserver1
May 27 13:16:13 ovpn-server1[5264]: OpenVPN 2.6.3 arm-buildroot-linux-gnueabi [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [MH/PKTINFO] [AEAD]
May 27 13:16:13 ovpn-server1[5264]: library versions: OpenSSL 1.1.1t 7 Feb 2023, LZO 2.10
May 27 13:16:13 ovpn-server1[5265]: NOTE: when bridging your LAN adapter with the TAP adapter, note that the new bridge adapter will often take on its own IP address that is different from what the LAN adapter was previously set to
May 27 13:16:13 ovpn-server1[5265]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
May 27 13:16:13 ovpn-server1[5265]: Diffie-Hellman initialized with 2048 bit key
May 27 13:16:13 ovpn-server1[5265]: TUN/TAP device tap21 opened
May 27 13:16:13 ovpn-server1[5265]: TUN/TAP TX queue length set to 1000
May 27 13:16:13 ovpn-server1[5265]: ovpn-up 1 server tap21 1500 0 init
May 27 13:16:13 ovpn-server1[5265]: Socket Buffers: R=[524288->524288] S=[524288->524288]
May 27 13:16:13 ovpn-server1[5265]: UDPv4 link local (bound): [AF_INET][undef]:1194
May 27 13:16:13 ovpn-server1[5265]: UDPv4 link remote: [AF_UNSPEC]
May 27 13:16:13 ovpn-server1[5265]: MULTI: multi_init called, r=256 v=256
May 27 13:16:13 ovpn-server1[5265]: Initialization Sequence Completed
May 27 13:23:27 wlceventd: wlceventd_proc_event(530): eth6: Auth 84:F3:EB:86:88:FE, status: Successful (0), rssi:0
May 27 13:23:27 wlceventd: wlceventd_proc_event(559): eth6: Assoc 84:F3:EB:86:88:FE, status: Successful (0), rssi:-45
May 27 13:23:32 wlceventd: wlceventd_proc_event(511): eth6: Disassoc 84:F3:EB:86:88:FE, status: 0, reason: Disassociated because sending station is leaving (or has left) BSS (8), rssi:0
May 27 13:23:32 wlceventd: wlceventd_proc_event(494): eth6: Deauth_ind 84:F3:EB:86:88:FE, status: 0, reason: Deauthenticated because sending station is leaving (or has left) IBSS or ESS (3), rssi:0
May 27 13:24:26 ovpn-server1[5265]: (Boston client IP -- removed for privacy):38229 VERIFY OK: depth=1, C=TW, ST=TW, L=Taipei, O=ASUS, CN=RT-AX86U_Pro, emailAddress=me@myhost.mydomain
May 27 13:24:26 ovpn-server1[5265]: (Boston client IP -- removed for privacy):38229 VERIFY OK: depth=0, C=TW, ST=TW, L=Taipei, O=ASUS, CN=client, emailAddress=me@myhost.mydomain
May 27 13:24:26 ovpn-server1[5265]: (Boston client IP -- removed for privacy):38229 peer info: IV_VER=2.6.3
May 27 13:24:26 ovpn-server1[5265]: (Boston client IP -- removed for privacy):38229 peer info: IV_PLAT=linux
May 27 13:24:26 ovpn-server1[5265]: (Boston client IP -- removed for privacy):38229 peer info: IV_TCPNL=1
May 27 13:24:26 ovpn-server1[5265]: (Boston client IP -- removed for privacy):38229 peer info: IV_MTU=1600
May 27 13:24:26 ovpn-server1[5265]: (Boston client IP -- removed for privacy):38229 peer info: IV_NCP=2
May 27 13:24:26 ovpn-server1[5265]: (Boston client IP -- removed for privacy):38229 peer info: IV_CIPHERS=AES-256-GCM:AES-128-GCM:AES-256-CBC:AES-128-CBC:CHACHA20-POLY1305
May 27 13:24:26 ovpn-server1[5265]: (Boston client IP -- removed for privacy):38229 peer info: IV_PROTO=990
May 27 13:24:26 ovpn-server1[5265]: (Boston client IP -- removed for privacy):38229 peer info: IV_LZO_STUB=1
May 27 13:24:26 ovpn-server1[5265]: (Boston client IP -- removed for privacy):38229 peer info: IV_COMP_STUB=1
May 27 13:24:26 ovpn-server1[5265]: (Boston client IP -- removed for privacy):38229 peer info: IV_COMP_STUBv2=1
May 27 13:24:26 ovpn-server1[5265]: (Boston client IP -- removed for privacy):38229 TLS: move_session: dest=TM_ACTIVE src=TM_INITIAL reinit_src=1
May 27 13:24:26 ovpn-server1[5265]: (Boston client IP -- removed for privacy):38229 TLS: tls_multi_process: initial untrusted session promoted to trusted
May 27 13:24:26 ovpn-server1[5265]: (Boston client IP -- removed for privacy):38229 Control Channel: TLSv1.3, cipher TLSv1.3 TLS_AES_256_GCM_SHA384, peer certificate: 1024 bit RSA, signature: RSA-SHA256
May 27 13:24:26 ovpn-server1[5265]: (Boston client IP -- removed for privacy):38229 [client] Peer Connection Initiated with [AF_INET](Boston client IP -- removed for privacy):38229 (via [AF_INET]88.1.204.132%ppp0)
May 27 13:24:26 ovpn-server1[5265]: client/(Boston client IP -- removed for privacy):38229 MULTI: no dynamic or static remote--ifconfig address is available for client/(Boston client IP -- removed for privacy):38229
May 27 13:24:26 ovpn-server1[5265]: client/(Boston client IP -- removed for privacy):38229 SENT CONTROL [client]: 'PUSH_REPLY,route 0.0.0.0 255.255.255.255 net_gateway,route-gateway dhcp,ping 15,ping-restart 60,peer-id 0,cipher AES-256-GCM,protocol-flags cc-exit tls-ekm dyn-tls-crypt,tun-mtu 1500' (status=1)
May 27 13:24:26 ovpn-server1[5265]: client/(Boston client IP -- removed for privacy):38229 MULTI: Learn: 4a:b1:c3:00:a3:5f@0 -> client/(Boston client IP -- removed for privacy):38229
May 27 13:24:27 ovpn-server1[5265]: client/(Boston client IP -- removed for privacy):38229 Data Channel: cipher 'AES-256-GCM', peer-id: 0
May 27 13:24:27 ovpn-server1[5265]: client/(Boston client IP -- removed for privacy):38229 Timers: ping 15, ping-restart 120
May 27 13:24:27 ovpn-server1[5265]: client/(Boston client IP -- removed for privacy):38229 Protocol options: protocol-flags cc-exit tls-ekm dyn-tls-crypt
May 27 13:24:31 ovpn-server1[5265]: client/(Boston client IP -- removed for privacy):38229 MULTI: Learn: 3c:ec:ef:d9:23:15@0 -> client/(Boston client IP -- removed for privacy):38229
May 27 13:24:31 ovpn-server1[5265]: client/(Boston client IP -- removed for privacy):38229 MULTI: Learn: 4c:ed:fb:8f:dd:28@0 -> client/(Boston client IP -- removed for privacy):38229
May 27 13:24:31 ovpn-server1[5265]: client/(Boston client IP -- removed for privacy):38229 MULTI: Learn: 54:e6:fc:f9:56:d0@0 -> client/(Boston client IP -- removed for privacy):38229
May 27 13:24:31 ovpn-server1[5265]: client/(Boston client IP -- removed for privacy):38229 MULTI: Learn: 00:a0fd:ad:41@0 -> client/(Boston client IP -- removed for privacy):38229
May 27 13:24:34 kernel: Unable to handle kernel NULL pointer dereference at virtual address 00000000000002b8
May 27 13:24:34 kernel: Mem abort info:
May 27 13:24:34 kernel: ESR = 0x96000007
May 27 13:24:34 kernel: Exception class = DABT (current EL), IL = 32 bits
May 27 13:24:34 kernel: SET = 0, FnV = 0
May 27 13:24:34 kernel: EA = 0, S1PTW = 0
May 27 13:24:34 kernel: Data abort info:
May 27 13:24:34 kernel: ISV = 0, ISS = 0x00000007
May 27 13:24:34 kernel: CM = 0, WnR = 0
May 27 13:24:34 kernel: user pgtable: 4k pages, 39-bit VAs, pgdp = 000000005aa2c669
May 27 13:24:34 kernel: [00000000000002b8] pgd=000000002535b003, pud=000000002535b003, pmd=0000000025556003, pte=0000000000000000

I've had the same problem for 7+ months, unfortunately. I believe it's something in the drivers for AX class routers, I've tried two different models, just like you...

I'm currently still using my AC as my server to avoid this until hopefully one day this is fixed.

For me, having an AX class router hosting the *server* for the TAP VPN tunnel is the problem. Using an AX as a client works just fine, so long as the server is my AC.

The tunnel connects, traffic flows but moments later the server reboots...

 

[AndrewL733]

Hi @seabass. Thanks for chiming in. It's good to hear I'm not alone. As I said in my posts, I get crashing regardless of whether the AX router is the "server" or the "client". You seem to be slightly luckier, although one day I'm sure you'd like to have an AX router on both sides.

Would you mind sharing with me what AX router you have? Presumably because you're in the Asuswrt-merlin forum, it's an Asus router running merlin firmware. But it would be good to confirm that.

I have been reaching out to ASUS for tech support but they seem to be radio silent. Each time I chat with them to see what's happening with my case, they just seem to create a new case and ask me for all the details all over again.

 

[seabass]

Look here to see my story...

Internet access inconsistency and a repeatable crash with GT-AX6000 and GT-AX11000 vs my RT-AC class routers

I recently decided to upgrade my primary home router (AC5300) to an AX class router to match my remote location, which is already running an AX68U and is connected to my home via a site-to-site TAP based Open VPN connection. Since the firmware versioning between AC and AX seems to be diverging...

www.snbforums.com

Note those earlier issues regarding inconsistent connectivity disappeared in the next version onwards / order of operations during setup, but the TAP VPN crash still remains as of 388.2

Also, like you, I made a script to block DHCP crossing over the TAP tunnel. You can see that here:

How to block DHCP protocol over a TAP site-to-site VPN (useful if you host separate DHCP subnet ranges at each site as part of a superset subnet)

Crafted for an Asus router with Merlin firmware. Enable SSH access to your router (administration/system) **I don't recommend exposing SSH access to the internet unless you are really comfortable with the risk. Enable JFFS custom scripts in router (admin/system page) This example is...

www.snbforums.com

 

[seabass]

Hi @seabass. Thanks for chiming in. It's good to hear I'm not alone. As I said in my posts, I get crashing regardless of whether the AX router is the "server" or the "client". You seem to be slightly luckier, although one day I'm sure you'd like to have an AX router on both sides.

Would you mind sharing with me what AX router you have? Presumably because you're in the Asuswrt-merlin forum, it's an Asus router running merlin firmware. But it would be good to confirm that.

I have been reaching out to ASUS for tech support but they seem to be radio silent. Each time I chat with them to see what's happening with my case, they just seem to create a new case and ask me for all the details all over again.

Look here for my similar story...

Thread 'Internet access inconsistency and a repeatable crash with GT-AX6000 and GT-AX11000 vs my RT-AC class routers' https://www.snbforums.com/threads/i...d-gt-ax11000-vs-my-rt-ac-class-routers.83165/

 

[sojomy]

Hey guys, I know it's been a couple months since the last message here, but I have some information that might be related to the issue you 2 are both seeing. I'm not using a Router to router bridge, but a laptop to router bridge using the OpenVPN Windows client on the user's computer and the OpenVPN server in the Asus RT-AX86U router. We were using the Asus RT-AC86U without any issues. When this client now tries to connect to the OpenVPN Server on the RT-AX86U router, the router reboots in 1-2 seconds. I can connect to this router from other devices using the exact same OpenVPN client software, OpenVPN client config file, and the same OpenVPN User account in the router. The problem is definitely related to something on this laptop, but I can't figure out what it is.

However! Here's the part that you guys might be interested in. If I disable the 2.4GHz Radio in the router under Wireless->Professional, the problem goes away for this client! @AndrewL733 are you able to replicate this issue over and over? If so, can you try disabling the 2.4GHz radio on the RT-AX86U router and see if the router reboot issue goes away? If so, there is some link between the 2.4GHz radio and the OpenVPN Server using the TAP protocol, but I have no idea why that would be.

I can't leave the 2.4GHz radio disabled as a fix because some of the devices in our office don't support 5.0GHz. But I've tried older and newer versions of the OpenVPN client software, and I've used multiple versions of the Merlin firmware, including the one that was just release today. I cannot seem to get this issue fixed with the 2.4GHz radio enabled, but only for 1 user! I can connect from my computer outside the office with no issues.

Any other thoughts with that new info? @seabass ?

​

 

[AndrewL733]

Hello @sojomy . Unfortunately I can no longer test out your possible "solution" although it sounds very promising. After nearly 2 months of making no progress with high-level ASUS technical support, I decided to return the router. I'm still just using my old RT-AC68U on one end and my less-old RT-AC86U on the other end. In the meantime, I decided to purchase 2 inexpensive Beelink EQ12 N100 Mini Desktop Computers (they're about $170 each on Amazon US) and I'm now setting them up to be my Site-to-Site VPN Gateways instead of using the ASUS Routers. There are many advantages to this approach:

• For VPN processing, they are much more powerful than an ASUS router. They also consume very little power -- about 5 watts.
• I can run an up-to-date Linux distribution on them (in my case, Ubuntu Server 22.04 LTS for X86_64)
• I can run any Linux-compatible VPN software. I am trying out Zerotier (free, seems to work great, very easy to configure) and it looks like I have a way to make a Layer2 ethernet bridge with Wireguard and something called GRETAP. At least Zerotier outperforms OpenVPN so far. And I have high hopes for Wireguard + GRETAP.

So, my plan in the future is to buy the best AX router I can find -- at the best price -- and not worry about having to run the VPN on the router.