vpn problems in asuswrt merlin 3004.388.8

[moreamazingnick]

Hi,

I have the following setup up and running and it is working like a charm
OpnSense <-OpenVPN Site2Site-> RT-AX58U 3004.388.7 (asuswrt merlin)
OpnSense <-OpenVPN Roadwarrior-> (iPhone/Mac)

I can connect from the OpenVPN Roadwarrior to like everything.

Today I tried to replicate that for a friend and it didn't work.
OpnSense <-OpenVPN Site2Site-> RT-AX58U 3004.388.8 (asuswrt merlin)
OpnSense <-OpenVPN Roadwarrior-> (iPhone/Mac)

* The Site2Site tunnel is ok
* I can access the OpnSense and everything behind it.
* I can't access the the RT-AX58U via vpn over the two tunnels

I went back home and realised the difference that I was using RT-AX58U 3004.388.7.
So I upgraded my router to RT-AX58U 3004.388.8 and it the problem was the same.
Traffic is rejected by the RT-AX58U. It is the same VPN config.

To get it working again I downgraded my setup to RT-AX58U 3004.388.7 and everything was fine.

I hope that someone can replicated that, otherwise im stuck with 3004.388.7.

Thanks for your help and

Best Regards Nicolas

 

[bennor]

May or may not be relevant to your setup. But note (section in bold in quote below) what the 3004.388.8 change log indicates for VPN.

3004.388.8 (21-July-2024)
- NOTE: RT-AX56U is exceptionally included in this release.
- NEW: Rewrote VPN killswitch implementation. The new method
uses an always present routing rule to prohibit access to
the main routing table, so it will be active even if the
user manually stops a client. Removing the prohibit rule
requires disabling the killswitch on the webui.
The rules are also created before WAN goes up, to reduce
the risks of leaks between WAN going up and VPN connecting.

*** Make sure to double check that you don't have any
unwanted killswitch enabled if you have connectivity issues
following the upgrade to this firmware

 

[eibgrad]

Several other users in the following thread are reporting similar problems.

Release - Asuswrt-Merlin 3004.388.8_2 is now available

Asuswrt-Merlin 3004.388.8 is now available for supported Wifi 6 models. This release implements a new VPN killswitch method, and fixes a number of issues. July 31st: 3004.388.8_2 is now available. Changes since 3004.388.8: 3004.388.8_2 (31-July-2024) - UPDATED: OpenVPN to 2.6.12. -...

www.snbforums.com

Which side is the OpenVPN client vs. OpenVPN server? It's not clear from your description.

 

[eibgrad]

May or may not be relevant to your setup. But note (section in bold in quote below) what the 3004.388.8 change log indicates for VPN.

Based on what little I've been able to glean so far from those reporting similar problems in the release thread, it *appears* the problem is coming from the ASUS when it's the OpenVPN server side. And if that's the case, it would also appear the VPN kill switch is NOT relevant. But I have to admit, we're still at the stage where NO ONE has been able/willing to provide us w/ detailed dumps of their configuration(s), including ifconfig, ip route, iptables, etc. And until we do, it's going to prove VERY difficult to find the culprit.

Clearly something is wrong given the number of similar reportings.

 

[moreamazingnick]

May or may not be relevant to your setup. But note (section in bold in quote below) what the 3004.388.8 change log indicates for VPN.

Thanks for your reply,

I checked that, and i think killswitch is only enabled for redirect gateway, which is not enabled for this config.
The clients from the the Asus Router can get to the firewall (opnsense) and beyond but everything attached the firewall can not reach out to the router.

 

[moreamazingnick]

Several other users in the following thread are reporting similar problems.

Release - Asuswrt-Merlin 3004.388.8_2 is now available

Asuswrt-Merlin 3004.388.8 is now available for supported Wifi 6 models. This release implements a new VPN killswitch method, and fixes a number of issues. July 31st: 3004.388.8_2 is now available. Changes since 3004.388.8: 3004.388.8_2 (31-July-2024) - UPDATED: OpenVPN to 2.6.12. -...

www.snbforums.com

Which side is the OpenVPN client vs. OpenVPN server? It's not clear from your description.

OpnSense provides both servers (site2site and roadwarrior)
RT-AX58U is the Site2Site Client

 

[Analog-1]

Based on what little I've been able to glean so far from those reporting similar problems in the release thread, it *appears* the problem is coming from the ASUS

This Merlin release contains the same GPL from all the way back to 388.6.

 

[eibgrad]

Then let's dump the configuration on the ASUS OpenVPN client. Ideally, both when it works and when it doesn't, then compare them. Or at least when it's NOT working.

ifconfig
ip route
ip route show table ovpnc1
brctl show
ip rule
cat /tmp/etc/openvpn/client1/config.ovpn
cat /jffs/openvpn/vpndirector_rulelist
iptables -vnL
iptables -t nat -vnL
iptables -t mangle -vnL

Since I'm not sure which OpenVPN client you're using, I've assumed #1. If not, change the references as necessary. It's ok to mask your public IP, but just do so in an obvious and consistent fashion.

 

[moreamazingnick]

here are 5 commands executed on the working vpn using 3004.388.7

 

[moreamazingnick]

and the 5..

 

[moreamazingnick]

and now the broken setup / 5 commands

 

[moreamazingnick]

and the last 5 of a broken setup

 

[eibgrad]

Hi,

I have the following setup up and running and it is working like a charm
OpnSense <-OpenVPN Site2Site-> RT-AX58U 3004.388.7 (asuswrt merlin)
OpnSense <-OpenVPN Roadwarrior-> (iPhone/Mac)

I can connect from the OpenVPN Roadwarrior to like everything.

Today I tried to replicate that for a friend and it didn't work.
OpnSense <-OpenVPN Site2Site-> RT-AX58U 3004.388.8 (asuswrt merlin)
OpnSense <-OpenVPN Roadwarrior-> (iPhone/Mac)

* The Site2Site tunnel is ok
* I can access the OpnSense and everything behind it.
* I can't access the the RT-AX58U via vpn over the two tunnels

Ok, I want to get very precise here to make absolutely sure I understand the issue.

When you say Site2Site is OK, what exactly does that mean? Are you saying that IP network(s) behind the OpnSense router are accessible from the IP network(s) behind the RT-AX58U, and vice versa?

Because your reference to "the two tunnels" initially had me puzzled. But I now think what you mean is that the problem is limited to the iPhoneMac, when you are *chaining* OpenVPN tunnels.

Well here's what I found. In the configuration that works, the ip rules database is as follows:

0:    from all lookup local
10001:    from all lookup ovpnc1
32766:    from all lookup main
32767:    from all lookup default

BUT, when it's NOT working, that same ip rules database is as follows:

0:    from all lookup local
10001:    from all iif br0 lookup ovpnc1
32766:    from all lookup main
32767:    from all lookup default

Notice the difference. In the latter case, the rule is *limited* to those source IPs coming from the LAN (br0). In the former case, it's *anything*, be it the LAN, WAN, VPN, whatever. And in the case of the iPhone/Mac, they are NOT coming from the LAN, but the OpenVPN server's tunnel (e.g., tun1)! So they will never be routed over to the ASUS router.

I have no idea why this was changed, but it seems to me this is the issue. You could try replacing that rule w/ the prior form of the rule to see if it now works.

ip rule del iif br0 lookup ovpnc1 prior 10001
ip rule add lookup ovpnc1 prior 10001

That's NOT a long-term solution. I just want to verify that indeed it is the problem.

 

[RMerlin]

I don't remember the reason because that code was developed months ago. But I recall that there was an issue related to specifying the interface, either related to the killswitch or to Guest Network Pro. I can't remember what it was, but that was the reason why I had to change it.

The closest I can find in my git commits relative to this is this commit:

commit c16e25979a085d71627baeb1971096fc75a4733d
Author: Eric Sauvageau <rmerl@lostrealm.ca>
Date:   Wed May 29 17:55:00 2024 -0400

    libovpn: revert copying main table to vpn table, and have RGW_ALL rule include all interfaces
    
    This allows LAN routes to still be reachable even if the killswitch
    is active.  This allows to retain LAN access, and also stun to retrieve
    its public IP address.
    
    The killswitch rules are still interface-specific, to ensure we don't affect
    other connections.
    
    Added a "noroutecopy" nvram as a mean to disable this copy behaviour, for
    troubleshooting purposes.

 

[eibgrad]

I don't remember the reason because that code was developed months ago. But I recall that there was an issue related to specifying the interface, either related to the killswitch or to Guest Network Pro. I can't remember what it was, but that was the reason why I had to change it.

The closest I can find in my git commits relative to this is this commit:

commit c16e25979a085d71627baeb1971096fc75a4733d
Author: Eric Sauvageau <rmerl@lostrealm.ca>
Date:   Wed May 29 17:55:00 2024 -0400

    libovpn: revert copying main table to vpn table, and have RGW_ALL rule include all interfaces
  
    This allows LAN routes to still be reachable even if the killswitch
    is active.  This allows to retain LAN access, and also stun to retrieve
    its public IP address.
  
    The killswitch rules are still interface-specific, to ensure we don't affect
    other connections.
  
    Added a "noroutecopy" nvram as a mean to disable this copy behaviour, for
    troubleshooting purposes.

Whatever the justification for the change, it seems to me it will prove problematic. I'm still NOT 100% convinced it explains the OP's problem, since as he describes it, the iPhone/Mac clients are (apparently) connecting to the OpenVPN server on the OpnSense machine, NOT the AX-58U. Then again, it's the only difference I see between the working and non-working configurations.

But imagine a situation where the OpenVPN server *is* established on the AX-58U rather than the OpnSense, and you wanted to allow OpenVPN clients of that server to access the IP networks available over the local OpenVPN client. That's NOT possible now. The OpenVPN client is forced into the main routing table, which does NOT contain the necessary routes.

Again, hard to tell at this point whether this is the explanation for the OP, or others in the release thread, or whether perhaps there's some other issue at play. A lot of the provided descriptions are quite muddled and imprecise. But even if it isn't, seems to me this is going to rear it's ugly head eventually. The only saving grace is it's relatively uncommon to be chaining tunnels in this fashion.

 

[eibgrad]

P.S. Thanks to the OP for providing these dumps. I know it's a pain. But it was like pulling teeth to get ppl in the release thread to do it. They seem to think we have magical powers to determine the underlying problems. Most of the time, these dumps tell the story, at least if you know what to look for.

 

[Purana]

I also have similar problems.

My setup is bi-directional LAN to LAN VPN via OpenVPN in TUN mode, without internet redirect.
Server side (Site 1) is a GT-AX6000 and client side (Site 2) is a RT-AX86u Pro.
After updating both routers to v388.8, the VPN connection became one-way,
client side can access server side normally, but server side failed to access client side.

Everything works fine after client side router rolling back to v388.7.

attached are my dump file, hope someone can find out the root cause

Server side v388.7 dump files, both sites can access each other:

 

[Purana]

I also have similar problems.

My setup is bi-directional LAN to LAN VPN via OpenVPN in TUN mode, without internet redirect.
Server side (Site 1) is a GT-AX6000 and client side (Site 2) is a RT-AX86u Pro.
After updating both routers to v388.8, the VPN connection became one-way,
client side can access server side normally, but server side failed to access client side.

Everything works fine after client side router rolling back to v388.7.

attached are my dump file, hope someone can find out the root cause

Server side v388.7 dump files, both sites can access each other:

Server side v388.7 dump files, both sites can access each other:

 

[Purana]

Server side v388.7 dump files, both sites can access each other:

Client side v388.7 dump files, both sites can access each other:

 

[Purana]

Client side v388.7 dump files, both sites can access each other:

Client side v388.7 dump files, both sites can access each other: