OpenVPN Client: mishandled route directives

[eibgrad]

ASUS RT-AC68U - 386.14

I'm trying to help the OP w/ the following problem.

VPN Director Question

In the VPN director page, is it possible to add a rule to direct access to a domain to WAN? The reason why I am attempting that is to allow Voice over Wifi (VoWifi) to work together with VPN. I have discovered that in order for VoWifi to work on a phone which is directed through VPN, I need...

www.snbforums.com

One approach I was considering is to add route directives to the custom config field of the OpenVPN client to route the following domains through the WAN rather than the VPN.

route epdg.epc.mnc001.mcc505.pub.3gppnetwork.org 255.255.255.255 net_gateway
route epdg.epc.mnc071.mcc505.pub.3gppnetwork.org 255.255.255.255 net_gateway

I wasn't sure if it would work because I can recall from prior discussions regarding the behavior of the VPN Director, that sometimes the router chooses to strip out routes, be they route directives like the above, or even static routes defined in the main routing table.

To my surprise, I discovered that static routes *do* get copied into the OpenVPN client routing table (e.g., ovpnc1). Not sure if and when this changed, or whether it was always this way and I'm just having a false memory.

Anyway, I approached this use of route directives w/ the same skepticism, and sure enough, this time the routes did NOT show up in the OpenVPN client's routing table. Initially I chalked it up to the router stripping these routes out intentionally. But then I checked the syslog and got the following.

Aug  2 16:01:05 ovpn-client1[2760]: ovpn-up 1 client tun11 1500 0 10.80.0.110 10.80.0.109 init
Aug  2 16:01:06 openvpn-routing: Add pushed route: /usr/sbin/ip route add 149.135.226.11/255.255.255.255 via 192.168.63.1 dev tun11   table ovpnc1
Aug  2 16:01:06 openvpn-routing: Add pushed route: /usr/sbin/ip route add 149.135.233.3/255.255.255.255 via 192.168.63.1 dev tun11   table ovpnc1
Aug  2 16:01:06 openvpn-routing: Add pushed route: /usr/sbin/ip route add 149.135.224.26/255.255.255.255 via 192.168.63.1 dev tun11   table ovpnc1
Aug  2 16:01:06 openvpn-routing: Add pushed route: /usr/sbin/ip route add 149.135.232.3/255.255.255.255 via 192.168.63.1 dev tun11   table ovpnc1
Aug  2 16:01:06 openvpn-routing: Add pushed route: /usr/sbin/ip route add 101.168.246.65/255.255.255.255 via 192.168.63.1 dev tun11   table ovpnc1
Aug  2 16:01:06 openvpn-routing: Add pushed route: /usr/sbin/ip route add 101.168.246.193/255.255.255.255 via 192.168.63.1 dev tun11   table ovpnc1
Aug  2 16:01:06 openvpn-routing: Add pushed route: /usr/sbin/ip route add 149.135.224.24/255.255.255.255 via 192.168.63.1 dev tun11   table ovpnc1
Aug  2 16:01:06 openvpn-routing: Add pushed route: /usr/sbin/ip route add 149.135.226.9/255.255.255.255 via 192.168.63.1 dev tun11   table ovpnc1
Aug  2 16:01:06 openvpn-routing: Add pushed route: /usr/sbin/ip route add 10.80.0.1/255.255.255.255 via 10.80.0.109 dev tun11   table ovpnc1
Aug  2 16:01:06 openvpn-routing: Setting client 1 routing table's default route through the tunnel

It appears the router attempted to add the routes, but used the wrong network interface! tun11 is the tunnel, while my intention w/ the net_gateway option was the WAN (192.168.63.1), which is actually eth0. No wonder it didn't work.

Thinking maybe this was intentional, I changed from using the VPN Director to Yes (all). At least now the router shouldn't interfere w/ my choice to route whatever I want, wherever I want. NOPE. Same result. Even using No didn't work!

Because the OpenVPN client config file is using the route-exec directive w/ all "Redirect internet gateway through tunnel" options, it's clear the route directives are *always* being managed by the ovpn-* processes under the /tmp/etc/openvpn/client1 directory. I'd be interested in what these are doing, and whether it's intentional or just a bug, but they're binary, NOT text (scripts).

As I said, I could imagine such manipulation being intentional for the VPN Director, but NOT Yes (all), and certainly NOT for No. It just appears to be a bug. And it might just be an oversight in believing that route directives would/could only be routed over the VPN, when that isn't always the case. You can use either vpn_gateway (the default) or net_gateway to route either through the VPN or WAN, respectively.

I suspect few ppl have noticed this issue since the VPN Director is, by far, the most common method for managing routing between the VPN and WAN. But as illustrated in the link above, there are situations where the use of route directives comes in handy, esp. when you need/want to use domain names rather than explicit public IPs (a limitation of the VPN Director).

 

[RMerlin]

It's a limitation in OpenVPN itself. The routes are passed as environment variables, and libovpn adds these routes one by one based on what is stored in these variables. There is no variable specifying the interface according to the man page:

route_{parm}_{n}
A set of variables which define each route to be added, and are set prior to --up script execution.

parm will be one of network, netmask", gateway, or metric.

n is the OpenVPN route number, starting from 1.

If the network or gateway are resolvable DNS names, their IP address translations will be recorded rather than their names as denoted on the command line or configuration file.

I am already processing all of these four variables:

                i = 0;
                while (1) {
                        i++;
                        sprintf(buffer, "route_network_%d", i);
                        network_env = getenv(buffer);
                        sprintf(buffer, "route_netmask_%d", i);
                        netmask_env = getenv(buffer);
                        sprintf(buffer, "route_gateway_%d", i);
                        gateway_env = getenv(buffer);
                        sprintf(buffer, "route_metric_%d", i);
                        metric_env = getenv(buffer);

                        if (!network_env || !netmask_env || !gateway_env)
                                break;

                        if ( (inet_pton(AF_INET, network_env, &network) == 1)
                            && (inet_pton(AF_INET, netmask_env, &netmask) == 1)) {

                                snprintf(buffer, sizeof (buffer),"/usr/sbin/ip route add %s/%s via %s dev %s %s %s table ovpnc%d",
                                        network_env, netmask_env, gateway_env, dev_env, (metric_env ? "metric" : ""), (metric_env ? metric_env : ""), unit);
                                if (verb >= 3)
                                        logmessage("openvpn-routing","Add pushed route: %s", buffer);
                                system(buffer);
                        }
                }

 

[eibgrad]

Where is dev_env coming from? According to the docs, the network interface is not provided, which means the routing system will resolve it (presumably correctly). And that's the problem child here. dev_env is apparently always being set to tun1x, but I don't where that is being done. It's NOT part of that code. Seems to me if the route function doesn't provide the network interface, then you should just not provide it in the route command and let the routing system resolve it.

Now that I think about it, is this perhaps a change you made wrt that other problem we had some time ago where the user was connected to multiple, concurrent OpenVPN clients and it happened they where using the *same* IP network on their respective tunnels, and I suggested we add tunnel's network interface to disambiguate them?

VPN Director not applying rules to right VPN profile

Hi, I'm running into a routing issue when configuring multiple VPN profiles using VPN Director on my LINKSYS (base model: RT-AC68U) firmware version 386.3_2. I have 2 VPN profiles with the following external addresses: OVPN1) Connected (Local: 10.16.0.43 - Public: 185.240.246.99) OVPN3)...

www.snbforums.com

I could be offbase here, but it is curious it involves a similar type of issue (i.e., the network interface specification when creating routes). Just wondering if maybe that has had an influence here.

 

[RMerlin]

Where is dev_env coming from?

That would be the tunnel interface:

                // Apply pushed routes
                dev_env = _safe_getenv("dev");

Now that I think about it, is this perhaps a change you made wrt that other problem we had some time ago where the user was connected to multiple, concurrent OpenVPN clients and it happened they where using the *same* IP network on their respective tunnels, and I suggested we add tunnel's network interface to disambiguate them?

According to git blame, that line was already there when the whole VPN routing engine was implemented with the following commit:

commit 5e111e9fdf89c5fd67d47d85eef85856c9ee5c19
Author: Eric Sauvageau <merlin@asuswrt-merlin.net>
Date:   Fri Jun 4 11:34:37 2021 -0400

    libovpn: rewrote OpenVPN client routing implementation
  
      - Move vpnrouting.sh functionality into libovpn handlers
      - Now all routes are manually created by the handlers instead
        of automatically by the openvpn process
      - Improve behaviour of the option to redirect Internet traffic
        through a tunnel - now it will actually control the default
        gateway for a tunnel rather than let the openvpn server
        determine it.
      - Make killswitch option work in non-policy mode
      - Killswitch will now be cleared when a VPN client is
        manually stopped (by removing all rules pointed at
        that instance's routing table)
      - Remove strict policy routing mode.  Previously configured
        clients will switch to regular policy routing at boot time.

Personally, I don't like the idea of leaving the interface empty, for instance because of the scenario you mention where someone might have similar routes in different tunnels.

I had never seen someone use a route that specified an interface before (didn't even know it was possible since the OpenVPN man does not even mention it in the list of available environment variables). So I think for the very rare case where someone might want to use that, they would be better handling it through VPN Director, or through a custom script. These would be very unusual setup scenarios.

A more appropriate fix would be for OpenVPN itself to provide that interface in an environment variable like it does for all other parameters of the route config setting, so I could rely on it.

 

[eibgrad]

Under normal circumstances, there isn't a need for including the network interface w/ the route command *assuming* you haven't created any routing ambiguities by having the same IP network available from more than one network interface. If that happens, it's typically considered a configuration error and needs to be corrected by the user, something you and I agreed upon even back then. And obviously so does OpenVPN given the lack of it in their API.

What we have w/ the VPN Director is something OpenVPN is not anticipating, namely, a routing ambiguity that *can* be corrected by the firmware due to how policy based routing has been implemented, specifically because each is associated w/ a unique routing table (ovpnc1, ovpnc2, etc.). Many other firmwares don't even allow multiple, concurrent OpenVPN clients. And if they do, will do NOTHING to resolve any such ambiguities. It's what makes Merlin firmware unique.

Problem is, we are (apparently) resolving it unconditionally, as if the *only* routes possible are those intended for the table's default gateway (vpn_gateway), when in fact it might be intended for the ISP's default gateway (net_gateway). It's even possible those routes could be *pushed* by the OpenVPN server! So we can't just tell users to NOT specify such routes when using the VPN Director.

As it stands now, we resolve ambiguity in the case of a specific user misconfiguration w/ policy based routing, but at the expense of properly handling route directives intended for the ISP's default gateway.

If it has to be this way, I'd prefer we NOT add the network interface AT ALL (which is considered normal and acceptable) and force the user to correct the misconfiguration. At least the route directives intended for the net_gateway network interface will work correctly. And we can legitimately argue to the user they are responsible for the misconfiguration and must be the one to correct it.

BTW, as I alluded to in the other thread, I had assumed that any route directives specified in custom config would be IGNORED given the presence of the VPN Director. I seem to recall that being part of the discussion back in those days. It only became a recent curiosity to me when I saw in the syslog that these route directives were in fact being processed, but incorrectly in the case of net_gateway.

I'd almost be happier if we actually *did* ignore them, and indicate as much in the syslog. At least we could then make the argument that once you're relying on the VPN Director, such directives are superfluous (except in the case of push'd route directives). Of course, that comes at the expense of domain name resolution, a known shortcoming of the VPN Director. Not ideal, but it's at least consistent and well-defined in terms of its behavior.

My preference, however, would be to include the network interface, but not do so *blindy* and just assume it should be the vpn_gateway. We need to notice whether the OpenVPN API function has been passed the net_gateway and in that instance NOT include the network interface, and instead let the routing system resolve it. At that point, it seems to me everything now works.

Would I prefer the OpenVPN API provided the network interface? Of course, but as I said in the beginning, it's not normally necessary. It's just a quirk of the firmware's policy based routing that specifying the network interface becomes handy in letting it resolve routing ambiguities that would otherwise be the sole responsibility of the user. But if that means breaking the route directives in the process, I'd rather we didn't. Instead, we just tell the user "hey, you created the problem, you fix it".

 

[RMerlin]

To be honest, I've grown tired of trying to acomodate every unusual edge case users are trying to create by running multiple tunnels, with multiple concurrent needs and rules. It's like trying to plug a leak, to find two more leaks springing elsewhere. I simply no longer have the will to keep trying to handle every new "creative" ways people come up with in their network setups - too much efforts for a very small handful of users. As I say, every time I try to address some weird case, I end up getting some coming up saying "Yes, but I also want it to..." So for now, my answer to these is: "Simplify your setup, configure things manually with custom scripts, or build a Linux server where you have full manual control over everything".

If someone else can provide a patch that clearly improve things without creating new issues, I'm willing to consider it. But at this time, my time and energies are better spent elsewhere, sorry.

 

[RMerlin]

Where did you see that you can specify a network interface with the route parameter BTW? The OpenVPN 2.6 man page does not lists this as a valid parameter. It only mentions the four parameters currently offered as env variables:

--route args   
Add route to routing table after connection is established. Multiple routes can be specified. Routes will be automatically torn down in reverse order prior to TUN/TAP device close.

Valid syntaxes:

route network/IP
route network/IP netmask
route network/IP netmask gateway
route network/IP netmask gateway metric
This option is intended as a convenience proxy for the route(8) shell command, while at the same time providing portable semantics across OpenVPN's platform space.

netmask
defaults to 255.255.255.255 when not given
gateway
default taken from --route-gateway or the second parameter to --ifconfig when --dev tun is specified.
metric
default taken from --route-metric if set, otherwise 0.
The default can be specified by leaving an option blank or setting it to default.

The network and gateway parameters can also be specified as a DNS or /etc/hosts file resolvable name, or as one of three special keywords:

vpn_gateway
The remote VPN endpoint address (derived either from --route-gateway or the second parameter to --ifconfig when --dev tun is specified).
net_gateway
The pre-existing IP default gateway, read from the routing table (not supported on all OSes).
remote_host
The --remote address if OpenVPN is being run in client mode, and is undefined in server mode.

A server-pushed route will never specify an interface. Firstly because it has no way of knowing what interfaces exist on the client in the first place.

 

[eibgrad]

It wasn't my intention to suggest or even imply that the user (or YOU on behalf of the user) was expected to provide the network interface (if I did, then I misspoke). As I said, OpenVPN doesn't even require it whether via the route directives or the API for the reasons you suggest; it can't be known with any degree of reliability, be it the client or server. That should normally be resolved by the routing system itself. We were just trying to accommodate the user in that one particular instance because it seemed simple enough given how the VPN Director is implemented. I just didn't think we would apply that logic so broadly as to break the route directives in the process.

As far as growing tired of the changes, I completely understand. In fact, I personally believe you've been far too accommodating! There's NO WAY I would have changed the kill switch behavior as you did, let alone do it TWICE! Or the recent revelation about the kill switch behavior in the face of multiple, concurrent OpenVPN clients. My primary concern with any changes is breaking things, esp. anything related to normal OpenVPN behavior. As I said, had I known we would break the route directives, I would have just told the user "too bad, fix your configuration". I'm only willing to accommodate changes provided I have reasonable assurance we don't break things. And given the complexity of OpenVPN, it's not always obvious when that might happen.

Anyway, as with all my comments, I seek not just to affect change, but to clarify the situation and provide information for posterity's sake, in case it becomes an issue in the future for other users. I can at least point to this discussion and make clear why things work (or don't work) as they do, and leave it to the user to act accordingly, be that living with it, or considering the alternatives.

Case and point; that script provided by @Ranger802004 to deal w/ the lack of name resolution by the VPN Director. Notice I did NOT suggest or even hint YOU should now support domain names w/ the VPN Director. I would be opposed for the very reasons you stated. But at least if the route directives worked as expected, @js28194 would not be in the position of needing yet another script to fix his problems and inadvertently creating routing conflict(s) in the ip rule database.

Heck, how many times have I complained about all the DNS configuration options in the router. And now we have the topper; the DNS Director. And all this stems from something that at its core is nothing but a lookup service between a name and an ip address! It's just out of control imo. As soon as I see a thread that suggests a DNS leak, I turn tail in the other direction! We are often too accommodating for our own good.