VPN routing goes wrong

[MvW]

First of all, I'm aware of YazFi and X3MRouting, but I wanted to set it up myself first, before I have to dig into those scripts. I've done PBR in the past, but the memory has faded so it feels like I'm reinventing the wheel once more.

I have set up two ProtonVPN clients. I have selected policy based routing strict and disabled accept DNS (as I want NextDNS CLI to handle all DNS requests). I've divided the clients over both VPN Clients, some individual, some by range using an IP calculator. How can I check in a terminal window that each client is using the assigned interface (eth0, tun11 or tun12). I've no knowledge of IPtables, so if someone could provide an example with a tiny bit of explanation, I've got something new to study on.

Oh, one more question: I can reach all the clients from my LAN. Are the clients which are (should be) directed through tun11 isolated from the clients (mostly IoT-devices) which are in the PBR-table for VPN Client 2?

You're help is greatly appreciated.

Best regards,
Marco

 

[MvW]

Okay, something's wrong. I ran a diagnostics test and my laptop which should be routed through a VPN-tunnel within my own country (The Netherlands) but is being routed through a second tunnel to another country Belgium. So I need to setup things different. I have currently two VPN Clients setup on my router, but traffic isn't routed correctly. Can someone check my config, as I'm an absolute noob when it comes to VPN. Is there a misconfiguration or where do I start to correct this?

I'm looking for an easy (let say not too complicated) solution. Can @Jack Yaz s' YazFi help me organize this better or should I be looking at @Xentrk s' x3mRouting? I'm not interested in streaming TV from other countries, I just want private traffic run through a VPN within my own country (for speed), IoT separated from my LAN and re-routed to another country (but still available from within my LAN) and my kids game console and his game PC kept outside the VPN tunnels (als for speed).

Here are screenshots of both client configs. DNS is disabled on purpose, NextDNS CLI is handling these (and doing a better job then expected, as it actually uses it's low latency servers in Brussels for the VPN client connected to Belgium and Amsterdam for the traffic outside and through the tunnel within the Netherlands, to my surprise).

Client Config 1 (NL)

Client Config 2 (BE)

If needed I can provide more info, just let me know what you need.

Your help is much appreciated.

Best regards,
Marco

Edit: is it just my eyes or do these screenshots become absolutely unreadable after uploading? Seriously wondering why I've been blurring parts before upload, LOL.

 

[Xentrk]

If you have more than one client running, add an entry to the router in OpenVPN client 1 to egress via the WAN e.g. 192.168.1.1. I've experienced similar routing experienced without the entry. Adding it got things working as expected. Also, make sure the OpenVPN clients are not using the same port number.

 

[MvW]

Thanks for your reply. I'll give that a try. Both clients are different ports btw. What does egress (and ingress) mean btw? It is outgoing and incoming traffic through specified destinations?

 

[MvW]

I just checked, but my router is already there (Client 1) with destination WAN:

 

[Jack Yaz]

Depending on the provider, you're probably being given a client ip in the same subnet for both clients, which causes problems in the routing table. Try using 1 udp and 1 tcp. On nordvpn this gives addresses in separate ranges

 

[MvW]

Depending on the provider, you're probably being given a client ip in the same subnet for both clients, which causes problems in the routing table. Try using 1 udp and 1 tcp. On nordvpn this gives addresses in separate ranges

That's not the cause either, both IP's are assigned are in completely different ranges. I have a feeling that because on one of the clients 'the remainder' (192.x.x.0/24) of the local IP's is redirected, this is rule is dominant. But this line is at the end of the PBR-table of client 2. It seems like the only way to avoid this (if I want to stick to this way) is to specify each client separately.

If I should switch to YazFi, would this be simpler to configure? I've looked through the instructions x3mrouting (no offense, @Xentrk), but that's way beyond my level of comprehension.

 

[alecmascot]

I found this write-up worked for me: https://x3mtek.com/policy-rule-routing-on-asuswrt-merlin-firmware/
Unfortunately I cannot read your images.
I have Client 1 to NL and Client 2 to GB
I run Client 1 as "Policy Rules strict" and Client 2 as "Policy Rules"
Client 1 has the high priority routings and Client 2 catches everything else.

 

[eibgrad]

If you have more than one client running, add an entry to the router in OpenVPN client 1 to egress via the WAN e.g. 192.168.1.1. I've experienced similar routing experienced without the entry. Adding it got things working as expected. Also, make sure the OpenVPN clients are not using the same port number.

I just checked, but my router is already there (Client 1) with destination WAN:

View attachment 32856

Except in one rare situation, this rule is pointless.

Whenever PBR (policy based routing) is enabled, it has the side-effect of taking the router itself OFF the VPN (it's what sometimes causes DNS leaks!). And even if the router is *implicitly* routed over the VPN due to some other rule (e.g., "192.168.1.0/24 VPN"), it's still pointless. All the router's internet-bound packets are going to use the WAN ip as their source, NOT the LAN ip (192.168.1.1)!

It's NOT that it's doing any harm, but if you believe this rule is somehow making a difference in whether something works, I fail to see how that's possible. NOT unless the service on the router is specifically bound *only* to the LAN (which is rare).

The only time I've seen this rule being useful is for remote access of the GUI over the WAN (and even then, you have to question whether this is safe). Because the WAN's public IP is DNAT'd to the LAN ip of the router for these purposes (effectively binding it to the LAN), if you have a rule like "192.168.1.0/24 VPN'' which binds the router's LAN ip to the VPN, if it wasn't for the "192.168.1.1. WAN" rule, the replies would NOW be routed over the VPN and NOT back over the WAN, making the GUI inaccessible. But that's a rare situation, and NOT typically the reason users add this rule. More commonly, they are under the misconception that this will bind all the router's actions to specified network interface (WAN or VPN).

In most cases, I wouldn't even mention it given it does no harm. But when I hear claims this rule *fixed* something (other than this issue w/ remote access to the GUI), I have to wonder if something else is amiss.

 

[MvW]

I found this write-up worked for me: https://x3mtek.com/policy-rule-routing-on-asuswrt-merlin-firmware/
Unfortunately I cannot read your images.
I have Client 1 to NL and Client 2 to GB
I run Client 1 as "Policy Rules strict" and Client 2 as "Policy Rules"
Client 1 has the high priority routings and Client 2 catches everything else.

I cannot even read my own screenshots, so for those interested, here they are on imgur, in full resolution. (Click to view in original resolution).

Client 1:

https://imgur.com/a/9IiUqf8

Client 2:

https://imgur.com/a/QUxjnxh

x3mrouting looks to complex, beyond my comprehension, my head's a mess, I don't know where to start and due to home schooling for my kiddo I don't have to luxury to take days to transition. Don't get me wrong, but I really need someones help with matters like these.

@eibgrad I was actually hoping your eye would fall on this thread. Can you or anyone else please tell me if what I would like is actually possible and if so, how to achieve it. I've seen you explaining on many threads what won't work, but would really appreciate if you could help me to make things actually work (if possible as outlined in post #2) .

Your help is really much appreciated.

Best regards,
Marco

 

[eibgrad]

How can I check in a terminal window that each client is using the assigned interface (eth0, tun11 or tun12). I've no knowledge of IPtables, so if someone could provide an example with a tiny bit of explanation, I've got something new to study on.

Dump connection tracking. It's the only means I've found to be 100% positive.

cat /proc/net/nf_conntrack

Or do so continuously...

watch -tn10 'cat /proc/net/nf_conntrack'

What matters is the dst field on the second pair of src/dst fields. That will contain the IP address to which the router is expecting a reply, which you can then map back to its specific network interface using ifconfig.

Oh, one more question: I can reach all the clients from my LAN. Are the clients which are (should be) directed through tun11 isolated from the clients (mostly IoT-devices) which are in the PBR-table for VPN Client 2?

Regardless how individual devices on the private network are bound to the various OpenVPN clients, all remain accessible among each other since they are all *local* to each other. All that differs is how they are routed wrt the internet.

 

[eibgrad]

Admittedly, saying what won't work is easier than what will work since the latter is often harder to define, or at least NOT always made clear to others. Users often don't tell you all the *finer* requirements left unmentioned!

Anyway, here's what I can say in terms of what will work, in general.

Whenever you use multiple, concurrent OpenVPN clients, you *must* use PBR (policy based routing) since this is the only *sane* way to control what devices use which VPN (if any). Without that as a starting assumption, it becomes nearly impossible to be assured things will work as expected.

Also, you have to be careful in using the strict option w/ PBR. Frankly, I don't think that additional feature was necessary. It seems like overkill. But that's another story for another day. But given its behavior (which is to strip out any routes over the WAN or other VPNs), that can lead to unexpected results (at least if you don't know what the strict option does). For example, it's often easier and more straightforward to use static routes (in the form of route directives in custom config) to control where *destination* IPs are directed rather than PBR. But strict mode will strip them out, forcing you to use PBR for destination IPs as well. When all else fails, I would remove the strict option and retest. It's just one less complication.

Then you get into the whole issue of DNS (another can of worms). Since PBR takes the router itself off the VPN, the potential exists for DNS leaks (at least using traditional port 53, udp/tcp DNS). But if you're using NextDNS, it doesn't really matter since those DNS transactions are encrypted, regardless if accessed over the WAN or VPN. But then some ppl don't like the idea of their ISP even *knowing* they're using NextDNS, so they insist it be routed over the VPN anyway!

All that said, if all you're doing is multiple, concurrent OpenVPN clients, none of their respective IP tunnel networks conflict, you're using PBR based on source IP only, have NOT created any rules that are ambiguous as to which client uses which VPN, and using NextDNS (so the issue of where DNS is accessed is moot), I don't see where this should cause any problems.

Of course, the trick is being satisfied it's working as you expect, and why I recommend dumping connection tracking. So far you've provided no proof things are NOT working as expected. You just seem to be concerned they may NOT, mostly out of a lack of knowledge about how to make that determination.

 

[MvW]

Thanks for your explanation, @eibgrad, I really appreciate it. I don't know how to provide proof without disclosing sensitive info. What I made me discover something went wrong, is that I had to run the diagnostic tool provided by NextDNS. The device specified at the bottom of client 1, should have an endpoint (?) in Amsterdam. Running diagnostics tool I found an endpoint in Brussels. When comparing the assigned external IP-adresses, which are both in completely different ranges I found out that the laptop specified in the PBR table for Client 1 (Amsterdam) ends up in Brussels. So, where I thought I (or NextDNS) was having DNS issues, because I noticed Brussels was preferred over Amsterdam, the NextDNS CLI Client was acting exactly like it should, contacting the nearest DNS server. But that raised the question how a device specified under Client 1 ends up in at the endpoint (I'm not sure if that's correct terminology, but I mean 'the other side of the tunnel) of Client 2. The closest thing I can provide as 'proof' is that if you're willing to believe that Client 1 goes to Amsterdam, that whatsmyip.org shouldn't show this:

The other clients which are correctly directed to Amsterdam, actually do show Amsterdam and a whole different IP.

It's getting late here. I'm gonna read you previous post tomorrow more thoroughly and see if I can figure this out. I'll be back. (I can't seem to write that without hearing Arnold.)

Best regards and thanks so far,
Marco

 

[eibgrad]

Let's make sure we're both on the same page here as to how this should be working. Because I'm getting the impression you're expecting the NextDNS log to report the public IP bound to the local LAN client (WAN or VPN), which is NOT correct.

When you configure NextDNS w/ the router, that reconfigures DNSMasq to use NextDNS as a proxy. Which network interface the NextDNS proxy will use depends on how the router is configured wrt its OpenVPN clients (if any), which *should* be the WAN since all of them are presumably using PBR. And as such, the NextDNS log will report the WAN ip for any local LAN clients, irrespective of how that local LAN client itself is bound to the internet, be it the WAN or one of several OpenVPN clients.

IOW, NextDNS is NOT reporting the public IP bound to the local LAN client, but rather the public IP of the device hosting the NextDNS proxy.

Or else I'm misunderstanding your question/issue.

 

[MvW]

Let's make sure we're both on the same page here as to how this should be working. Because I'm getting the impression you're expecting the NextDNS log to report the public IP bound to the local LAN client (WAN or VPN), which is NOT correct.

Hate to disagree with you, but that's actually exactly what I'm seeing. In the online logfiles NextDNS is providing I can see which external IP is bound to it, even though it's behind my router on the other end of the tunnel. It shows my WAN IP for many devices (which I will investigate later to day),

https://imgur.com/a/eFj0Mdw

but my laptop, which is in the PBR table for Client 1 but connects through the VPN of Client 2, does actually show the external IP assigned to tunnel 2:

https://imgur.com/a/92Eg151

Both devices are connected to my router, one is routed properly the other isn't. Both are using two different NextDNS DNS-servers.

The only thing I noticed is that for devices connecting through WAN the local IP is in the pop-over in their logbook, for devices routed through (one of the tunnels) it's not. Maybe that's expected behaviour, I don't know. I don't have enough knowledge of VPNs.

 

[eibgrad]

Both are using two different NextDNS DNS-servers.

I don't understand what you mean by that statement.

AFAIK, when you install NextDNS, it creates a proxy that gets referenced by DNSMasq.

admin@merlin-lab1:/jffs# cat /tmp/etc/dnsmasq.conf | grep -E 'server|no-resolv'
no-resolv
no-resolv
server=127.0.0.1#5342

I can even see in the router's syslog the https connections established by NextDNS.

Apr  7 08:03:01 nextdns[31443]: Connected 188.172.251.1:443 (con=30ms tls=0ms, )
Apr  7 08:04:01 nextdns[31443]: Connected 137.220.51.178:443 (con=18ms tls=125ms, TLS13)

And how these are bound to the WAN of my router (which is an internal router, so it has a private IP of 192.168.63.102).

ipv4     2 tcp      6 8 TIME_WAIT src=192.168.63.102 dst=188.172.251.1 sport=39327 dport=443 src=188.172.251.1 dst=192.168.63.102 sport=443 dport=39327 [ASSURED]
ipv4     2 tcp      6 71 TIME_WAIT src=192.168.63.102 dst=137.220.51.178 sport=40486 dport=443 src=137.220.51.178 dst=192.168.63.102 sport=443 dport=40486 [ASSURED] mark=0 use=2

It's up to that proxy to determine which of those NextDNS servers (connections) are accessed in response to any DNS requests, irrespective of the specific local client, or how that client may be bound wrt the internet (WAN or VPN). And what NextDNS reports in its own log for the given local client is whatever response it gets from a public IP checker on the router.

If I run the following from the router so it continuously reports to me whatever it sees as the public IP ...

while :; do wget -qO - http://ipinfo.io/ip; echo; sleep 10; done

… it's always consistent w/ whatever is reported in the NextDNS logs, NOT whatever network interface that particular client happens to be using.

In the end, I find the issue moot. All that matters is what connection tracking is reporting, NOT what any external diagnostic tools may be reporting, esp. since it's NOT always clear how they've been implemented. Third-party diagnostic tools often produce erroneous results, or results that are misinterpreted by the user. And why I continually tell ppl to check w/ connection tracking on the router. THAT is definitive!

 

[MvW]

Thanks for your effort, but I've given up on it. I just don't get half of what you're saying, the instructions are to complicated for me to comprehend which makes me feel really stupid, which is most likely caused by English not being my native language on one hand and on the other hand a lack of knowledge and a lot of real life sorrows on my head. I've reset both OpenVPN clients to their defaults, i.e. they're blank now, traffic is not being tunneled right know. As soon as I feel better I'm gonna start with one client only and see where that will lead me.

Thanks again for the time you were willing to invest but I'm just not up to this task, it's way over my head.

Best regards,
Marco