Policy routing with 2 VPN connections

[a665368]

Hi all, I'm enjoying experimenting with Merlin's firmware (donation on its way, thanks for such a great resource). I'm having some problems setting up policy routing with 2 VPN connections, although I can get it working with 1 VPN connection. If anyone can point me in the right direction that'd be really great.

I've got two VPN connections established (clients 1 and 2). I want to use each of those for certain traffic and use the WAN connection for everything else.

I've set up dnsmasq.conf.add like this and I know it's executing because I can list the ipsets later and see their entries being added to over time:

ipset=/abc.com/www.abc.com/abc
ipset=/def.com/www.def.com/def
ipset=/xyz.com/www.xyz.com/xyz

I've set up services-start like this and I know it's executing:

#!/bin/sh
service restart_dnsmasq
modprobe xt_set
ipset create abc hash:net family inet hashsize 1024 maxelem 65536
ipset create def hash:ip,port
ipset create xyz hash:net family inet hashsize 1024 maxelem 65536

I've set up nat-start like this and I know it's executing:

#!/bin/sh
ipset add def 104.25.113.26,80
ipset add def 104.25.112.26,80

# Set up the routing rules
# Rule for VPN client 1
ip rule del fwmark 0x1000
ip rule add fwmark 0x1000 table ovpnc1 prio 9991
# Rule for VPN client 2
ip rule del fwmark 0x2000
ip rule add fwmark 0x2000 table ovpnc2 prio 9992

ip route flush cache

# By default all traffic goes through the WAN

# Set up routes for the ipsets
iptables -t mangle -A PREROUTING -i br0 -p tcp -m set --match-set abc src,dst -j MARK --set-mark 0x1000/0x1000
iptables -t mangle -A PREROUTING -i br0 -p tcp -m set --match-set def src,dst -j MARK --set-mark 0x1000/0x1000
iptables -t mangle -A PREROUTING -i br0 -p tcp -m set --match-set xyz src,dst -j MARK --set-mark 0x2000/0x2000

I'm sure I must have missed something basic but if someone can help me out I'd sure appreciate it. Thanks!

EDIT: I should have added that all traffic is still going through the WAN so the rules I've set up maybe aren't even being used.

 

[Xentrk]

Did you try using the policy routing feature available in the web gui for your use case? Guide link is here:
https://www.snbforums.com/threads/h...r-pia-and-other-vpn-providers-380-65_4.30851/

In a nutshell, you assign static ip addresses for each client. In the OpenVPN Client page, you then specify traffic as policy rules and specify which ip addresses should go thru the vpn tunnel.

 

[a665368]

Hi Xentrk, thanks for your reply. I did experiment with that and it works really well for many cases, but I'd like the flexibility to direct traffic for any ipset through any interface.

My ipsets could contain URLs, IP addresses, IP addresses with port numbers, anything really. For example, I want all the devices on the LAN to be able to access:
- a particular web server (by IP address) on port 80 through VPN1
- a group of web servers (by names) through VPN1
- another web server (by names) via https through VPN2
- mail servers through VPN2
- other web servers through the WAN.

 

[Martineau]

I'm sure I must have missed something basic but if someone can help me out I'd sure appreciate it. Thanks!

EDIT: I should have added that all traffic is still going through the WAN so the rules I've set up maybe aren't even being used.

Ensure you have 'Policy Rules' enabled for both VPN Clients, and check the tables

ip route show table 111
ip route show table 112

then check the selective routing

iptables -nvL PREROUTING -t mangle --line

ip   rule

Adding entries to IPSETs requires the IPSET to already exist.... so are the ipset add entries in nat-start actually being added to the 'def' IPSET?

ipset   list   def | grep  "104.25.113.26"

EDIT: If your intention is for a match on a target socket, then shouldn't the iptables rule be changed:

iptables -t mangle -A PREROUTING -i br0 -p tcp -m set --match-set def src,dst -j MARK --set-mark 0x1000/0x1000

     changed to

iptables -t mangle -A PREROUTING -i br0 -p tcp -m set --match-set def dst,dst -j MARK --set-mark 0x1000/0x1000

 

[a665368]

Hi Martineau, thanks for your reply, it was very helpful. My iptables statements weren't properly formed.

I've managed to get it working as follows, although I wonder if my use of ACCEPT could have side-effects. I have some web servers in the VPN1 rules which were being over-ridden by my last rule, which directs traffic to ports 80 and 443 via VPN2, so I added ACCEPT after each to avoid that. Is there a better way to do this? Thank you.

/jffs/scripts/nat-start

#!/bin/sh

ip rule del prio 9990
ip rule add fwmark 7 table main prio 9990
ip rule del fwmark 1
ip rule add fwmark 1 table ovpnc1 prio 9991
ip rule del fwmark 2
ip rule add fwmark 2 table ovpnc2 prio 9992

ip route flush cache

# WAN (traffic that must be FORCED to go via ISP)
iptables -v -t mangle -A PREROUTING -i br0 -p all -m set --match-set wan-force  dst -j MARK --set-mark 0x7000
iptables -v -t mangle -A PREROUTING -i br0 -p all -m set --match-set wan-force  dst -j ACCEPT

# VPN 1
iptables -v -t mangle -A PREROUTING -i br0 -p all -m set --match-set vpn-1a   dst -j MARK --set-mark 0x1000
iptables -v -t mangle -A PREROUTING -i br0 -p all -m set --match-set vpn-1a   dst -j ACCEPT
iptables -v -t mangle -A PREROUTING -i br0 -p all -m set --match-set vpn-1b   dst -j MARK --set-mark 0x1000
iptables -v -t mangle -A PREROUTING -i br0 -p all -m set --match-set vpn-1b   dst -j ACCEPT

# VPN 2
iptables -v -t mangle -A PREROUTING -i br0 -p all -m set --match-set vpn-2a   dst -j MARK --set-mark 0x2000
iptables -v -t mangle -A PREROUTING -i br0 -p all -m set --match-set vpn-2a   dst -j ACCEPT
iptables -v -t mangle -A PREROUTING -i br0 -p all -m set --match-set vpn-2b   dst -j MARK --set-mark 0x2000
iptables -v -t mangle -A PREROUTING -i br0 -p all -m set --match-set vpn-2b   dst -j ACCEPT

# All http and https traffic to go via VPN 2
iptables -v -t mangle -A PREROUTING -i br0 -p tcp -m multiport --dports 80,443 -j MARK --set-mark 0x2000

# Everything else goes via WAN (ISP)

 

[a665368]

Not working quite right. I had to abandon my last rule of directing all http and https traffic via VPN 2 because it turned out there were a couple of services that make http and https calls that have to go via the WAN and I couldn't find all of them.

I also ran into a problem where, immediately after rebooting the router, VPN client 1 said that it had a routing conflict (VPN client 2 was ok) and where nothing would go through VPN client 1 even after stopping and starting the client. I wondered if this was the problem Martineau described:
https://www.snbforums.com/threads/o...service-state-apply-button.30410/#post-237935

I guess it could be scripted so that if no-one is there to reboot the router again, it can fix itself, e.g. by running the route flush Martineau described and restarting the VPN client, if some condition is met, but my scripting skills are not that good.

Thanks to anyone who can offer a suggestion. It's been a good weekend project.

 

[Martineau]

I had to abandon my last rule of directing all http and https traffic via VPN 2 because it turned out there were a couple of services that make http and https calls that have to go via the WAN and I couldn't find all of them.

If you can't identify which http/https requests should explicitly be routed via the VPNs then it is difficult to propose a solution.

NOTE: Because '-j MARK' is non-terminating, you could reverse the rules so that the 'most generic' MARKS are matched/applied first, then the 'most specific' MARKs are matched/applied last - saves having to add the duplicate ACCEPT rules.

However, keeping the rules in the more intuitive 'exact' down to 'generic' order, then use of the 'duplicate' ACCEPT rules can prove to be more efficient (and less prone to error!), as it saves having to trundle needlessly through the other rules if you have just processed an actual 'exact' match.

I also ran into a problem where, immediately after rebooting the router, VPN client 1 said that it had a routing conflict (VPN client 2 was ok) and where nothing would go through VPN client 1 even after stopping and starting the client. I wondered if this was the problem Martineau described:
https://www.snbforums.com/threads/o...service-state-apply-button.30410/#post-237935

Ensure that your VPN ISP does actually allow multiple VPN instances from the same device. Not all VPN providers do.

My ISP provider does allow multiple connections from the router, and although I still have a VPN start-up script to ensure the 'conflicting routes' are removed, my original routing conflict problem went away after one of the recent firmware upgrades.

ip rule add fwmark 7 table main prio 9990
ip rule add fwmark 1 table ovpnc1 prio 9991
ip rule add fwmark 2 table ovpnc2 prio 9992

Shouldn't fwmarks 1,2 and 7 actually be 0x1000,0x2000 and 0x7000 ??

Also I strongly advise when setting the fwmarks that you always include the mask 0xNNNN/0xNNNN in the iptable rules , to ensure you preserve any other fwmarks on the packet that are already set by the firmware, unless of course you want to explicitly wipe ALL of them except yours!

I personally create the following static rules for IPSET matching...so no need for further iptables IPSET selective routing rules in the future! ?

iptables -nvL PREROUTING -t mangle --line

Chain PREROUTING (policy ACCEPT 6244K packets, 5160M bytes)
num   pkts bytes target     prot opt in     out     source               destination
<snip>
2        0     0 MARK       tcp  --  br0    *       0.0.0.0/0            0.0.0.0/0            match-set WAN0 src,dst,dst MARK or 0x7000
3        0     0 MARK       tcp  --  br0    *       0.0.0.0/0            0.0.0.0/0            match-set VPN1 src,dst,dst MARK or 0x1000
4        0     0 MARK       tcp  --  br0    *       0.0.0.0/0            0.0.0.0/0            match-set VPN2 src,dst,dst MARK or 0x2000
5        0     0 MARK       tcp  --  br0    *       0.0.0.0/0            0.0.0.0/0            match-set VPN3 src,dst,dst MARK or 0x3000
6        0     0 MARK       tcp  --  br0    *       0.0.0.0/0            0.0.0.0/0            match-set VPN4 src,dst,dst MARK or 0x4000
7        0     0 MARK       tcp  --  br0    *       0.0.0.0/0            0.0.0.0/0            match-set VPN5 src,dst,dst MARK or 0x5000
8        0     0 MARK       tcp  --  br0    *       0.0.0.0/0            0.0.0.0/0            match-set WAN0D dst,dst MARK or 0x7000
9        0     0 MARK       tcp  --  br0    *       0.0.0.0/0            0.0.0.0/0            match-set VPN1D dst,dst MARK or 0x1000
10       0     0 MARK       tcp  --  br0    *       0.0.0.0/0            0.0.0.0/0            match-set VPN2D dst,dst MARK or 0x2000
11       0     0 MARK       tcp  --  br0    *       0.0.0.0/0            0.0.0.0/0            match-set VPN3D dst,dst MARK or 0x3000
12       0     0 MARK       tcp  --  br0    *       0.0.0.0/0            0.0.0.0/0            match-set VPN4D dst,dst MARK or 0x4000
13       0     0 MARK       tcp  --  br0    *       0.0.0.0/0            0.0.0.0/0            match-set VPN5D dst,dst MARK or 0x5000
14       0     0 MARK       tcp  --  br0    *       0.0.0.0/0            0.0.0.0/0            multiport sports 5000:5001 MARK or 0x7000
15       0     0 MARK       tcp  --  br0    *       10.88.8.111          0.0.0.0/0            multiport dports 80,443 MARK or 0x1000

So now I only need to simply create/manage individual IPSETs containing destinations/ports to the appropriate high-level WAN/VPN target.

e.g. IPSET list VPN1 contains several IPSETs (in this case 6) whose members are routed via VPN Client 1 New York (aka ovpnc1) via the single iptables PREROUTING rule:

ipset list VPN1

     Name: VPN1
     Type: list:set
     Revision: 0
     Header: size 8
     Size in memory: 56
     References: 1
     Members:
        VPN1Domains
        VPN1IPs
        VPN1IPPorts
        CBS_Shows
        Hulu
        Netflix

This is quite flexible, although the downside to using tiers of IPSET lists rather than discrete iptables rules is that you can't see quickly how much traffic is being individually routed

i.e. VPN1 may contain in total, tens or hundreds of members in the individual IPSETs so how would you know how many PREROUTING 'match-set VPN1 pkts' were contributed by say the IPSET 'CBS_Shows' ?

Hope this is useful.

 

[a665368]

Thanks Martineau, that was really helpful! I now have a list of lists, much easier to manage.

Is there any way to dynamically modify an ipset consisting of host or domain names? For example, in dnsmasq.conf.add we can initialise a set:

ipset=/google.com/yahoo.com/searchengineset

But I can't see an ipset command that can add duckduckgo.com to that set, or remove yahoo.com from that set, without changing the dnsmasq.conf.add file and restarting the dnsmasq service.

With CDNs now being used for video streaming services and their geoblocking techniques, a lot of traffic still has to go through WAN0 otherwise the streaming is blocked. Probably it means that DNS queries also have to go through the same interface because if they went via a VPN instead, they'd most likely return addresses that didn't match with the CDN traffic going via WAN0 and the streaming would be blocked. I suppose dnsmasq could be configured to query the WAN0 DNS server for certain domains, and VPN DNS servers for other domains, but that's yet more configuration.

Also, I tried setting a rule for icmp traffic to use the WAN or VPN interface, to match the ipsets for those interfaces. The results were strange - the route seemed to start in the right direction, traverse the world, and then head back to the country I'm actually in. Don't know how that worked! But it was helpful to test whether the ipsets were working.

It continues to be an interesting project.

 

[a665368]

Hi again, does anyone know the answer re adding domains to the dnsmasq set on the fly please?

 

[Xentrk]

I tested the script written by the OP with some modifications. A test file listing the client IP addresses for VPN Client 1 and 2 are are stored in files. The script will create and ipset list for the WAN traffic and read from these files to create ipset lists for VPN Client 1 and 2. Use editor of your choice to populate the IP addresses in the files.

#!/bin/sh
# Uncomment the line below for debugging
#set -x
ipset create LAN_GW hash:net family inet hashsize 1024 maxelem 65536
ipset create OVPNC1 hash:net family inet hashsize 1024 maxelem 65536
ipset create OVPNC2 hash:net family inet hashsize 1024 maxelem 65536

# extract LAN ip addresses
ipset add LAN_GW $(nvram get lan_ipaddr)

# extract OVPNC1 ip addresses
OVPNC1_FILE="/jffs/scripts/OVPNC1"
OUT1=$(awk '{ print $1 }' $OVPNC1_FILE)
  for ip in $OUT1
    do
      ipset add OVPNC1 $ip
  done

# extract OVPNC2 ip addresses
OVPNC2_FILE="/jffs/scripts/OVPNC2"
OUT2=$(awk '{ print $1 }' $OVPNC2_FILE)
  for ip in $OUT2
    do
      ipset add OVPNC2 $ip
  done

ip rule del fwmark 0x7000/0x7000
ip rule add fwmark 0x7000/0x7000 table 254 prio 9990

#VPN Client 1
ip rule del fwmark 0x1000/0x1000
ip rule add fwmark 0x1000/0x1000 table 111 prio 9991

#VPN Client 2
ip rule del fwmark 0x2000/0x2000
ip rule add fwmark 0x2000/0x2000 table 112 prio 9992

ip route flush cache

###########################################################
# wan
###########################################################
iptables -t mangle -D PREROUTING -i br0 -p tcp -m set --match-set LAN_GW src,dst -j MARK --set-mark 0x7000/0x7000
iptables -t mangle -A PREROUTING -i br0 -p tcp -m set --match-set LAN_GW src,dst -j MARK --set-mark 0x7000/0x7000

###########################################################
# OVPNC1
###########################################################
iptables -t mangle -D PREROUTING -i br0 -p tcp -m set --match-set OVPNC1 src,dst -j MARK --set-mark 0x1000/0x1000
iptables -t mangle -A PREROUTING -i br0 -p tcp -m set --match-set OVPNC1 src,dst -j MARK --set-mark 0x1000/0x1000

###########################################################
# OVPNC2
###########################################################
iptables -t mangle -D PREROUTING -i br0 -p tcp -m set --match-set OVPNC2 src,dst -j MARK --set-mark 0x2000/0x2000
iptables -t mangle -A PREROUTING -i br0 -p tcp -m set --match-set OVPNC2 src,dst -j MARK --set-mark 0x2000/0x2000

To change routing on demand, delete the entry from one ipset list and add it to the other:

ipset del OVPNC2 192.168.10.152

ipset add OVPNC1 192.168.10.152