x3mRouting x3mRouting ~ Selective Routing for Asuswrt-Merlin Firmware

[Adamm]

Decided to dig into Selective Routing again recently, it seems something has changed in relation to the AX88U?

A small test to confirm this, visit (http://whatismyip.host/my-ip-address-details) and note your IP address, then;

ip rule add fwmark 0x8000/0x8000 table 254 prio 9990

ip rule add fwmark 0x1000/0x1000 table 111 prio 9991

iptables -I PREROUTING -t mangle -d 34.233.244.94 -j MARK --set-mark 0x1000/0x1000

Now when visiting (http://whatismyip.host/my-ip-address-details) you should see your VPN IP.


It seems packets are being marked correctly by IPTables as the counter is increasing, but they are never forwarded to OpenVPN Client 1;

skynet@RT-AX88U-DC28:/tmp/home/root# iptables -t mangle -vL PREROUTING
Chain PREROUTING (policy ACCEPT 1200 packets, 393K bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 MARK       all  --  tun21  any     anywhere             anywhere             MARK xset 0x1/0x7
    5   480 MARK       all  --  tun11  any     anywhere             anywhere             MARK xset 0x1/0x7
 2483  626K BWDPI_FILTER  udp  --  ppp0   any     anywhere             anywhere
   15  1932 MARK       all  --  any    any     anywhere             ec2-34-233-244-94.compute-1.amazonaws.com  MARK or 0x1000

skynet@RT-AX88U-DC28:/tmp/home/root# ip rule
0:    from all lookup local
9990:    from all fwmark 0x8000/0x8000 lookup main
9991:    from all fwmark 0x1000/0x1000 lookup ovpnc1
32766:    from all lookup main
32767:    from all lookup default

 

[Xentrk]

Decided to dig into Selective Routing again recently, it seems something has changed in relation to the AX88U?

A small test to confirm this, visit (http://whatismyip.host/my-ip-address-details) and note your IP address, then;

ip rule add fwmark 0x8000/0x8000 table 254 prio 9990

ip rule add fwmark 0x1000/0x1000 table 111 prio 9991

iptables -I PREROUTING -t mangle -d 34.233.244.94 -j MARK --set-mark 0x1000/0x1000

Now when visiting (http://whatismyip.host/my-ip-address-details) you should see your VPN IP.


It seems packets are being marked correctly by IPTables as the counter is increasing, but they are never forwarded to OpenVPN Client 1;

skynet@RT-AX88U-DC28:/tmp/home/root# iptables -t mangle -vL PREROUTING
Chain PREROUTING (policy ACCEPT 1200 packets, 393K bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 MARK       all  --  tun21  any     anywhere             anywhere             MARK xset 0x1/0x7
    5   480 MARK       all  --  tun11  any     anywhere             anywhere             MARK xset 0x1/0x7
 2483  626K BWDPI_FILTER  udp  --  ppp0   any     anywhere             anywhere
   15  1932 MARK       all  --  any    any     anywhere             ec2-34-233-244-94.compute-1.amazonaws.com  MARK or 0x1000

skynet@RT-AX88U-DC28:/tmp/home/root# ip rule
0:    from all lookup local
9990:    from all fwmark 0x8000/0x8000 lookup main
9991:    from all fwmark 0x1000/0x1000 lookup ovpnc1
32766:    from all lookup main
32767:    from all lookup default

I noticed the iptables you posted is missing the "-i br0" parameter. Try this:

iptables -I PREROUTING -t mangle -i br0 -d 34.233.244.94 -j MARK --set-mark 0x1000/0x1000

 

[PeterV]

It seems packets are being marked correctly by IPTables as the counter is increasing, but they are never forwarded to OpenVPN Client 1;

Additionally to this recommendation

I noticed the iptables you posted is missing the "-i br0" parameter. Try this:

iptables -I PREROUTING -t mangle -i br0 -d 34.233.244.94 -j MARK --set-mark 0x1000/0x1000

please check the result of

ip route show table 111

 

[Adamm]

I noticed the iptables you posted is missing the "-i br0" parameter. Try this:

iptables -I PREROUTING -t mangle -i br0 -d 34.233.244.94 -j MARK --set-mark 0x1000/0x1000

Doesn't make a difference, br0 traffic would have been "filtered" regardless as no interface was specified. I assume the issue lies with the fwmark rules, just not sure exactly whats changed in the last few releases.

Additionally to this recommendation

please check the result of

ip route show table 111

skynet@RT-AX88U-DC28:/tmp/home/root# ip route show table 111
10.84.10.5 dev tun11  proto kernel  scope link  src 10.84.10.6
192.168.1.0/24 dev br0  proto kernel  scope link  src 192.168.1.1

 

[PeterV]

Doesn't make a difference, br0 traffic would have been "filtered" regardless as no interface was specified. I assume the issue lies with the fwmark rules, just not sure exactly whats changed in the last few releases.

skynet@RT-AX88U-DC28:/tmp/home/root# ip route show table 111
10.84.10.5 dev tun11  proto kernel  scope link  src 10.84.10.6
192.168.1.0/24 dev br0  proto kernel  scope link  src 192.168.1.1

It's no default route rule inside this table. This is a root cause for your problem.
You can do it manually, or simple add a DummyVPN record on OpenVPN Client Screen ~ Policy Routing Section

 

[Adamm]

It's no default route rule inside this table. This is a root cause for your problem.
You can do it manually, or simple add a DummyVPN record on OpenVPN Client Screen ~ Policy Routing Section

Interesting, was never aware of this previously. Thanks for the tip.

Edit; Do you have the syntax to achieve this manually? Bit sloppy on my ip route stuff

skynet@RT-AX88U-DC28:/tmp/home/root# ip route show table 111
default via 10.66.10.5 dev tun11
10.66.10.5 dev tun11 proto kernel scope link src 10.66.10.6
192.168.1.0/24 dev br0 proto kernel scope link src 192.168.1.1

 

[PeterV]

Interesting, was never aware of this previously. Thanks for the tip.

Edit; Do you have the syntax to achieve this manually? Bit sloppy on my ip route stuff

Will try, but with for 100% it will be right recommendation
As usually, before recommend something, I have to check, how it works. But now I have no access to device now.
At least it have to be something like

ip route add default via 10.84.10.5 dev tun11 table 111

I.e. after via have to be placed other side of vpn tunnel.
That's why it's better to use DummyVPN, to process the route via scripts

 

[Adamm]

Will try, but with for 100% it will be right recommendation
As usually, before recommend something, I have to check, how it works. But now I have no access to device now.
At least it have to be something like

ip route add default via 10.84.10.5 dev tun11 table 111

I.e. after via have to be placed other side of vpn tunnel.
That's why it's better to use DummyVPN, to process the route via scripts

Yeah experimenting with it now its probably easier to use the dummy rule approach as the OpenVPN local IP is dynamic and there's no clean consistent way to retrieve it. Thanks again for the help.

 

[Martineau]

@Martineau thanks for the update. However, comparing your iptables rules with @kman there's a minor difference which I don't understand.

iptables -t mangle -A PREROUTING -i tun21 -m set --match-set PANDORA dst -j MARK --set-xmark 0x4000/0x4000

iptables -t mangle -A PREROUTING -i tun21 -m set --match-set PANDORA dst -j MARK --set-mark 0x1000/0x1000

See Selective Routing fwmarks for my unofficial recommended usage.

@MartineauThe --set-mark vs --set-xmark flag seems to be different, why is this?

See the appropriate section of the iptables man page

MARK

This target is used to set the Netfilter mark value associated with the packet. It can, for example, be used in conjunction with routing based on fwmark (needs iproute2). If you plan on doing so, note that the mark needs to be set in the PREROUTING chain of the mangle table to affect routing. The mark field is 32 bits wide.

--set-xmark value[/mask]

     Zeroes out the bits given by mask and XORs value into the packet mark ("nfmark"). If mask is omitted, 0xFFFFFFFF is assumed.

--set-mark value[/mask]

     Zeroes out the bits given by mask and ORs value into the packet mark. If mask is omitted, 0xFFFFFFFF is assumed.

Read the Boolean Algebra WiKi for an explanation of the terms XOR and OR, but basically, for the Selective IPSET Routing, since Asus/Broadcom use (undisclosed) fwmark bits for their own purposes, we need to ensure that only 1-bit is modified for our own unofficial use...hence we want to ensure the second line of the table is always applied

where

in1 is the current value
in2 is the desired value
out is the result​

Perhaps, @kman should use '--set-mark' to ensure the desired setting of the single RPDB usage bit is not inadvertently cleared by line 4.

 

[kman]

Good catch @lazmo88 . I didn't even realize the subtle difference between xmark vs mark. I just used @Xentrk script load_DNSMASQ_ipset_iface.sh to get the syntax.

FYI, I just updated Create_Routing_Rules() in load_DNSMASQ_ipset_iface.sh to include the rules.

# Route IPSET to target WAN or VPN
Create_Routing_Rules() {

  IPSET_NAME=$1

  iptables -t mangle -D PREROUTING -i br0 -m set --match-set "$IPSET_NAME" dst -j MARK --set-mark "$TAG_MARK" >/dev/null 2>&1
  #VPN Server 1
  iptables -t mangle -D PREROUTING -i tun21 -m set --match-set "$IPSET_NAME" dst -j MARK --set-mark "$TAG_MARK" >/dev/null 2>&1
  iptables -D POSTROUTING -t nat -s $(nvram get vpn_server1_sn)/24 -o tun11 -j MASQUERADE >/dev/null 2>&1
  if [ "$2" != "del" ]; then
    iptables -t mangle -A PREROUTING -i br0 -m set --match-set "$IPSET_NAME" dst -j MARK --set-mark "$TAG_MARK"
    #VPN Server 1
    iptables -t mangle -A PREROUTING -i tun21 -m set --match-set "$IPSET_NAME" dst -j MARK --set-mark "$TAG_MARK" 
    iptables -I POSTROUTING -t nat -s $(nvram get vpn_server1_sn)/24 -o tun11 -j MASQUERADE
    logger -st "($(basename "$0"))" $$ Selective Routing Rule via "$TARGET_DESC" created for "$IPSET_NAME" TAG fwmark "$TAG_MARK"
  else
    logger -st "($(basename "$0"))" $$ Selective Routing Rule via "$TARGET_DESC" deleted for "$IPSET_NAME" TAG fwmark "$TAG_MARK"
  fi
}

Thanks @Martineau for the guidance.

 

[Xentrk]

Good catch @lazmo88 . I didn't even realize the subtle difference between xmark vs mark. I just used @Xentrk script load_DNSMASQ_ipset_iface.sh to get the syntax.

FYI, I just updated Create_Routing_Rules() in load_DNSMASQ_ipset_iface.sh to include the rules.

# Route IPSET to target WAN or VPN
Create_Routing_Rules() {

  IPSET_NAME=$1

  iptables -t mangle -D PREROUTING -i br0 -m set --match-set "$IPSET_NAME" dst -j MARK --set-mark "$TAG_MARK" >/dev/null 2>&1
  #VPN Server 1
  iptables -t mangle -D PREROUTING -i tun21 -m set --match-set "$IPSET_NAME" dst -j MARK --set-mark "$TAG_MARK" >/dev/null 2>&1
  iptables -D POSTROUTING -t nat -s $(nvram get vpn_server1_sn)/24 -o tun11 -j MASQUERADE >/dev/null 2>&1
  if [ "$2" != "del" ]; then
    iptables -t mangle -A PREROUTING -i br0 -m set --match-set "$IPSET_NAME" dst -j MARK --set-mark "$TAG_MARK"
    #VPN Server 1
    iptables -t mangle -A PREROUTING -i tun21 -m set --match-set "$IPSET_NAME" dst -j MARK --set-mark "$TAG_MARK"
    iptables -I POSTROUTING -t nat -s $(nvram get vpn_server1_sn)/24 -o tun11 -j MASQUERADE
    logger -st "($(basename "$0"))" $$ Selective Routing Rule via "$TARGET_DESC" created for "$IPSET_NAME" TAG fwmark "$TAG_MARK"
  else
    logger -st "($(basename "$0"))" $$ Selective Routing Rule via "$TARGET_DESC" deleted for "$IPSET_NAME" TAG fwmark "$TAG_MARK"
  fi
}

Thanks @Martineau for the guidance.

Thanks for posting your solution @kman. I'll document the instructions on how to implement on the project GitHub page or create a wiki page.

And yes, thank you @Martineau for sharing your expertise and knowledge. Very much appreciated!

 

[Xentrk]

hello @Xentrk
i dont know if my requirement is related to the ones reported lately. i would appreciate your help whenever you can!

Your welcome. What is your specific requirement? Routing traffic to the WAN1 iface or routing VPN server traffic to use the rules created by x3mRouting?

 

[lazmo88]

Good catch @lazmo88 . I didn't even realize the subtle difference between xmark vs mark. I just used @Xentrk script load_DNSMASQ_ipset_iface.sh to get the syntax.

FYI, I just updated Create_Routing_Rules() in load_DNSMASQ_ipset_iface.sh to include the rules.

# Route IPSET to target WAN or VPN
Create_Routing_Rules() {

  IPSET_NAME=$1

  iptables -t mangle -D PREROUTING -i br0 -m set --match-set "$IPSET_NAME" dst -j MARK --set-mark "$TAG_MARK" >/dev/null 2>&1
  #VPN Server 1
  iptables -t mangle -D PREROUTING -i tun21 -m set --match-set "$IPSET_NAME" dst -j MARK --set-mark "$TAG_MARK" >/dev/null 2>&1
  iptables -D POSTROUTING -t nat -s $(nvram get vpn_server1_sn)/24 -o tun11 -j MASQUERADE >/dev/null 2>&1
  if [ "$2" != "del" ]; then
    iptables -t mangle -A PREROUTING -i br0 -m set --match-set "$IPSET_NAME" dst -j MARK --set-mark "$TAG_MARK"
    #VPN Server 1
    iptables -t mangle -A PREROUTING -i tun21 -m set --match-set "$IPSET_NAME" dst -j MARK --set-mark "$TAG_MARK"
    iptables -I POSTROUTING -t nat -s $(nvram get vpn_server1_sn)/24 -o tun11 -j MASQUERADE
    logger -st "($(basename "$0"))" $$ Selective Routing Rule via "$TARGET_DESC" created for "$IPSET_NAME" TAG fwmark "$TAG_MARK"
  else
    logger -st "($(basename "$0"))" $$ Selective Routing Rule via "$TARGET_DESC" deleted for "$IPSET_NAME" TAG fwmark "$TAG_MARK"
  fi
}

Thanks @Martineau for the guidance.

Does your updated script do all necessary steps to enable selective routing for VPN server clients > VPN client or is there something else to consider than this updated script? I see that it only routes VPN server tun21, can this be made universal by using tun2+ and tun1+ so it would affect all VPN server and client instances and then use the selective routing GUI or iface scripts to specify which VPN client instance to use?

I think I was only missing the --set-mark rule, but I'll need to verify this manually when I get time.

@Martineau, I wonder if you have ideas to enhance the selective routing GUI to include the options to select which interface IPSET gets routed through?

 

[Martineau]

@Martineau, I wonder if you have ideas to enhance the selective routing GUI to include the options to select which interface IPSET gets routed through?

I originally documented/discussed the concept/feasibility of Selective Routing of IPSETs via the GUI here.

However, whilst I still personally use the patched firmware, I decided that it probably wasn't worth the time and effort to formally support this extremely useful feature although @Xentrk included it in his project
x3mRouting Project: OpenVPN Client Screen

 

[andresmorago]

Your welcome. What is your specific requirement? Routing traffic to the WAN1 iface or routing VPN server traffic to use the rules created by x3mRouting?

Hello @Xentrk
I have several clients connected to the vpn server on my router.
The router is also acting as a vpn client.

routers vpn server clients are routed by default to vpn client 1. I would requiere then to follow the same selective routing that regular devices that are directly connected to the router have.

 

[Xentrk]

Hello @Xentrk
I have several clients connected to the vpn server on my router.
The router is also acting as a vpn client.

routers vpn server clients are routed by default to vpn client 1. I would requiere then to follow the same selective routing that regular devices that are directly connected to the router have.

@kman got it working a few posts above. I'll take a look to see how I can add an option to make the implementation more user friendly so you don't have to edit the code.

 

[lazmo88]

@kman got it working a few posts above. I'll take a look to see how I can add an option to make the implementation more user friendly so you don't have to edit the code.

Would it be possible to duplicate the script, make the edits and just run it? Or does the iface script depend on other scripts?

 

[andresmorago]

@kman got it working a few posts above. I'll take a look to see how I can add an option to make the implementation more user friendly so you don't have to edit the code.

Sorry I missed it. Will read above and try.

 

[Xentrk]

Would it be possible to duplicate the script, make the edits and just run it? Or does the iface script depend on other scripts?

The line below is routing VPN Server traffic with the IPv4 addresses specified in the IPSET list to the fwmark defined for the interface. For example, if you specify iface 1 when running load_DNSMASQ_ipset_iface.sh, it will set the fwmark to 0x1000/0x1000.

Note the suttle change in syntax: --set-xmark

iptables -t mangle -D PREROUTING -i tun21 -m set --match-set "$IPSET_NAME" dst -j MARK --set-xmark "$TAG_MARK" >/dev/null 2>&1
iptables -t mangle -A PREROUTING -i tun21 -m set --match-set "$IPSET_NAME" dst -j MARK --set-xmark "$TAG_MARK"

So editing the script like @kman should take care of it.

These lines force all VPN Server clients to get routed to OpenVPN Client 1 (e.g. tun11).

iptables -D POSTROUTING -t nat -s $(nvram get vpn_server1_sn)/24 -o tun11 -j MASQUERADE >/dev/null 2>&1
iptables -I POSTROUTING -t nat -s $(nvram get vpn_server1_sn)/24 -o tun11 -j MASQUERADE

I am leaning towards a separate script for routing all VPN Server traffic to an OpenVPN client and another for routing VPN Server Traffic using an IPSET list to one of the OpenVPN interfaces. The user would pass parameters to the script to specify the VPN Server interface of the incoming traffic and the IPSET list and OpenVPN client interface you want to route the VPN Server traffic to.

I'll start working on the updates this upcoming weekend.

 

[kman]

Would it be possible to duplicate the script, make the edits and just run it? Or does the iface script depend on other scripts?

I'm just duplicating the script and running it without any other scripts. Using it as a standalone script.