What's new

iptables, ipsets and VPN 'kill switch'

  • SNBForums Code of Conduct

    SNBForums is a community for everyone, no matter what their level of experience.

    Please be tolerant and patient of others, especially newcomers. We are all here to share and learn!

    The rules are simple: Be patient, be nice, be helpful or be gone!

ipset-dns looks like a promising utility!
Code:
# opkg install ipset-dns
Installing ipset-dns (2017-10-08-ade2cf88-1) to root...
Downloading http://bin.entware.net/armv7sf-k2.6/ipset-dns_2017-10-08-ade2cf88-1_armv7-2.6.ipk
Configuring ipset-dns.
# ipset-dns x3mRouting_BBC 53 10.9.0.1
Usage: ipset-dns ipv4-ipset ipv6-ipset port upstream
Oops!
According to the example, I need to create a entry in dnsmasq.conf.add:
server=/x3mRouting_BBC/127.0.0.1#39128

I then restarted dnsmasq. Then entered:

#ipset-dns x3mRouting_BBC 39128 10.9.0.1
Usage: ipset-dns ipv4-ipset ipv6-ipset port upstream

I'll have to pick it up later.
 
Based on this, /usr/sbin/updown.sh overrides can be done by placing my required updown.sh updates in /jffs/scripts/openvpn-event.

Correct??
Well.....it depends on what you need to customise, although I personally prefer to use a global method such as How to patch /usr/sbin/vpnrouting.sh updown.sh etc. for these firmware scripts, without having to explicitly find and replace all calling references to them.
On both versions, vpnrouting.sh is logging several error messages when trying to delete routes:
Code:
ERROR: Linux route delete command failed: external program exited with error status: 2
Explanation for 'ERROR: Linux route delete command failed: external program exited with error status: 2'

Is the _IPSET list entry in the web gui a dummy description or the actual IPSET list name?
It's the actual IPSET name (MAX length 12 chars even though the field accepts up to 15 chars)

i.e. Currently only the first character is reserved '_' but you may deduce that another two characters of the field may ultimately need to be commandeered to represent the IPSET dimension.
Is the IP address you are using for the IPSET list entry in the Web GUI also non reserved and assigned dhcp entry?
When using the Description field either to import an external rules file or selectively route an IPSET, then the dummy (place holder) source IP should be a value from a private IP range that doesn't overlap your LAN subnet.

e.g. my LAN is 10.xxx.xxx.xxx so I use 172.16.1.xxx for dummy VPN Client 1, 172.16.2.xxx for dummy VPN Client 2 etc.
So far, I have been unsuccessful in my attempts at the correct iptables syntax to route an ipset list to 10.9.0.1 in the DNVPN3 Chain.

Do I need a similar entry for the ipset list?
No?...although I'm not 100% sure what your IPSET 'x3mRouting_BBC' contains.

i.e. If IPSET 'x3mRouting_BBC' simply contains LAN IPs then the following might work:
Code:
iptables -t nat -A DNSVPN3 -m set --match-set x3mRouting_BBC src -j DNAT --to-destination 10.9.0.1

However, if your IPSET contains target destination URLs (as opposed to LAN devices) still unresolved such as 'ns1.recursive.dnsbycomodo.com' then I'm not sure what you are trying to achieve?
 
Last edited:
ipset-dns looks like a promising utility!
As per the declaration on the ipset-dns home page, isn't this what the 'ipset=' directive in dnsmasq now does?

i.e. dynamically saves the resolved IPs that are associated with a selected domain into an IPSET.
Code:
ipset=/fast.com/netflix.com/netflix.net/nflxext.com/nflximg.com/nflximg.net/nflxvideo.com/nflxvideo.net/amazonaws.com/NETFLIX
 
Well.....it depends on what you need to customise, although I personally prefer to use a global method such as How to patch /usr/sbin/vpnrouting.sh updown.sh etc. for these firmware scripts, without having to explicitly find and replace all calling references to them.

Explanation for 'ERROR: Linux route delete command failed: external program exited with error status: 2'


It's the actual IPSET name (MAX length 12 chars even though the field accepts up to 15 chars)

i.e. Currently only the first character is reserved '_' but you may deduce that another two characters of the field may ultimately need to be commandeered to represent the IPSET dimension.

When using the Description field either to import an external rules file or selectively route an IPSET, then the dummy (place holder) source IP should be a value from a private IP range that doesn't overlap your LAN subnet.

e.g. my LAN is 10.xxx.xxx.xxx so I use 172.16.1.xxx for dummy VPN Client 1, 172.16.2.xxx for dummy VPN Client 2 etc.

No?...although I'm not 100% sure what your IPSET 'x3mRouting_BBC' contains.

i.e. If IPSET 'x3mRouting_BBC' simply contains LAN IPs then the following might work:
Code:
iptables -t nat -A DNSVPN3 -m set --match-set x3mRouting_BBC src -j DNAT --to-destination 10.9.0.1

However, if your IPSET contains target destination URLs (as opposed to LAN devices) still unresolved such as 'ns1.recursive.dnsbycomodo.com' then I'm not sure what you are trying to achieve?
The clarification you provided helps a lot. I'll let you know how my development turns out. I may have other questions for you. Thanks again for being generous with sharing your knowledge.
 
As per the declaration on the ipset-dns home page, isn't this what the 'ipset=' directive in dnsmasq now does?

i.e. dynamically saves the resolved IPs that are associated with a selected domain into an IPSET.
Code:
ipset=/fast.com/netflix.com/netflix.net/nflxext.com/nflximg.com/nflximg.net/nflxvideo.com/nflxvideo.net/amazonaws.com/NETFLIX
You're right. I missed the explanation in bold text at the top of the web page. It was getting late in my time zone when I came across it. It is now replaced by the IPSET functionality built into dnsmasq.
Upstream Dnsmasq Support
This functionality has now been written directly into dnsmasq, which should be much easier to use than this project. See the --ipset option.


I did get the syntax to work though. A "" entry was missing which represents and IPV6 ipset list name.

e.g.
Code:
ipset-dns youtube "" 39128 8.8.8.8
ipset-dns netflix "" 39129 8.8.8.8

 
@Martineau,

I made changes to vpnrouting.sh and updown.sh to accomodate fwmarks, routes and assignment of VPN DNS when using Accept DNS Configuration = Exclusive.

I can only get the selective routing rules to work by ordering the PRIOR numbers per the example below, which is the reverse of how I would prefer for each OpenVPN client fwmark/bitmask.

Code:
0:      from all lookup local
9990:   from all fwmark 0x7000/0x7000 lookup main
9991:   from all fwmark 0x5000/0x5000 lookup ovpnc5
9992:   from all fwmark 0x4000/0x4000 lookup ovpnc4
9993:   from all fwmark 0x3000/0x3000 lookup ovpnc3
9994:   from all fwmark 0x2000/0x2000 lookup ovpnc2
9995:   from all fwmark 0x1000/0x1000 lookup ovpnc1
10101:  from 192.168.22.150 lookup ovpnc1
10102:  from 192.168.22.151 lookup ovpnc1
10103:  from 192.168.22.154 lookup ovpnc1
10301:  from 192.168.22.155 lookup ovpnc2
10302:  from 192.168.22.156 lookup ovpnc2
<snip>
10701:  from 192.168.22.152 lookup ovpnc4
10901:  from 192.168.22.153 lookup ovpnc5
32766:  from all lookup main
32767:  from all lookup default

I've had this issue ever since I started using 3 OpenVPN Clients in my setup last year. I tested this with 5 OpenVPN Clients running.

I tried testing using different combinations of the OpenVPN clients, i.e switching server locations between OVPNC1 and OVPNC3, with no impact. I get routing issues when ordering the fwmark/bitmask combo by OpenVPN client number. e.g.

Code:
9991:   from all fwmark 0x1000/0x1000 lookup ovpnc1
9992:   from all fwmark 0x2000/0x2000 lookup ovpnc2
9993:   from all fwmark 0x3000/0x3000 lookup ovpnc3
9994:   from all fwmark 0x4000/0x4000 lookup ovpnc4
9995:   from all fwmark 0x5000/0x5000 lookup ovpnc5

For example, if I define the website ipaddress.com to use ovpc3, it shows the IP of the OVPNC1 interface.

whatismyipaddress.com is assigned to the OVPNC5 interface. But it shows the IP of OVPNC1 interface.

In another example, one of the streaming services does not work unless the PRIO numbers are reversed. The streaming service is routed to OVPNC3. But the routing does not work unless OVPNC3 is listed before OVPNC2 and OVPNC1.

The DummyVPN1 entry appears in DNSVPN1 Chain, but does not appear as a routed client device as I added a filter in vpnrouting.sh:
Code:
# iptables --line -t nat -nvL DNSVPN1

Chain DNSVPN1 (2 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 DNAT       all  --  *      *       172.16.0.111         0.0.0.0/0            to:10.9.0.1
2        0     0 DNAT       all  --  *      *       192.168.22.150       0.0.0.0/0            to:10.9.0.1
3        0     0 DNAT       all  --  *      *       192.168.22.151       0.0.0.0/0            to:10.9.0.1
4        0     0 DNAT       all  --  *      *       192.168.22.154       0.0.0.0/0            to:10.9.0.1

Rather than having a separate iptable entry for each IPSET list, I now have one entry for each "master" OpenVPN Client interface ipset list.

Code:
# iptables -nvL PREROUTING -t mangle --line

Chain PREROUTING (policy ACCEPT 840K packets, 877M bytes)
num   pkts bytes target     prot opt in     out     source               destination
1     1181 1201K MARK       all  --  tun15  *       0.0.0.0/0            0.0.0.0/0            MARK xset 0x1/0x7
2    31043   27M MARK       all  --  tun14  *       0.0.0.0/0            0.0.0.0/0            MARK xset 0x1/0x7
3     4205 4555K MARK       all  --  tun13  *       0.0.0.0/0            0.0.0.0/0            MARK xset 0x1/0x7
4     274K  376M MARK       all  --  tun12  *       0.0.0.0/0            0.0.0.0/0            MARK xset 0x1/0x7
5     6176 6577K MARK       all  --  tun11  *       0.0.0.0/0            0.0.0.0/0            MARK xset 0x1/0x7
6      328 51120 MARK       tcp  --  br0    *       0.0.0.0/0            0.0.0.0/0            match-set WAN0 dst MARK or 0x7000
7     1545  243K MARK       tcp  --  br0    *       0.0.0.0/0            0.0.0.0/0            match-set OVPNC1 dst MARK or 0x1000
8     7801 2703K MARK       tcp  --  br0    *       0.0.0.0/0            0.0.0.0/0            match-set OVPNC2 dst MARK or 0x2000
9     2879  307K MARK       tcp  --  br0    *       0.0.0.0/0            0.0.0.0/0            match-set OVPNC3 dst MARK or 0x3000
10      20  1491 MARK       tcp  --  br0    *       0.0.0.0/0            0.0.0.0/0            match-set OVPNC4 dst MARK or 0x4000
11    5814 1876K MARK       tcp  --  br0    *       0.0.0.0/0            0.0.0.0/0            match-set OVPNC5 dst MARK or 0x5000
The downside is not being able to view the packet stats for each IPSET list. I am considering a feature in the script to give the user the ability to switch between the view above and a debug mode which will list iptables chain stats for each ipset list.
 
I made changes to vpnrouting.sh and updown.sh to accomodate fwmarks, routes and assignment of VPN DNS

Where are you placing the mount commands so it fires at boot time?
init-start

EDIT: March 2019 ...more elegant GUI to allow IPSETs to be correctly managed.

2019-03-22_22-24-39.png
upload_2018-8-25_14-37-29.png

So having embarked on hacking the firmware scripts ;), perhaps you may now wish to incorporate my clunky GUI hacks to accommodate your vpnrouting.sh tweaks? :D

upload_2018-8-25_14-37-29.png


e.g. Quick tutorial to allow use of versatile IPSETs via the Selective routing VPN Client Policy Routing table GUI.

Essentially we need to add two items to the GUI 'Iface' pull-down menu (and optionally change a title header etc. and modify the underlying vpnrouting.sh script to recognise the two new options which you have already done.

Surely it is about time this was added to the firmware?

1. Requires edits to '/www/Advanced_OpenVPNClient_Content.asp'
Code:
cp /www/Advanced_OpenVPNClient_Content.asp /jffs/scripts/Advanced_OpenVPNClient_Content.asp
Patches for Firmware v384.xx
Code:
sed -r -ibak 's/<th><#1707#><\/th>/<th>IPSET or <#1707#><\/th>   <!-- Martineau Hack -->/' /jffs/scripts/Advanced_OpenVPNClient_Content.asp
sed -r -i    '/<option value="WAN">WAN<\/option>/a <option value="SRC">SRC<\/option>   <!-- Martineau Hack -->/' /jffs/scripts/Advanced_OpenVPNClient_Content.asp
sed -r -i    '/<option value="WAN">WAN<\/option>/a <option value="DST">DST<\/option>   <!-- Martineau Hack -->/' /jffs/scripts/Advanced_OpenVPNClient_Content.asp
sed -r -i    's/alert\(\"<#1778#>\"\);/alert\(\"<#290#> \" + \"<#861#> \" + \"<#636#>\"\);   <!-- Martineau Hack -->/' /jffs/scripts/Advanced_OpenVPNClient_Content.asp
Code:
sed -r -ibak 's/<th width=\"24%\"><#935#><\/th>/<th width=\"24%\">IPSET or <#935#><\/th>   <!-- Martineau Hack -->/' /jffs/scripts/Advanced_OpenVPNClient_Content.asp
sed -r -i    '/<option value="WAN">WAN<\/option>/a <option value="SRC">SRC<\/option>   <!-- Martineau Hack -->/' /jffs/scripts/Advanced_OpenVPNClient_Content.asp
sed -r -i    '/<option value="WAN">WAN<\/option>/a <option value="DST">DST<\/option>   <!-- Martineau Hack -->/' /jffs/scripts/Advanced_OpenVPNClient_Content.asp
sed -r -i    's/alert\(\"<#996#>\"\);/alert\(\"<#197#> \" + \"<#102#> \" + \"<#418#>\"\);   <!-- Martineau Hack -->/' /jffs/scripts/Advanced_OpenVPNClient_Content.asp

2. Map the new custom VPN Client HTML page to the old page
Code:
mount -o bind /jffs/scripts/Advanced_OpenVPNClient_Content.asp /www/Advanced_OpenVPNClient_Content.asp

df

Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/root                34816     34816         0 100% /
devtmpfs                127744         0    127744   0% /dev
tmpfs                   127848      2224    125624   2% /tmp
/dev/mtdblock4           64256     17108     47148  27% /jffs
/dev/mtdblock4           64256     17108     47148  27% /usr/sbin/vpnrouting.sh
/dev/mtdblock4           64256     17108     47148  27% /www/Advanced_OpenVPNClient_Content.asp
/dev/mtdblock4           64256     17108     47148  27% /usr/sbin/updown.sh
3. Restart the web server to render the new custom GUI
Code:
service restart_httpd


I can only get the selective routing rules to work by ordering the PRIOR numbers per the example below, which is the reverse of how I would prefer for each OpenVPN client fwmark/bitmask.

Not 100% sure what issue you are describing....?o_O

i.e. rather than use prio 9990-9995, my idea was that exploiting unused RPDB rules 10000,10100,10300,10500,10700 and 10900 for fwmarks should be maintained by vpnrouting.sh but you must explicitly prevent vpnrouting.sh from 'accidentally' always'deleting prio 10000 whenever (and ONLY) VPN Client 1 is started/stopped. :eek:


 
Last edited:
Just a reminder.....part of the reason that Merlin implemented the vpnrouting via routing tables is that the TrendMicro modules had a habit of re-writing the iptables rules without notifying the rest of the firmware. Maybe that has changed, but I wouldn't count on it.
 
Just a reminder.....part of the reason that Merlin implemented the vpnrouting via routing tables is that the TrendMicro modules had a habit of re-writing the iptables rules without notifying the rest of the firmware. Maybe that has changed, but I wouldn't count on it.

I haven't forgotten (having done this for years), and I have always stated that to safeguard any custom firewall rules then use say

nat-start


to check if any VPN Clients are UP, and if necessary they are simply restarted (possibly inconvenient), and let either the openvpn-event (or modified vpnrouting.sh) scripts reinstate the desired custom rules.
 
init-start

So having embarked on hacking the firmware scripts ;), perhaps you may now wish to incorporate my clunky GUI hacks to accommodate your vpnrouting.sh tweaks? :D

View attachment 14203

e.g. Quick tutorial to allow use of versatile IPSETs via the Selective routing VPN Client Policy Routing table GUI.

Essentially we need to add two items to the GUI 'Iface' pull-down menu (and optionally change a title header etc. and modify the underlying vpnrouting.sh script to recognise the two new options which you have already done.

Surely it is about time this was added to the firmware?

1. Requires edits to '/www/Advanced_OpenVPNClient_Content.asp'
Code:
cp /www/Advanced_OpenVPNClient_Content.asp /jffs/scripts/Advanced_OpenVPNClient_Content.asp
Patches for Firmware v384.xx
Code:
sed -r -ibak 's/<th><#1707#><\/th>/<th>IPSET or <#1707#><\/th>   <!-- Martineau Hack -->/' /jffs/scripts/Advanced_OpenVPNClient_Content.asp
sed -r -i    '/<option value="WAN">WAN<\/option>/a <option value="SRC">SRC<\/option>   <!-- Martineau Hack -->/' /jffs/scripts/Advanced_OpenVPNClient_Content.asp
sed -r -i    '/<option value="WAN">WAN<\/option>/a <option value="DST">DST<\/option>   <!-- Martineau Hack -->/' /jffs/scripts/Advanced_OpenVPNClient_Content.asp
sed -r -i    's/alert\(\"<#1778#>\"\);/alert\(\"<#290#> \" + \"<#861#> \" + \"<#636#>\"\);   <!-- Martineau Hack -->/' /jffs/scripts/Advanced_OpenVPNClient_Content.asp
Code:
sed -r -ibak 's/<th width=\"24%\"><#935#><\/th>/<th width=\"24%\">IPSET or <#935#><\/th>   <!-- Martineau Hack -->/' /jffs/scripts/Advanced_OpenVPNClient_Content.asp
sed -r -i    '/<option value="WAN">WAN<\/option>/a <option value="SRC">SRC<\/option>   <!-- Martineau Hack -->/' /jffs/scripts/Advanced_OpenVPNClient_Content.asp
sed -r -i    '/<option value="WAN">WAN<\/option>/a <option value="DST">DST<\/option>   <!-- Martineau Hack -->/' /jffs/scripts/Advanced_OpenVPNClient_Content.asp
sed -r -i    's/alert\(\"<#996#>\"\);/alert\(\"<#197#> \" + \"<#102#> \" + \"<#418#>\"\);   <!-- Martineau Hack -->/' /jffs/scripts/Advanced_OpenVPNClient_Content.asp

2. Map the new custom VPN Client HTML page to the old page
Code:
mount -o bind /jffs/scripts/Advanced_OpenVPNClient_Content.asp /www/Advanced_OpenVPNClient_Content.asp

df

Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/root                34816     34816         0 100% /
devtmpfs                127744         0    127744   0% /dev
tmpfs                   127848      2224    125624   2% /tmp
/dev/mtdblock4           64256     17108     47148  27% /jffs
/dev/mtdblock4           64256     17108     47148  27% /usr/sbin/vpnrouting.sh
/dev/mtdblock4           64256     17108     47148  27% /www/Advanced_OpenVPNClient_Content.asp
/dev/mtdblock4           64256     17108     47148  27% /usr/sbin/updown.sh
3. Restart the web server to render the new custom GUI
Code:
service restart_httpd

Not 100% sure what issue you are describing....?o_O

i.e. rather than use prio 9990-9995, my idea was that exploiting unused RPDB rules 10000,10100,10300,10500,10700 and 10900 for fwmarks should be maintained by vpnrouting.sh but you must explicitly prevent vpnrouting.sh from 'accidentally' always'deleting prio 10000 whenever (and ONLY) VPN Client 1 is started/stopped. :eek:
Thank you @Martineau!

nat-start is working for mounting vpnrouting.sh and updown.sh now. I saw the error in my code when I first posted it into the post. I removed the question once I noticed the mistake. I was accidentally mounting /usr/sbin/vpnrouting to itself! No wonder it did not work! :eek:

The code below is working inside my main script called from nat-start:

Code:
if [ "$(df | grep -c "/usr/sbin/vpnrouting.sh")" -eq "0" ]; then
   mount -o bind /jffs/scripts/vpnrouting.sh /usr/sbin/vpnrouting.sh
fi

if [ "$(df | grep -c "/usr/sbin/updown.sh")" -eq "0" ]; then
   mount -o bind /jffs/scripts/updown.sh /usr/sbin/updown.sh
fi

I will change my fwmark/bitmask prio numbers to use the RPDB rule prio numbers
10000,10100,10300,10500,10700 and 10900 to see if it fixes my routing issues. Routing issues start occurring once a third OpenVPN clients comes into play.

Your use of placing IPSET lists inside the web gui has evolved this past week! No more underscores in the first column! I looked into this method while analyzing vpnrouting during my updates. I'll give the method a try after I update my RPDB rule prio numbers. I think I have a better idea of what you are accomplishing now.

I noticed I had an extra "dst" directive in my iptables command (e.g. dst,dst) that was not needed. Fortunately,iptables safely ignores any extra directives. I noticed in one of your posts that you are using src,dst as your directives in youriptables command for IPSET lists. Can you elaborate on this? What hash type are you using for your IPSET lists, i.e. hash:net,net?

If I create an ipset list with this hash type hash:net,net coupled with the ipset directive in dnsmasq.conf.add (e.g. ipset=/cnn.com/CNN), the ipset list CNN will not get populated. The IPSET list has to be defined as hash:ip or hash:net.

Regarding ipset, when looking at ipset man page, I read about using skbinfo and skbmark to store fwmark/bitmask info and wondered if the method can be exploited for our selective routing use case. https://lwn.net/Articles/612064/ gives an example:
Code:
ipset create mark_values hash:net skbinfo
ipset add mark_values 8.8.8.8/32 skbmark 0x1/0xffff
...
iptables -t mangle -A OUTPUT -p tcp -o iface \
   -j SET --map-set mark_values dst --map-mark
Another example on the slide deck on the netdevconf.org site.

I tried it out by creating an ipset list call "foo" and added an LAN client IP 192.168.22.152 followed by skbmark 0x1000/0x1000. That part worked. However, the --map-set and --map-mark appear to be deprecated. Perhaps this was the method that was used to mark packets in a prior version of ipset and has now been replaced with the method we currently use e.g.

Code:
iptables -t mangle -A PREROUTING -i br0 -m set --match-set MYSETLIST dst -j MARK --set-mark 0x1000/0x1000
 
nat-start is working for mounting vpnrouting.sh and updown.sh now. I saw the error in my code when I first posted it into the post. I removed the question once I noticed the mistake. I was accidentally mounting /usr/sbin/vpnrouting to itself! No wonder it did not work! :eek:
I (perhaps naively) assumed that there is no way that the mount/mapping can go AWOL? - hence I perform it once-only in init-start, but I recognise your diligence to ensure your crucial mods are still in place!:)
Your use of placing IPSET lists inside the web gui has evolved this past week! No more underscores in the first column!
For some reason I mistakenly had it in my head that what I wanted to do wasn't possible.... based on a previous firmware query: HTML / .asp modification but clearly (in this specific requirement) I can influence the desired script behaviour that is executed, and that D'oh! :rolleyes: I was already hacking GUI HTML e.g. '/www/require/modules/menuTree.js' etc. see RefreshWWW.sh HTML patch for @kvic NTPD Tools TAB

Now ideally each row to be added should preferably be implemented by a drop-menu for the desired field layout (since we write to the same NVRAM vpn_clientX_clientlist variable anyway)

'Desc/SourceIP/TargetIP/IFACE'
Code:
Description   xxx.xxx.xxx.xxx   xxx.xxx.xxx.xxx   VPN/WAN
'IPSET/Dimension/Dimension/Dimension'
Code:
IPSET_name   SRC/DST   SRC/DST   SRC/DST
but being too lazy to compile a custom firmware I'd rather hack a solution, so although IPSET fwmark routing can be achieved surreptitiously in the background, it is convenient (in the GUI) to at least quickly review (i.e. add/remove) what I've actually currently got selectively routed!:D

I noticed in one of your posts that you are using src,dst as your directives in your iptables command for IPSET lists.

Can you elaborate on this? What hash type are you using for your IPSET lists, i.e. hash:net,net?
There are far better web-hosted tutorials on using multi-dimension IPSETs but rather than describe 'src,dst' I'll describe -
e.g. a 3-dimension IPSET
Code:
ipset new test hash:ip,port,ip hashsize 64

ipset add test 10.0.0.1,tcp:8080,7.7.7.7

ipset list test

Name: test
Type: hash:ip,port,ip
Revision: 5
Header: family inet hashsize 64 maxelem 65536
Size in memory: 136
References: 0
Number of entries: 1
Members:
10.0.0.1,tcp:8080,7.7.7.7
So using iptables rule '-m matchset test src,dst,dst -j MARK --set-mark $FWMARK_OVPNC4', means LAN device 10.0.0.1 (src) traffic to 7.7.7.7 (dst) using TCP Port 8080 (dst) is via VPN Client4.

Suppose I add
Code:
ipset add test 10.0.0.2-10.0.0.10,udp:4444,4.4.4.4
Now using the same IPSET I could use another iptables rule '-m matchset test dst,dst,src - j ACCEPT' and any UDP Port 4444 (dst) traffic from 4.4.4.4 (src) to any LAN device in the range 10.0.0.2-10.0.0.10 (dst) is allowed!

NOTE: Clearly some combinations are not bidirectional
e.g. src,dst is invalid for the 2-dimensional IPSET
Code:
bitmap:ip,mac
as MACs can only ever match on 'src'.
If I create an ipset list with this hash type hash:net,net coupled with the ipset directive in dnsmasq.conf.add (e.g. ipset=/cnn.com/CNN), the ipset list CNN will not get populated.

The IPSET list has to be defined as hash:ip or hash:net.
I assume the dnsmasq 'ipset=' directive only allows a 1-dimensional but allows both a 'src' and 'dst' so either will match?, but iptables could capture the src,dst pair by writing direct to the IPSET
(That is what I do in PortScanBlocking.sh)
 
I (perhaps naively) assumed that there is no way that the mount/mapping can go AWOL? - hence I perform it once-only in init-start, but I recognise your diligence to ensure your crucial mods are still in place!:)
Now that you remind me, that was the first place I tested the code because it is right after the jffs parition is mounted! I defiantly was using the bad version when I tested it and didn't catch it at the time. I'll move the code to init-start. Thanks for catching that one.

For some reason I mistakenly had it in my head that what I wanted to do wasn't possible.... based on a previous firmware query: HTML / .asp modification but clearly (in this specific requirement) I can influence the desired script behaviour that is executed, and that D'oh! :rolleyes: I was already hacking GUI HTML e.g. '/www/require/modules/menuTree.js' etc. see RefreshWWW.sh HTML patch for @kvic NTPD Tools TAB

Now ideally each row to be added should preferably be implemented by a drop-menu for the desired field layout (since we write to the same NVRAM vpn_clientX_clientlist variable anyway)

'Desc/SourceIP/TargetIP/IFACE'
Code:
Description   xxx.xxx.xxx.xxx   xxx.xxx.xxx.xxx   VPN/WAN
'IPSET/Dimension/Dimension/Dimension'
Code:
IPSET_name   SRC/DST   SRC/DST   SRC/DST
but being too lazy to compile a custom firmware I'd rather hack a solution, so although IPSET fwmark routing can be achieved surreptitiously in the background, it is convenient (in the GUI) to at least quickly review (i.e. add/remove) what I've actually currently got selectively routed!:D


There are far better web-hosted tutorials on using multi-dimension IPSETs but rather than describe 'src,dst' I'll describe -
e.g. a 3-dimension IPSET
Code:
ipset new test hash:ip,port,ip hashsize 64

ipset add test 10.0.0.1,tcp:8080,7.7.7.7

ipset list test

Name: test
Type: hash:ip,port,ip
Revision: 5
Header: family inet hashsize 64 maxelem 65536
Size in memory: 136
References: 0
Number of entries: 1
Members:
10.0.0.1,tcp:8080,7.7.7.7
So using iptables rule '-m matchset test src,dst,dst -j MARK --set-mark $FWMARK_OVPNC4', means LAN device 10.0.0.1 (src) traffic to 7.7.7.7 (dst) using TCP Port 8080 (dst) is via VPN Client4.

Suppose I add
Code:
ipset add test 10.0.0.2-10.0.0.10,udp:4444,4.4.4.4
Now using the same IPSET I could use another iptables rule '-m matchset test dst,dst,src - j ACCEPT' and any UDP Port 4444 (dst) traffic from 4.4.4.4 (src) to any LAN device in the range 10.0.0.2-10.0.0.10 (dst) is allowed!

NOTE: Clearly some combinations are not bidirectional
e.g. src,dst is invalid for the 2-dimensional IPSET
Code:
bitmap:ip,mac
as MACs can only ever match on 'src'.

I assume the dnsmasq 'ipset=' directive only allows a 1-dimensional but allows both a 'src' and 'dst' so either will match?, but iptables could capture the src,dst pair by writing direct to the IPSET
(That is what I do in PortScanBlocking.sh)
Absolutely Brilliant! Very Creative use of the Web Gui, IPSET and iptables! I'll give the method a try tomorrow.

Thanks again for the very thorough explanation. I now understand how to implement. The method opens up some new possibilities!
 
Last edited:
Not 100% sure what issue you are describing....?o_O

i.e. rather than use prio 9990-9995, my idea was that exploiting unused RPDB rules 10000,10100,10300,10500,10700 and 10900 for fwmarks should be maintained by vpnrouting.sh but you must explicitly prevent vpnrouting.sh from 'accidentally' always'deleting prio 10000 whenever (and ONLY) VPN Client 1 is started/stopped. :eek:

I created ipset lists for websites I want to route selectively. The websites below will be assigned to an interface for testing purposes by assigning them to an IPSET list that is added to the master setlist for each interface. In other words, no matter what interface the client device is assigned to, the site should traverse through the tunnel using the fwmark/bitmask rule. When I order the OVPNCx fwmark/bitmask from prio 9991=OVPNC1...9995=OVPNC5, I get weird routing issues. Here are my results. My testing device is assigned to OVPNC4:
Code:
wan0  whatismyip.network    OK
tun11 pinfo.info        OK
tun12 detectmyip.net        OK
tun13 ipaddress.com        Error - goes to tun11 iface
tun14 whatismypublicip.com    OK
tun15 whatismyipaddress.com    Error -goes to tun11 iface

This prio order fixes the routing issues:
Code:
9990:   from all fwmark 0x7000/0x7000 lookup main
9991:   from all fwmark 0x5000/0x5000 lookup ovpnc5
9992:   from all fwmark 0x4000/0x4000 lookup ovpnc4
9993:   from all fwmark 0x3000/0x3000 lookup ovpnc3
9994:   from all fwmark 0x2000/0x2000 lookup ovpnc2
9995:   from all fwmark 0x1000/0x1000 lookup ovpnc1
I also tested using the WAN=10000, OVPNC1=10100, OVPNC2=10300, OVPNC3=10500, etc prio numbers and experienced the routing issues when they were placed in sequential order.

If I route the individual websites to the tunnel using
Code:
ip rule add to $IP table 112 prior $PRIO
and give the prio a lower number than the fwmark prio numebers, routing for the websites work.

I'll test to see if the ipset and iptables methods in your prior post will resolve the issue. Thanks again for the tips. I look forward to implementing.
 
This prio order fixes the routing issues:
Code:
9990:   from all fwmark 0x7000/0x7000 lookup main
9991:   from all fwmark 0x5000/0x5000 lookup ovpnc5
9992:   from all fwmark 0x4000/0x4000 lookup ovpnc4
9993:   from all fwmark 0x3000/0x3000 lookup ovpnc3
9994:   from all fwmark 0x2000/0x2000 lookup ovpnc2
9995:   from all fwmark 0x1000/0x1000 lookup ovpnc1
Who needs to endure the possible trauma associated with running three (or more) concurrent VPN Clients? - sheer madness! :p:p
Anyway I'm sure you must have realised by now exactly what is causing your frustration?:oops:

We have 7 possible interfaces 2 WAN, and either 2 or 5 VPN Clients depending on the firmware
Code:
WAN0
WAN1        Dual-WAN
VPN1
VPN2
and 3 additional VPN clients on RMerlin firmware
Code:
VPN3
VPN4
VPN5

For the one-byte tagging we only have 4 unique fwmarks,

Code:
0x1000
0x2000
0x4000
0x8000
and two overlapping 'fluke's
Code:
0x7000
0xF000

For easier coding it was my (lazy) decision to match a VPN Client with its numerical fwmark, and the use of the fwmark mask has subtly altered the execution implementation when using 3 (or more) VPN clients.....does Binary representation ring any bells?:oops:

As I alluded to previously, there will come a time when the fwmarks used for VPN tagging will need to be reviewed...perhaps Today is the day?!:eek:

e.g. assuming that for performance you should try and load CPU:1 with designated VPN Client traffic as CPU:0 is usually already (over-busy?) hosting the bulk of the router's main tasks.
Code:
CPU:1 hosts VPN Clients 1,3 and 5
CPU:0 hosts VPN Clients 2 and 4

0x01000     VPN1
0x04000     VPN3
0x0F000     VPN5
0x02000     VPN2
0x07000     VPN4
0x08000     WAN0
0x?????     WAN1        Dual-WAN secondary loses out unless we can steal an used VPN Client's tag?

So as it is no longer prudent to use prio 10000,10100,10300,10500,10700 and 10900 I now recommend reverting to my original use of the prio 9990-9995 range with the revised fwmark assignment and proposed formal static descending RPDB order as follows:

Code:
9990   from   all   fwmark   0x8000/0x8000   table main          (or table 100 for Dual-WAN primary)
9991   from   all   fwmark   0xF000/0xF000   table ovpnc5
9992   from   all   fwmark   0x7000/0x7000   table ovpnc4
and whilst the next three rules may be in any order.....
Code:
9993   from   all   fwmark   0x4000/0x4000   table ovpnc3
9994   from   all   fwmark   0x2000/0x2000   table ovpnc2
9995   from   all   fwmark   0x1000/0x1000   table ovpnc1
 
Who needs to endure the possible trauma associated with running three (or more) concurrent VPN Clients? - sheer madness! :p:p
Anyway I'm sure you must have realised by now exactly what is causing your frustration?:oops:

We have 7 possible interfaces 2 WAN, and either 2 or 5 VPN Clients depending on the firmware
Code:
WAN0
WAN1        Dual-WAN
VPN1
VPN2
and 3 additional VPN clients on RMerlin firmware
Code:
VPN3
VPN4
VPN5

For the one-byte tagging we only have 4 unique fwmarks,

Code:
0x1000
0x2000
0x4000
0x8000
and two overlapping 'fluke's
Code:
0x7000
0xF000

For easier coding it was my (lazy) decision to match a VPN Client with its numerical fwmark, and the use of the fwmark mask has subtly altered the execution implementation when using 3 (or more) VPN clients.....does Binary representation ring any bells?:oops:

As I alluded to previously, there will come a time when the fwmarks used for VPN tagging will need to be reviewed...perhaps Today is the day?!:eek:

e.g. assuming that for performance you should try and load CPU:1 with designated VPN Client traffic as CPU:0 is usually already (over-busy?) hosting the bulk of the router's main tasks.
Code:
CPU:1 hosts VPN Clients 1,3 and 5
CPU:0 hosts VPN Clients 2 and 4

0x01000     VPN1
0x04000     VPN3
0x0F000     VPN5
0x02000     VPN2
0x07000     VPN4
0x08000     WAN0
0x?????     WAN1        Dual-WAN secondary loses out unless we can steal an used VPN Client's tag?

So as it is no longer prudent to use prio 10000,10100,10300,10500,10700 and 10900 I now recommend reverting to my original use of the prio 9990-9995 range with the revised fwmark assignment and proposed formal static descending RPDB order as follows:

Code:
9990   from   all   fwmark   0x8000/0x8000   table main          (or table 100 for Dual-WAN primary)
9991   from   all   fwmark   0xF000/0xF000   table ovpnc5
9992   from   all   fwmark   0x7000/0x7000   table ovpnc4
and whilst the next three rules may be in any order.....
Code:
9993   from   all   fwmark   0x4000/0x4000   table ovpnc3
9994   from   all   fwmark   0x2000/0x2000   table ovpnc2
9995   from   all   fwmark   0x1000/0x1000   table ovpnc1
Thank you for the explanation. I spent the greater part of yesterday testing and researching why this was happening. Last night and this morning, I spent time reviewing the Policy Routing Handbook in attempt to shed some light on the situation. I was coming to the conclusion that it may have something to do with the priority of the tables in /etc/iproute2/rt_tables in addition to an issue with the order of the fwmark/bitmask combo. I never would have gotten to the bottom of the issue without your help and detailed explanation. I'm very GRATEFUL! I'll implement the changes and give you an update.

The primary reason I prefer the 9990 to 9995 prio for the fwmark/bitmask interfaces is that it assigns a prio with a higher priority (which is a lower prio number) than the LAN clients. I'm using the default firmware prio numbers for LAN clients assigned to an OpenVPN client. With this setup, no matter what interface a LAN client is assigned to, traffic marked with the fwmark/bitmask combo will get a higher priority. For example, my laptop is assigned to OVPNC3. But Netflix needs to go thru OVPNC2. So, I need to have the rule for Netflix listed as a higher priority than the LAN client. But I have yet to fully implement your iptables/ipset matching rule techniques.

Chain PREROUTING (policy ACCEPT 6244K packets, 5160M bytes)
num pkts bytes target prot opt in out source destination
<snip>
2 0 0 MARK tcp -- br0 * 0.0.0.0/0 0.0.0.0/0 match-set WAN0 src,dst,dst MARK or 0x7000
3 0 0 MARK tcp -- br0 * 0.0.0.0/0 0.0.0.0/0 match-set VPN1 src,dst,dst MARK or 0x1000
4 0 0 MARK tcp -- br0 * 0.0.0.0/0 0.0.0.0/0 match-set VPN2 src,dst,dst MARK or 0x2000
5 0 0 MARK tcp -- br0 * 0.0.0.0/0 0.0.0.0/0 match-set VPN3 src,dst,dst MARK or 0x3000
6 0 0 MARK tcp -- br0 * 0.0.0.0/0 0.0.0.0/0 match-set VPN4 src,dst,dst MARK or 0x4000
7 0 0 MARK tcp -- br0 * 0.0.0.0/0 0.0.0.0/0 match-set VPN5 src,dst,dst MARK or 0x5000
8 0 0 MARK tcp -- br0 * 0.0.0.0/0 0.0.0.0/0 match-set WAN0D dst,dst MARK or 0x7000
9 0 0 MARK tcp -- br0 * 0.0.0.0/0 0.0.0.0/0 match-set VPN1D dst,dst MARK or 0x1000
10 0 0 MARK tcp -- br0 * 0.0.0.0/0 0.0.0.0/0 match-set VPN2D dst,dst MARK or 0x2000
11 0 0 MARK tcp -- br0 * 0.0.0.0/0 0.0.0.0/0 match-set VPN3D dst,dst MARK or 0x3000
12 0 0 MARK tcp -- br0 * 0.0.0.0/0 0.0.0.0/0 match-set VPN4D dst,dst MARK or 0x4000
13 0 0 MARK tcp -- br0 * 0.0.0.0/0 0.0.0.0/0 match-set VPN5D dst,dst MARK or 0x5000
14 0 0 MARK tcp -- br0 * 0.0.0.0/0 0.0.0.0/0 multiport sports 5000:5001 MARK or 0x7000
15 0 0 MARK tcp -- br0 * 10.88.8.111 0.0.0.0/0 multiport dports 80,443 MARK or 0x1000
 
I just finished testing. The routing works except thru OVPNC5. The website whatismyipaddress.com is configured to use OVPNC5. It reports the IP address of the WAN iface.

In the code, I use a capital "F" in the fwmark/bitmask combo e.g. 0xF000/0xF000. iptables and ip rule outbout with lower case "f":
Code:
9991   from  all   fwmark   0xf000/0xf000   table ovpnc5
I get no error messages and the lower case "f" appears to be commonly used in fwmark/bitmask when I did a search on the web.

I also tried 0xFF00/0xFF00 with no effect. Routing started working again thru OVPNC5 when I changed the fwmark/bitmask combo to 0x3000/0x3000:

Code:
9990:   from all fwmark 0x8000/0x8000 lookup main
9991:   from all fwmark 0x3000/0x3000 lookup ovpnc5
9992:   from all fwmark 0x7000/0x7000 lookup ovpnc4
9993:   from all fwmark 0x4000/0x4000 lookup ovpnc3
9994:   from all fwmark 0x2000/0x2000 lookup ovpnc2
9995:   from all fwmark 0x1000/0x1000 lookup ovpnc1

upload_2018-8-28_19-18-17.png
 
Very Creative use of the Web Gui, IPSET and iptables!!

My html/.asp coding skills are basic to say the least , but here is a preview of my latest GUI v1.20 IPSET VPN Selective routing GUI

EDIT: Sometimes the 'Video not found' error is displayed: o_O
Simply type 'IPSET_GUI' into the search box on the error page and the video should be shown
:rolleyes:

A minor issue is that the pull-down to populate the 'IPSET Source IP' always populates the original top line for 'Description/Source IP'....probably due to some new-fangled .css method or something basic I have missed:oops:
I have yet to display/create a separate IPSET table, but a single consolidated table should suffice for the time being. ;)

So in vpnrouting.sh, simply check the 4th field of the parsed NVRAM variable and if it isn't 'VPN' or 'WAN' assume it is going to be one of the following 'DST/SRC/DDD/SSS/SD/DS' etc. for a 3-dimensional or 2-dimensional IPSET.

Clearly it is no longer practical to provide 'sed' commands to effect the patches, but if you wish to test the new hacks, please let me know.
 
Last edited:
My html/.asp coding skills are basic to say the least , but here is a preview of my latest GUI hacks v1.20 IPSET Selective Routing GUI.

A minor issue is that the pull-down to populate the 'IPSET Source IP' always populates the original top line for 'Description/Source IP'....probably due to some new-fangled .css method or something basic I have missed:oops:
I have yet to display/create a separate IPSET table, but a single consolidated table should suffice for the time being. ;)

So in vpnrouting.sh, simply check the 4th field of the parsed NVRAM variable and if it isn't 'VPN' or 'WAN' assume it is going to be one of the following 'DST/SRC/DDD/SSS/SD/DS' etc. for a 3-dimensional or 2-dimensional IPSET.

Clearly it is no longer practical to provide 'sed' commands to effect the patches, but if you wish to test the new hacks, please let me know.
Yes, I'm very interested in testing. Thanks for including me. I have time to work on it tomorrow and the remainder the week.

Unfortunately, the Vimeo link didn't work.
 
Yes, I'm very interested in testing. Thanks for including me. I have time to work on it tomorrow and the remainder the week.

Unfortunately, the Vimeo link didn't work.

Please retry the video link:oops:

IPSET_GUI.png


I will PM you 'Advanced_OpenVPNClient_Content.asp'
 
Last edited:
@Martineau

Thank you again for letting me test with the enhanced VPN client gui! Kewl. I like it A LOT! I'm learning more about CSS myself since I launched my blog site. I have a ways to go though. For now, I have gotten by with what WordPress generates. But I need to go deeper!

For preliminary testing, I put code in for DST for testing and commented out the rest in vpnrouting.sh.

Code:
######################## Xentrk updates for Martineau's awesome Web GUI IPSET features
# Proccess IPSET lists
# 'DST/SRC/DDD/SSS' 3-dimensional IPSET list
# 'SD/DS/DD/' 2-dimensional IPSET
#
     if [ "$TARGET_ROUTE" = "DST" ]; then
        ipset add OVPNC${VPN_UNIT} "$DESC"
        continue
#    elif [ "$TARGET_ROUTE" = "SRC" ]; then
#        ipset add OVPNC${VPN_UNIT} "$DESC"
#        continue
#    elif [ "$TARGET_ROUTE" = "DDD" ]; then 
#        ipset add OVPNC${VPN_UNIT} "$DESC"
#        continue
#    elif [ "$TARGET_ROUTE" = "SSS" ]; then
#        ipset add OVPNC${VPN_UNIT} "$DESC"
#        continue
#    elif [ "$TARGET_ROUTE" = "SD" ]; then
#        ipset add OVPNC${VPN_UNIT} "$DESC"
#        continue
<snip>
    fi
######################################
The lists are deleted properly from the master IPSET list for the VPN Client interface when deleted in the web gui or the client is turned off. Everything went with out a hitch.

I assume the dnsmasq 'ipset=' directive only allows a 1-dimensional but allows both a 'src' and 'dst' so either will match?, but iptables could capture the src,dst pair by writing direct to the IPSET
(That is what I do in PortScanBlocking.sh)

Can you please elaborate on this one some more and how you are generating the src for your destination IPSET lists? I want to follow your method as much as possible with minimal deviation. I'm still confused with the iptables matching on src,dst,dst for the VPNx master or list:set IPSET lists. I understand that if an iptables command matches a two-dimension IPSET list using the src,dst,dst, the 3rd dst is ignored.

All of my IPSET lists are for destinations addresses. I use a combination of IPSET lists I generated from the ipset=/somedomain.com/SETLIST feature in dnsmasq and from pulling IPv4 using AS numbers from ipinfo.io. I use one-dimension IPSET lists and the dst match on the iptables command, e.g.
Code:
iptables -t mangle -A PREROUTING -i br0 -m set --match-set OVPNC1 dst -j MARK --set-mark 0x2000/0x2000

In your VPN1 master set list, you are matching on src,dst,dst. My confusion is how you are creating the src content for your streaming media IPSET lists like Netflix, Hulu or CBS_Shows, especially when using ipset=/ directive in dnsmasq.conf.add. You mentioned the ability to dynamically generate src. I am familiar with that feature for blocking purposes using the --add-set feature.

Code:
ipset create blacklist hash:ip

-A INPUT -p tcp -m multiport --dports 23,1433,2323,3306,3389,5060 -j SET --add-set blacklist src
-A INPUT -p udp -m multiport --dports 137,138,139,161,5060 -j SET --add-set blacklist src
-A INPUT -m set --match-set blacklist src -j LOG --log-prefix "IP Blacklisted INPUT "
-A INPUT -m set --match-set blacklist src -j DROP

I did a test and dynamically generated the src using the ipset/iptables feature to capture packet information and store it in another ipset list.
Code:
ipset create LAN_CLIENTS hash:net,net
iptables -t mangle -A PREROUTING -i br0 -m -set
iptables -t mangle -A PREROUTING -i br0 -s 192.168.22.0/24 -j SET --add-set LAN_CLIENTS src,dst
iptables -t mangle -A PREROUTING -i br0 -m set --match-set LAN_CLIENTS src,dst -j MARK --set-mark 0x3000/0x3000

Another idea I had is to output the current IPSET list contents and store them in a text file. Then, parse thru the text file and prepend my LAN IP in CIDR format to a new IPSET list:
Code:
ipset -L x3mRouting_NETFLIX | grep -E "([0-9]{1,3}[\\.]){3}[0-9]{1,3}" > /jffs/scripts/NETFLIX_NETADDR
ipset -L x3mRouting_AMAZONAWS_US | grep -E "([0-9]{1,3}[\\.]){3}[0-9]{1,3}" > jffs/scripts/AMAZON_NETADDR

ipset create NETFLIX hash:net,het
ipset create AMAZONAWS_US hash:net,net

awk '{print "add NETFLIX 192.168.22.0/24,"$1}' "/jffs/scripts/NETFLIX_NETADDR" | ipset restore -!
awk '{print "add AMAZONAWS_US 192.168.22.0/24,"$1}' "/jffs/scripts/AMAZON_NETADDR" | ipset restore -!

One more question! Are you placing your LAN clients in an IPSET list (e.g VPN1IPs) rather than having vpnrouting.sh do the routing using the ip rule add xxx.xxx.xxx.xxx prio xxxxx command?
 

Latest threads

Support SNBForums w/ Amazon

If you'd like to support SNBForums, just use this link and buy anything on Amazon. Thanks!

Sign Up For SNBForums Daily Digest

Get an update of what's new every day delivered to your mailbox. Sign up here!
Top