x3mRouting x3mRouting ~ Selective Routing for Asuswrt-Merlin Firmware

[Xentrk]

x3mRouting Version 2.0.0 now available! June 30, 2020

Version 2.0.0 Update Process - update instructions
Version 2.0.0 Changes - see what's changed!

There are numerous enhancements and changes to x3mRouting. Please refer to the Version 2.0.0 Changes section for a description of all changes. Afterwards, read the updated instructions on the README to become familiar with the new features and usage instructions.

If coming from the first generation of x3mRouting, please read the updated instructions to become familiar with the new features and usage instructions. Refer to the Version 2.0.0 Update Process section for the update instructions.

x3mRouting 2.0.0 Overview

x3mMenu

The command to access the x3mRouting menu has been changed from x3mRouting to x3mMenu.

x3mRouting

The separate scripts for:

• IPSET list creation and routing using the ASN, Amazon AWS, dnsmasq and manual methods
• VPN Server to VPN Client routing (route_all_vpnserver.sh)
• VPN Server to IPSET routing (route_ipset_vpnserver.sh)

have been removed and the features combined into one script called x3mRouting.sh. x3mRouting has been configured as a command with a symbolic link to /jffs/scripts/x3mRouting/x3mRouting.sh. This allows /jffs/scripts/x3mRouting/x3mRouting.sh to be run from any location without specifying the path or "sh" command.

Running x3mRouting will automatically perform the set-up. A help option has also been added. Type x3mRouting help at the command line to list usage notes or refer to post 2.

Local Repository

The local repository on the router remains as /jffs/scripts/x3mRouting. The repository contains the x3mRouting user and openvpn-event scripts.

Utility File Repository

Utility files used by the x3mRouting features are located in /jffs/scripts/addons/x3mRouting. These include the nvram files for those who use the LAN Client Routing option.

Advanced_OpenVPNClient_Content.asp

The modified screen now supports VPN Bypass Routing for IPSET lists and is compatible with 384.18 firmware modifications.

autoscan.sh and getdomainnames.sh Scripts

Two scripts have been added to assist in determining the domain names used by a streaming service or website. autoscan.sh will search dnsmasq.log file for all top level domain names passed by the 'autoscan=' parm (e.g. sh autoscan.sh autoscan=netflix,nflx). getdomainnames.sh will return all domain names collected used by a streaming service or website for a particular device. All manual steps in the prior version have been automated.

Please refer to the Version 2.0.0 Changes section for a description of all changes.

x3mRouting 2.0.0 Update Options

You will not be able to update to Version 2.0.0 using amtm or the existing x3mRouting Menu due to the scope of the installation menu changes. However, you should update amtm before updating x3mRouting to get the changes made for the new version.

The recommended option is to utilize the update feature of the x3mMenu to convert old entries to the new usage syntax and perform all necessary clean up from the previous version. Alternatively, you can choose to remove the current version, which requires removing any old references in nat-start, and manually configuring the new version of x3mRouting.

Install the New Menu and Select the [ u ] Update x3mRouting to Version 2.0.0 option (Recommended Method)

1. Install the x3mMenu

sh -c "$(curl -sL https://raw.githubusercontent.com/Xentrk/x3mRouting/master/Install_x3mRouting.sh)"

2. Select the [ u ] Update x3mRouting to Version 2.0.0 option

3. nat-start, vpnclientX-route-up, and vpnclientX-route-pre-down files will be scanned for any references to the old version of x3mRouting. A conversion file will be created and stored in /jffs/scripts/x3mRouting/x3mRouting_Conversion.sh

4. View the x3mRouting_Conversion.sh file and confirm entries.

5. Run the x3mRouting_Conversion.sh script to create the routing rules and set-up.

Please read the Version 2.0.0 Update Process for the complete update instructions.

Remove the Current Version and Install the New Version

1. Remove the current installation of x3mRouting using the existing menu. To access the menu, type x3mRouting at the command prompt and select the option to remove the repository.

2. Edit /jffs/scripts/nat-start files to remove any references to the old scripts

3. Install the x3mMenu

sh -c "$(curl -sL https://raw.githubusercontent.com/Xentrk/x3mRouting/master/Install_x3mRouting.sh)"

4. Run the x3mRouting script to create the routing rules for IPSET lists, VPN Server to VPN Client, and VPN Server to IPSET lists.

amtm

If you didn't update amtm before updating to x3mRouting Version 2.0.0, do so now.

Grateful

Thank you to everyone who contributed to the success of the project. Please see the
Acknowledgements section on the README for a complete list of contributors.

 

[maghuro]

Hell yeah!

Good job! Everything's smooth as a baby a**

 

[thelonelycoder]

Congrats @Xentrk for this major overhaul and improvement.

@all Please confirm that the amtm u update check sucessfully checks with the new version. Thanks.
New installs in amtm automatically install this new version.

 

[andresmorago]

Great! Will test this afternoon!

 

[Luizlp10]

It is working great for me and it felt a lot more simple to set and use. Thank you Xentrk for your hard work!

 

[Torson]

Congrats @Xentrk for this major overhaul and improvement.

@all Please confirm that the u update check sucessfully checks with the new version. Thanks.
New installs in amtm automatically install this new version.

I can confirm that the amtm duplicate item menu is gone and all looks and works great - thank you for that and to @Xentrk for the solid update.

 

[andresmorago]

i just went through the update process. i used to have some ipset entries on dnsmasq.conf.add which are now gone (they were backup). after running the conversion script, they werent recreated on the dnsmasq.conf.add
can i just put them back again manually?

this is my conversion script

#!/bin/sh
# Source File====> /jffs/scripts/x3mRouting/vpnclient1-route-up
# Original Entry=> sh /jffs/scripts/x3mRouting/load_MANUAL_ipset_iface.sh 1 amazon_vpn dir=/mnt/sda1/vpn_routes
sh /jffs/scripts/x3mRouting/x3mRouting.sh ALL 1 amazon_vpn

and nat-start

#!/bin/sh

/jffs/scripts/ntpmerlin ntpredirect # ntpMerlin
sh /jffs/scripts/x3mRouting/x3mRouting.sh ALL 1 amazon_vpn

i deleted the amazon_vpn just for the heck of it

x3mRouting ALL 1 amazon_vpn del

and created a new one

x3mRouting 1 0 amazon-route dnsmasq=ipinfo.io

but, when running this ipset method, do i have to run this line everytime i want to add a new website?

x3mRouting 1 0 amazon-route dnsmasq=XXXXXX.com

ipinfo.io isnt being routed through the vpn client

andresmorago@RT-AC3100-0548:/tmp/home/root# ip rule
0:      from all lookup local
9990:   from all fwmark 0x8000/0x8000 lookup main
9995:   from all fwmark 0x1000/0x1000 lookup ovpnc1
10001:  from 10.0.0.1 lookup main
32766:  from all lookup main
32767:  from all lookup default

andresmorago@RT-AC3100-0548:/tmp/home/root# iptables -nvL PREROUTING -t mangle --line
Chain PREROUTING (policy ACCEPT 23224 packets, 6413K bytes)
num   pkts bytes target     prot opt in     out     source               destination
1       18  1656 MARK       all  --  br0    *       0.0.0.0/0            0.0.0.0/0            match-set amazon-route dst MARK or 0x8000
andresmorago@RT-AC3100-0548:/tmp/home/root#

EDIT
i ended up uninstalling everything and starting from scratch.
ran

x3mRouting ALL 1 aws1

re-added all the ipset entried to dnsmasq.conf.add

##IPSET
##
ipset=/ifconfig.io/aws1
ipset=/pandora.com/aws1

 

[Xentrk]

i just went through the update process. i used to have some ipset entries on dnsmasq.conf.add which are now gone (they were backup). after running the conversion script, they werent recreated on the dnsmasq.conf.add
can i just put them back again manually?

this is my conversion script

#!/bin/sh
# Source File====> /jffs/scripts/x3mRouting/vpnclient1-route-up
# Original Entry=> sh /jffs/scripts/x3mRouting/load_MANUAL_ipset_iface.sh 1 amazon_vpn dir=/mnt/sda1/vpn_routes
sh /jffs/scripts/x3mRouting/x3mRouting.sh ALL 1 amazon_vpn

and nat-start

#!/bin/sh

/jffs/scripts/ntpmerlin ntpredirect # ntpMerlin
sh /jffs/scripts/x3mRouting/x3mRouting.sh ALL 1 amazon_vpn

i deleted the amazon_vpn just for the heck of it

x3mRouting ALL 1 amazon_vpn del

and created a new one

x3mRouting 1 0 amazon-route dnsmasq=ipinfo.io

but, when running this ipset method, do i have to run this line everytime i want to add a new website?

x3mRouting 1 0 amazon-route dnsmasq=XXXXXX.com

ipinfo.io isnt being routed through the vpn client

andresmorago@RT-AC3100-0548:/tmp/home/root# ip rule
0:      from all lookup local
9990:   from all fwmark 0x8000/0x8000 lookup main
9995:   from all fwmark 0x1000/0x1000 lookup ovpnc1
10001:  from 10.0.0.1 lookup main
32766:  from all lookup main
32767:  from all lookup default

andresmorago@RT-AC3100-0548:/tmp/home/root# iptables -nvL PREROUTING -t mangle --line
Chain PREROUTING (policy ACCEPT 23224 packets, 6413K bytes)
num   pkts bytes target     prot opt in     out     source               destination
1       18  1656 MARK       all  --  br0    *       0.0.0.0/0            0.0.0.0/0            match-set amazon-route dst MARK or 0x8000
andresmorago@RT-AC3100-0548:/tmp/home/root#

EDIT
i ended up uninstalling everything and starting from scratch.
ran

x3mRouting ALL 1 aws1

re-added all the ipset entried to dnsmasq.conf.add

##IPSET
##
ipset=/ifconfig.io/aws1
ipset=/pandora.com/aws1

The conversion script is looking for the "load_DNSMASQ_ipset_iface.sh" or "load_DNSMASQ_iface.sh" to convert from the old to the new. See the example below:

# Source File====> /jffs/scripts/nat-start
# Original Entry=> sh /jffs/scripts/x3mRouting/load_DNSMASQ_ipset_iface.sh 3 BBC_WEB bbc.co.uk,bbc.com,bbc.gscontxt.net,bbci.co.uk,bbctvapps.co.uk,ssl-bbcsmarttv.2cnt.net
sh /jffs/scripts/x3mRouting/x3mRouting.sh ALL 3 BBC_WEB dnsmasq=bbc.co.uk,bbc.com,bbc.gscontxt.net,bbci.co.uk,bbctvapps.co.uk,ssl-bbcsmarttv.2cnt.net

IPSET entries will get removed from dnsmasq.conf.add as part of the clean-up. They do get recreated when the x3mRouting script is run if the 'dnsmasq=' parameter is specified A backup of dnsmasq.conf.add gets created so you can restore from that file.

You are specifying the manual method when running the script, then manually entering the IPSET information. This will be an issue for the restore job and on system boot as the dsnmasq method stores the IPSET entries differently than the manual method. Please perform these steps to clean up the current config and recreate using the correct usage syntax:

x3mRouting ALL 1 aws1 del

Remove the following entries in dnsmasq.conf.add:

ipset=/ifconfig.io/aws1
ipset=/pandora.com/aws1

Restart dnsmasq: service restart_dnsmasq

Create the aws1 IPSET list using the dnsmasq method.

x3mRouting x3mRouting ALL 1 aws1 dnsmasq=ifconfig.io,pandora.com

 

[Xentrk]

@andresmorago I should have also mentioned that you should remove the save/restore aw1 file in /opt/tmp or the location that you specified as well due to the differences in the way the data is saved between the manual and dnsmasq method.

EDIT: With the dnsmasq method, the save/restore file will get created at 2:00 AM from a cron job. With dnsmasq method, dnsmasq will load the IPSET list when the domain is queried. You can view the IPSET entries collected by the dnsmasq method using the command ipset -L aws1. These entries are what gets saved by the cron job.

 

[andresmorago]

@Xentrk thanks. im getting some progress. i have some questions

*for each domain i want to add, do i need to run x3mRouting ALL 1 aws1 dnsmasq=XXXXX.com. cant i just run once
x3mRouting ALL 1 aws1 and then add the ipset lines to dnsmasq.conf.add?

*if i run the above command for each domain that i think of, i will see an individual entry at nat-start. wouldnt that be unnecessary processing for the router?

#!/bin/sh
/jffs/scripts/ntpmerlin ntpredirect # ntpMerlin
sh /jffs/scripts/x3mRouting/x3mRouting.sh ALL 1 aws1
sh /jffs/scripts/x3mRouting/x3mRouting.sh ALL 1 aws1 dnsmasq=espn.com

*is there a way to have a file with just plain domains per line, instead of having them at dnsmasq.conf.add?

thanks

 

[Xentrk]

@Xentrk thanks. im getting some progress. i have some questions

*for each domain i want to add, do i need to run x3mRouting ALL 1 aws1 dnsmasq=XXXXX.com. cant i just run once
x3mRouting ALL 1 aws1 and then add the ipset lines to dnsmasq.conf.add?

*if i run the above command for each domain that i think of, i will see an individual entry at nat-start. wouldnt that be unnecessary processing for the router?

#!/bin/sh
/jffs/scripts/ntpmerlin ntpredirect # ntpMerlin
sh /jffs/scripts/x3mRouting/x3mRouting.sh ALL 1 aws1
sh /jffs/scripts/x3mRouting/x3mRouting.sh ALL 1 aws1 dnsmasq=espn.com

*is there a way to have a file with just plain domains per line, instead of having them at dnsmasq.conf.add?

thanks

sh /jffs/scripts/x3mRouting/x3mRouting.sh ALL 1 aws1
sh /jffs/scripts/x3mRouting/x3mRouting.sh ALL 1 aws1 dnsmasq=espn.com

The first line listed above creates the IPSET list aws1 using the manual method. The manual method assumes you have pre-populated /opt/tmp/aws1 with IPv4 addresses. The second line then changes the IPSET list to be sourced from the dnsmasq method. After a boot, x3mRouting will see the first line in nat-start and attempt to restore aws1 using the manual method. An error will occur as the format of the save/restore file is no longer in the manual method format. It is now in dnsmasq format as a result of the 2nd line since.

The cleanest way to add a domain to the IPSET list is to first delete the current entry using the 'del' parameter and rerun the script with the new domain added to the 'dnsmasq=' parameter. You can enter the short cut to remove the current entry:

x3mRouting ipset_name=aws1 del

This is to avoid multiple IPSET entries in dnsmasq.conf.add and nat-start.

However, you can add a new top level domain without deleting the old entry. But your dnsmasq.conf.add, and nat-start will start getting messy. Here is an example

 x3mRouting ALL 2 NEWSPAPERS dnsmasq=nytimes.com

The above entry will get added to nat-start

dnsmasq.conf.add will contain the line:

ipset=/nytimes.com/NEWSPAPERS

You can rerun x3mRouting and add a domain using the command

 x3mRouting ALL 2 NEWSPAPERS dnsmasq=freep.com

dnsmasq.conf.add will now have two lines:

ipset=/nytimes.com/NEWSPAPERS
ipset=/freep.com/NEWSPAPERS

But this will also create two entries in nat-start:

sh /jffs/scripts/x3mRouting/x3mRouting.sh ALL 2 NEWSPAPERS dnsmasq=nytimes.com
sh /jffs/scripts/x3mRouting/x3mRouting.sh ALL 2 NEWSPAPERS dnsmasq=freep.com

I recommend this approach to adding a new domain. First, copy the entry in nat-start and paste it on the command line and add the 'del' parm. Press enter to remove the routing rules and entries in the supporting files. Then, use the up arrow on the command line to display the last command run. Use the backspace key to remove the 'del' parm and add the new domain name to the end of the list separated by a comma. Press enter and the new domain is now added to the IPSET list.

 

[Xentrk]

Here is how the save/restore file appears for dnsmasq method:

create CBS_WEB hash:net family inet hashsize 1024 maxelem 65536
add CBS_WEB 64.30.230.22
add CBS_WEB 72.246.189.226
add CBS_WEB 110.164.11.73
add CBS_WEB 23.52.171.136
<snip>

For all of the other methods (manual, ASN, Amazon), the save/restore file only contains a list of IPv4 addresses or CIDR values:

132.185.0.0/16
132.185.112.0/20
132.185.128.0/20
212.58.224.0/19
132.185.224.0/20

 

[h0me5k1n]

I updated but no conversion script was created ... I only had 6 x3mrouting entries anyway so I ended up removing x3mrouting and starting from scratch using the backed up nat-start file as a reference.

Previously I had a couple of rules that basically said "don't route Netflix or Amazon traffic through a VPN" (so that this traffic always bypasses VPN connections so I don't get any "region blocked" issues) but I get an error...

$ x3mRouting ALL 0 AMAZON aws_region=US,EU,GLOBAL
(x3mRouting): 2001 Starting Script Execution ALL 0 AMAZON aws_region=US,EU,GLOBAL
(x3mRouting): 2001 ERROR: Invalid Source 'ALL' and Destination (0) combination.

$x3mRouting ALL 0 NETFLIX dnsmasq=netflix.com,nflxext.com,nflximg.net,nflxso.net,nflxvideo.net
(x3mRouting): 5031 Starting Script Execution ALL 0 NETFLIX dnsmasq=netflix.com,nflxext.com,nflximg.net,nflxso.net,nflxvideo.net
(x3mRouting): 5031 ERROR: Invalid Source 'ALL' and Destination (0) combination.

This doesn't matter too much to me at the moment but ,if in future I want to selectively "exclude" traffic for a particular device when all its traffic is normally routed via the VPN, how would I ensure that this traffic always goes out of the WAN connection (0)? Would I have to do this per device or per VPN connection?

 

[Xentrk]

I updated but no conversion script was created ... I only had 6 x3mrouting entries anyway so I ended up removing x3mrouting and starting from scratch using the backed up nat-start file as a reference.

Previously I had a couple of rules that basically said "don't route Netflix or Amazon traffic through a VPN" (so that this traffic always bypasses VPN connections so I don't get any "region blocked" issues) but I get an error...

$ x3mRouting ALL 0 AMAZON aws_region=US,EU,GLOBAL
(x3mRouting): 2001 Starting Script Execution ALL 0 AMAZON aws_region=US,EU,GLOBAL
(x3mRouting): 2001 ERROR: Invalid Source 'ALL' and Destination (0) combination.

$x3mRouting ALL 0 NETFLIX dnsmasq=netflix.com,nflxext.com,nflximg.net,nflxso.net,nflxvideo.net
(x3mRouting): 5031 Starting Script Execution ALL 0 NETFLIX dnsmasq=netflix.com,nflxext.com,nflximg.net,nflxso.net,nflxvideo.net
(x3mRouting): 5031 ERROR: Invalid Source 'ALL' and Destination (0) combination.

This doesn't matter too much to me at the moment but ,if in future I want to selectively "exclude" traffic for a particular device when all its traffic is normally routed via the VPN, how would I ensure that this traffic always goes out of the WAN connection (0)? Would I have to do this per device or per VPN connection?

The "ALL" source parameter is only valid if you want to route ALL traffic for an IPSET through a VPN Client. What you are trying to do is called VPN Bypass Routing in the new version. x3mRouting needs to know the VPN Client you want to bypass for configuration purposes.

Use this approach to bypass the VPN Client for traffic matching an IPSET list and route to the WAN interface. This approach is often used when a rule to route a specific device or the entire LAN (e.g. 192.168.1.0/24) thru a VPN Client exists in the Policy Routing section of the OpenVPN Client Screen and an exception needs to be made to bypass the VPN Client for a service that blocks known VPN Servers. For example,

x3mRouting 1 0 NETFLIX asnum=AS2906

Please refer to the Usage Notes and VPN Bypass Routing examples on the README.

Did you look in /jffs/scripts/x3mRouting folder for the file x3mRouting_Conversion.sh? Or, PM me your nat-start from the old version? The conversion code is looking for the specific program names from the old version in nat-start. I ran another test and it picked up the entries.

 

[h0me5k1n]

@Xentrk thanks for the explanation... That makes sense! The readme is getting pretty big and I think I just skipped that

There was no conversion file... I'm not too worried about this but will pm you my old one.

 

[Xentrk]

@Xentrk thanks for the explanation... That makes sense! The readme is getting pretty big and I think I just skipped that

There was no conversion file... I'm not too worried about this but will pm you my old one.

Thank you. Yeah, the README is a lot to read. I want to off load some of the details in a wiki to make it easier on the eyes.

I see the PM and will look it over to determine why it didn't get created. With the prior version, the set-up had to be done manually. Which can result in lack of uniformity in implementation.

 

[Xentrk]

@andresmorago

I thought some more about the idea you mentioned about placing the top level domains in a file. Would that work better for your use case? Something like this:

x3mRouting ALL 1 MYIPSET dnsmasq_file=/opt/tmp/MYIPSET

/opt/tmp/MYIPSET

domain1.com
domain2.com
domain3.com
<snip>

The tricky part with the current implementation and the file method is what happens if you want to remove a domain from the list? The IPv4 address will be in the ipset list and the save/restore file. It may be too difficult to match the IPv4 address with the associated domain. If it is known, one can use the following command to remove an entry:

ipset del MYIPSET 91.83.231.25

The easier way is to remove the save/restore file and run the x3mRouting command with the del option to remove all traces of the old IPv4 reference. Then, run x3mRouting command to recreate the IPSET list. I'll have to add a note about this on the README.

 

[Xentrk]

3 July, 2020 Update:

The x3mRouting menu has been patched to fix the issue reported by @h0me5k1n which resulted in the x3mRouting_Conversion.sh file not getting created during the update process if the full path of the script location was not specified.

 

[diyguy]

I am unable to install this through amtm for some reason. I tried to install it but it just returns to the main menu. Any ideas ?

EDIT: nevermind, I see I needed Entware to be installed before this. Will try again.

also I am getting this error when I try to run autoscan and get domain names

Error: /opt/var/log/dnsmasq.log file does not exist

 

[Xentrk]

I am unable to install this through amtm for some reason. I tried to install it but it just returns to the main menu. Any ideas ?

EDIT: nevermind, I see I needed Entware to be installed before this. Will try again.

also I am getting this error when I try to run autoscan and get domain names

Error: /opt/var/log/dnsmasq.log file does not exist

dnsmasq logging needs to be enabled. You can use these instructions to set it up.
https://github.com/Xentrk/x3mRouting#enable-dnsmasq-logging

I will update the code to prompt the user if they want the script to perform the dnsmasq.log setup if it doesn't exist. Need a day or two though.