Asus merlin router, vpn, selective policy website routing

[MornFall]

Hey everyone,

So i have been browsing quite intensively all day the foum and the few different custom thingies for my RT-AC87R.
I was looking for a solution to use my vpn only for france.tv website.
But after spending all day , i can not find the solution .
I first installed merlin, then setup the USB key with the swap , then amtm and diversion.

I finished by trying x3mRouting ( with option #3 ), but i m running into the following issue :
After running :
"/jffs/scripts/x3mRouting/load_DNSMASQ_ipset.sh FRANCE france.tv "
and then doing "liststats" , i do see the IPSET FRANCE, but it stays at 0. no matter how long i wait.

I have tried quite a few things, like : Tools -> Other Settings page -> Wan: Use local caching DNS server as system resolver (default: No) changed to YES.
multiple reboot and attempts later, no matter what i do, it stays at 0.

@Xentrk : What am i missing please ?
I am a complete nuub and had to go find tutorial on how to ssh into a linux , so i m probably missing a simple step, but just can t figure it out...

 

[Xentrk]

Hey everyone,

So i have been browsing quite intensively all day the foum and the few different custom thingies for my RT-AC87R.
I was looking for a solution to use my vpn only for france.tv website.
But after spending all day , i can not find the solution .
I first installed merlin, then setup the USB key with the swap , then amtm and diversion.

I finished by trying x3mRouting ( with option #3 ), but i m running into the following issue :
After running :
"/jffs/scripts/x3mRouting/load_DNSMASQ_ipset.sh FRANCE france.tv "
and then doing "liststats" , i do see the IPSET FRANCE, but it stays at 0. no matter how long i wait.

I have tried quite a few things, like : Tools -> Other Settings page -> Wan: Use local caching DNS server as system resolver (default: No) changed to YES.
multiple reboot and attempts later, no matter what i do, it stays at 0.

@Xentrk : What am i missing please ?
I am a complete nuub and had to go find tutorial on how to ssh into a linux , so i m probably missing a simple step, but just can t figure it out...

You need to execute the Method 3 version of the script and specify the interface. In the example below, WAN is specified using 0.

sh /jffs/sripts/x3mRouting/load_DNSMASQ_ipset_iface.sh 0 FRANCE france.tv

Then, go to france.tv in a browswer and surf the site.

# ipset -L FRANCE

Name: FRANCE
Type: hash:net
Revision: 6
Header: family inet hashsize 1024 maxelem 65536
Size in memory: 372
References: 1
Number of entries: 1
Members:
23.40.133.29

You can try adding the "autoscan" parameter to pick up other domains that site is using that have france.tv as part of the domain name.

However, additional analysis may be required such as mining dnsmasq.log to see what other domains the site is using. See the post for an explanation.

 

[TonyK132]

PART 3

Question 7
The IP addresses for the ASN method are fairly static and don’t change that much. To test this, I turned off updating of the lists for a year on one of my routers and the streaming services continued to work. There is no cron job for the ASN method. Whenever the script runs, it does check to see if the list is older than 7 days and downloads a new list of IP addresses if so. I will consider changing the approach to use cron job though.

The only cron jobs are for the DNSMASQ scripts. The IP addresses are generated dynamically as one surfs the internet or streams. The cron job backs up the generated list to /opt/tmp at 2:00 AM. To change this, one would need to edit the script and reboot for the change to take effect.

Question 8
The project has been in soft launch mode so I could get feedback before a formal go-live. I’ll start preparing a formal announcement and thread for the project and work with @thelonelycoder to add x3mRouting to AMTM. I am targeting next weekend for a formal go-live. I have a full time job and things have been very busy for me. But will try my best to get this done.

This concludes the replies. Let me know if you have more questions. Thanks again for the feedback.

Very thorough responses, thanks much. I do appreciate that you have a day job, but also you care about this project. Please take my suggestions as mere suggestions for your consideration.

Roku is kind of an odd duck. When I wrote that comment, I was thinking of Roku videos that might require their own list of IP addrs. That means Roku would have 3 sets of addrs: their browsing, searches, etc, for which I would like to use my VPN; their videos, which would probably need to use the WAN like everyone else with their videos; and the other services that Roku supports, like Netflix, Amazon, HBO, etc. From my limited using of those services from Roku, the list IP addrs for Netflix and Amazon seem to also be the ones that Roku uses when they invoke those services, so nothing needs to change there if you are already filtering for Netflix, etc.

For HBO NOW, here are the ranges I got that seems to work, although I do not know if all 3 are needed:
206.203.65.0/24
206.203.67.0/24
206.203.69.0/24

For Roku,
50.224.7.0/24
although I have not tried to do a Roku video to see if it works.

In looking at the files in \opt\tmp, some have an "add" before the IP addr, others do not. Are you filtering out the add if it is there, so that that word is optional in the IP list files?

Again, thanks much for your hard work in doing this.

 

[MornFall]

You need to execute the Method 3 version of the script and specify the interface. In the example below, WAN is specified using 0.

sh /jffs/sripts/x3mRouting/load_DNSMASQ_ipset_iface.sh 0 FRANCE france.tv

Then, go to fance.tv in a browswer and surf the site the site.

# ipset -L FRANCE

Name: FRANCE
Type: hash:net
Revision: 6
Header: family inet hashsize 1024 maxelem 65536
Size in memory: 372
References: 1
Number of entries: 1
Members:
23.40.133.29

You can try adding the "autoscan" parameter to pick up other domains that site is using that have france.tv as part of the domain name.

However, additional analysis may be required such as mining dnsmasq.log to see what other domains the site is using. See the post for an explanation.

Thank you for your hardwork and answers.

I just tried ( re-tried actually... )
and no matter what it doesn t pick up any entries :



Romain@RT-AC87R-2FC8:/# /jffs/scripts/x3mRouting/load_DNSMASQ_ipset_iface.sh 0 FRANCE france.tv
(load_DNSMASQ_ipset_iface.sh): 7978 Starting Script Execution
(load_DNSMASQ_ipset_iface.sh): 7978 IPSET created: FRANCE hash:net family inet hashsize 1024 maxelem 65536
(load_DNSMASQ_ipset_iface.sh): 7978 CRON schedule created: #FRANCE# '0 2 * * * ipset save FRANCE'
(load_DNSMASQ_ipset_iface.sh): 7978 Selective Routing Rule via WAN created for FRANCE TAG fwmark 0x8000/0x8000
(load_DNSMASQ_ipset_iface.sh): 7978 Completed Script Execution

There i go browse the website, then go back in ssh and do :

Romain@RT-AC87R-2FC8:/# liststats
FRANCE - 0
Romain@RT-AC87R-2FC8:/# ipset -L FRANCE
Name: FRANCE
Type: hash:net
Revision: 6
Header: family inet hashsize 1024 maxelem 65536
Size in memory: 316
References: 1
Number of entries: 0
Members:
Romain@RT-AC87R-2FC8:/#

PS : No matter which site i try can be google, youtube, ect... it doesn t register any entry ( after making a new "load_DNSMASQ_ipset_iface.sh" command ofc)

 

[Martineau]

Thank you for your hardwork and answers.

I just tried ( re-tried actually... )
and no matter what it doesn t pick up any entries :



Romain@RT-AC87R-2FC8:/# /jffs/scripts/x3mRouting/load_DNSMASQ_ipset_iface.sh 0 FRANCE france.tv
(load_DNSMASQ_ipset_iface.sh): 7978 Starting Script Execution
(load_DNSMASQ_ipset_iface.sh): 7978 IPSET created: FRANCE hash:net family inet hashsize 1024 maxelem 65536
(load_DNSMASQ_ipset_iface.sh): 7978 CRON schedule created: #FRANCE# '0 2 * * * ipset save FRANCE'
(load_DNSMASQ_ipset_iface.sh): 7978 Selective Routing Rule via WAN created for FRANCE TAG fwmark 0x8000/0x8000
(load_DNSMASQ_ipset_iface.sh): 7978 Completed Script Execution

There i go browse the website, then go back in ssh and do :

Romain@RT-AC87R-2FC8:/# liststats
FRANCE - 0
Romain@RT-AC87R-2FC8:/# ipset -L FRANCE
Name: FRANCE
Type: hash:net
Revision: 6
Header: family inet hashsize 1024 maxelem 65536
Size in memory: 316
References: 1
Number of entries: 0
Members:
Romain@RT-AC87R-2FC8:/#

PS : No matter which site i try can be google, youtube, ect... it doesn t register any entry ( after making a new "load_DNSMASQ_ipset_iface.sh" command ofc)

Works for me..

Check dnsmasq is configured:

grep  "^ipset" /etc/dnsmasq.conf

ipset=/france.tv/FRANCE

Generate IP, and check if dnsmasq has auto-populated the Ipset:

nslookup france.tv

Server:    127.0.0.1
Address 1: 127.0.0.1 localhost.localdomain
Name:      france.tv
Address 1: 92.123.129.239 a92-123-129-239.deploy.static.akamaitechnologies.com


ipset -L FRANCE

Name: FRANCE
Type: hash:net
Revision: 6
Header: family inet hashsize 1024 maxelem 65536
Size in memory: 428
References: 1
Number of entries: 1
Members:
92.123.129.239

Repeat

nslookup france.tv

Server:    127.0.0.1
Address 1: 127.0.0.1 localhost.localdomain
Name:      france.tv
Address 1: 104.84.242.241 a104-84-242-241.deploy.static.akamaitechnologies.com


ipset -L FRANCE

Name: FRANCE
Type: hash:net
Revision: 6
Header: family inet hashsize 1024 maxelem 65536
Size in memory: 428
References: 1
Number of entries: 2
Members:
92.123.129.239
104.84.242.241

 

[Martineau]

sh /jffs/sripts/x3mRouting/load_DNSMASQ_ipset_iface.sh 0 FRANCE france.tv

Then, go to fance.tv in a browswer and surf the site the site.

Two typos

 

[consorts]

skimming this thread, it occurs to me that yafzi should add some form of "whitelist" feature in his ssh based gui making it easy for users to add domains or ip's that users want skipping their vpn and using their wan.

 

[Xentrk]

Thank you for your hardwork and answers.

I just tried ( re-tried actually... )
and no matter what it doesn t pick up any entries :



Romain@RT-AC87R-2FC8:/# /jffs/scripts/x3mRouting/load_DNSMASQ_ipset_iface.sh 0 FRANCE france.tv
(load_DNSMASQ_ipset_iface.sh): 7978 Starting Script Execution
(load_DNSMASQ_ipset_iface.sh): 7978 IPSET created: FRANCE hash:net family inet hashsize 1024 maxelem 65536
(load_DNSMASQ_ipset_iface.sh): 7978 CRON schedule created: #FRANCE# '0 2 * * * ipset save FRANCE'
(load_DNSMASQ_ipset_iface.sh): 7978 Selective Routing Rule via WAN created for FRANCE TAG fwmark 0x8000/0x8000
(load_DNSMASQ_ipset_iface.sh): 7978 Completed Script Execution

There i go browse the website, then go back in ssh and do :

Romain@RT-AC87R-2FC8:/# liststats
FRANCE - 0
Romain@RT-AC87R-2FC8:/# ipset -L FRANCE
Name: FRANCE
Type: hash:net
Revision: 6
Header: family inet hashsize 1024 maxelem 65536
Size in memory: 316
References: 1
Number of entries: 0
Members:
Romain@RT-AC87R-2FC8:/#

PS : No matter which site i try can be google, youtube, ect... it doesn t register any entry ( after making a new "load_DNSMASQ_ipset_iface.sh" command ofc)

Do you have Policy Rules enabled on the OpenVPN Client Screen?

 

[Xentrk]

Very thorough responses, thanks much. I do appreciate that you have a day job, but also you care about this project. Please take my suggestions as mere suggestions for your consideration.

Roku is kind of an odd duck. When I wrote that comment, I was thinking of Roku videos that might require their own list of IP addrs. That means Roku would have 3 sets of addrs: their browsing, searches, etc, for which I would like to use my VPN; their videos, which would probably need to use the WAN like everyone else with their videos; and the other services that Roku supports, like Netflix, Amazon, HBO, etc. From my limited using of those services from Roku, the list IP addrs for Netflix and Amazon seem to also be the ones that Roku uses when they invoke those services, so nothing needs to change there if you are already filtering for Netflix, etc.

For HBO NOW, here are the ranges I got that seems to work, although I do not know if all 3 are needed:
206.203.65.0/24
206.203.67.0/24
206.203.69.0/24

For Roku,
50.224.7.0/24
although I have not tried to do a Roku video to see if it works.

In looking at the files in \opt\tmp, some have an "add" before the IP addr, others do not. Are you filtering out the add if it is there, so that that word is optional in the IP list files?

Again, thanks much for your hard work in doing this.

When I looked up the ownership of the HBO NOW IP addresses you listed, I get a hit on https://bgp.he.net/AS10599#_whois. Doesn't seem right.

Following are some domain names I mined from browsing https://play.hbonow.com and using the getdomainnames.sh script to create the list:

artist.api.cdn.hbo.com
artist.api.lv3.cdn.hbo.com
assets.adobedtm.com
comet.api.hbo.com
dcs-edge-usw2-620097651.us-west-2.elb.amazonaws.com
dpm.demdex.net
e7808.g.akamaiedge.net
hbo.map.fastly.net
hbogo.com.102.122.2o7.net
js.stripe.com
m.stripe.com
m.stripe.network
play.hbonow.com
play.hbonow.com.c.footprint.net
smetrics.hbogo.com
ssl.cdn.hbo.com.c.footprint.net
stripecdn.map.fastly.net
subscribe.hbonow.com
tags.bluekai.com
telegraph.api.hbo.com
www.googletagmanager.com

Looks like HBO is using a CDN as most streaming services do to eliminate buffering. Content is cached on servers closer to the user. Mining dnsmasq and testing using the top level domain names requires some trial and error. Here is how you could apply the information above to the DNSMASQ script:

sh load_DNSMASQ_ipset_iface.sh 0 HBO hbonow.com,hbogo.com,hbo.com,102.122.207.net,fastly.net

The format of the IPSET backup files differ depending on the method used to collect the IP addresses. For the IPSETs that are created using the DNSMASQ method, I am using the ipset-save command to save the backup/restore file. It is responsible for the "add IPSET_NAME ip_address" format. I use the ipset-restore command at boot to restore the list.

The IP addresses that are created using the AMAZON, NETFLIX and MANUAL scripts come from online sources. The code loads the ipset list from the text file containing the IP addresses. The code adds the required syntax to follow the ipset-restore format it expects. It also loads the ipset list super fast and is the most efficient method.

awk '{print "add '"$IPSET_NAME"' " $1}' "$DIR/$IPSET_NAME" | ipset restore -!

 

[Xentrk]

Two typos

Yikes! Edited post.

 

[MornFall]

Works for me..

Check dnsmasq is configured:

grep  "^ipset" /etc/dnsmasq.conf

ipset=/france.tv/FRANCE

Generate IP, and check if dnsmasq has auto-populated the Ipset:

nslookup france.tv

Server:    127.0.0.1
Address 1: 127.0.0.1 localhost.localdomain
Name:      france.tv
Address 1: 92.123.129.239 a92-123-129-239.deploy.static.akamaitechnologies.com


ipset -L FRANCE

Name: FRANCE
Type: hash:net
Revision: 6
Header: family inet hashsize 1024 maxelem 65536
Size in memory: 428
References: 1
Number of entries: 1
Members:
92.123.129.239

Repeat

nslookup france.tv

Server:    127.0.0.1
Address 1: 127.0.0.1 localhost.localdomain
Name:      france.tv
Address 1: 104.84.242.241 a104-84-242-241.deploy.static.akamaitechnologies.com


ipset -L FRANCE

Name: FRANCE
Type: hash:net
Revision: 6
Header: family inet hashsize 1024 maxelem 65536
Size in memory: 428
References: 1
Number of entries: 2
Members:
92.123.129.239
104.84.242.241

Check dnsmasq is configured:
That seems to be the issue.

Romain@RT-AC87R-2FC8:/jffs/scripts/x3mRouting# grep  "^ipset" /etc/dnsmasq.conf
Romain@RT-AC87R-2FC8:/jffs/scripts/x3mRouting#

so probably a step that i missed. is there a guide i should follow to set that up ??

EDIT : I tried to edit the dnsmasq.conf myself and add the ipset France entry. But as soon as i restart dnsmasq service , it gets overwritten to what it was before...

 

[MornFall]

Do you have Policy Rules enabled on the OpenVPN Client Screen?

View attachment 18688

Mine is set to policy rules, but not strict, should i change that ?

 

[Xentrk]

Check dnsmasq is configured:
That seems to be the issue.

Romain@RT-AC87R-2FC8:/jffs/scripts/x3mRouting# grep  "^ipset" /etc/dnsmasq.conf
Romain@RT-AC87R-2FC8:/jffs/scripts/x3mRouting#

so probably a step that i missed. is there a guide i should follow to set that up ??

EDIT : I tried to edit the dnsmasq.conf myself and add the ipset France entry. But as soon as i restart dnsmasq service , it gets overwritten to what it was before...

You may not have dnsmasq configured properly. /etc/dnsmasq.conf is not intended for user edits.

On Asuswrt-Merlin, uses can customize dnsmasq configurations in the file /jffs/configs/dnsmasq.conf.add. The script writes the entry "ipset=/france.tv/FRANCE" to /jffs/configs/dnsmasq.conf.add. The script then bounces dnsmasq which appends the entry to /etc/dnsmasq.conf.

grep  "^ipset" /jffs/configs/dnsmasq.conf.add

Most people use Diversion ad-blocker which automatically configures dnsmasq during the setup. If you don't have it installed, review some of the dnsmasq settings listed on the wiki to see if that may be the issue.

Policy Rules vs Policy Rules (Strict)
https://github.com/RMerl/asuswrt-merlin/wiki/Policy-based-routing:

On the OpenVPN Clients page, set "Redirect Internet traffic" to either "Policy Rules" or "Policy Rules (strict)". Strict mode will take additional steps to ensure that there aren't any extra routes that could potentially bypass your tunnel, by only allowing routes that specifically target the tunnel's network interface. This is usually preferred, however this will interfere with any route you might have manually configured on your WAN interface, which is why it is a separate option.

 

[MornFall]

On Asuswrt-Merlin, uses can customize dnsmasq configurations in the file /jffs/configs/dnsmasq.conf.add. The script writes the entry "ipset=/france.tv/FRANCE" to /jffs/configs/dnsmasq.conf.add. The script then bounces dnsmasq which appends the entry to /etc/dnsmasq.conf.

There is no " dnsmasq.conf.add " in /jffs/configs/
I m gonna look and see if i can find what i missed in the setup.

EDIT: FOUND IT.. the step i missed was setting up dnsmasq , from the wiki link that you just gave me .

THANK YOU A TON for the asnwers, the work and everything else !

 

[Xentrk]

There is no " dnsmasq.conf.add " in /jffs/configs/
I m gonna look and see if i can find what i missed in the setup.

EDIT: FOUND IT.. the step i missed was setting up dnsmasq , from the wiki link that you just gave me .

THANK YOU A TON for the asnwers, the work and everything else !

Most of the messages are sent to the System Log since the script is meant to run in the background.

Look at the System Log for messages. Once in the System Log screen, you can search the system log by program name (load_DNSMASQ_ipset_iface.sh) using the search (ctrl-f) feature of the browser.

You can also try running the script in debug mode. Remove the comment from the line #set -x by removing the # symbol in front. You will then get verbose output.

I'll look at the code in more detail as time permits. Looks like I need to add additional error handling if the IPSET entry is not added to /jffs/configs/dnsmasq.conf.add.

Edit: Do you have a /jffs/configs directory?

 

[Xentrk]

There is no " dnsmasq.conf.add " in /jffs/configs/
I m gonna look and see if i can find what i missed in the setup.

EDIT: FOUND IT.. the step i missed was setting up dnsmasq , from the wiki link that you just gave me .

THANK YOU A TON for the asnwers, the work and everything else !

I found the problem. I will have a code update posted shortly.

 

[Xentrk]

@MornFall,

The code has been updated to fix the issue you encountered. /jffs/configs/dnsmasq.conf.add was not being created it it didn't exist.

Type x3mRouting at the command prompt to access the installation menu. Select option [5] = Check for updates to existing x3mRouting installation.

 

[Xentrk]

Here are some other domain names I mined while surfing the site:

6679851.fls.doubleclick.net
6713710.fls.doubleclick.net
a500.w10.akamai.net
a550.g1.akamai.net
a827.g1.akamai.net
accounts.eu1.gigya.com
amplify.outbrain.com
api-embed.webservices.francetelevisions.fr
c.contentsquare.net
cdn.polyfill.io
cdnjs.cloudflare.com
cdns.gigya.com
dart.l.doubleclick.net
dualstack.f3.shared.global.fastly.net
e4041.f.akamaiedge.net
e4041.g.akamaiedge.net
education.francetv.fr
eldamar03-loadbalanced.kameleoon.com
ftvingest-vh.akamaihd.net
hdfauthftv-a.akamaihd.net
lb.eu1.gigya.com
oneclient.sfx.ms
sivideo.webservices.francetelevisions.fr
stags.bluekai.com
staticftv-a.akamaihd.net
t.contentsquare.net
tag.aticdn.net
tlp-03.kameleoon.com
www.france.tv
www.https-rulesets.org

You may need to add a few of them to the list of domains, such as francetelevisions.fr and francetv.fr. All of the IP addresses are getting a hit on Akamai Technologies, Inc. which is a CDN.

 

[TonyK132]

When I looked up the ownership of the HBO NOW IP addresses you listed, I get a hit on https://bgp.he.net/AS10599#_whois. Doesn't seem right.

Following are some domain names I mined from browsing https://play.hbonow.com and using the getdomainnames.sh script to create the list:

artist.api.cdn.hbo.com
artist.api.lv3.cdn.hbo.com
assets.adobedtm.com
comet.api.hbo.com
dcs-edge-usw2-620097651.us-west-2.elb.amazonaws.com
dpm.demdex.net
e7808.g.akamaiedge.net
hbo.map.fastly.net
hbogo.com.102.122.2o7.net
js.stripe.com
m.stripe.com
m.stripe.network
play.hbonow.com
play.hbonow.com.c.footprint.net
smetrics.hbogo.com
ssl.cdn.hbo.com.c.footprint.net
stripecdn.map.fastly.net
subscribe.hbonow.com
tags.bluekai.com
telegraph.api.hbo.com
www.googletagmanager.com

Looks like HBO is using a CDN as most streaming services do to eliminate buffering. Content is cached on servers closer to the user. Mining dnsmasq and testing using the top level domain names requires some trial and error. Here is how you could apply the information above to the DNSMASQ script:

sh load_DNSMASQ_ipset_iface.sh 0 HBO hbonow.com,hbogo.com,hbo.com,102.122.207.net,fastly.net

The format of the IPSET backup files differ depending on the method used to collect the IP addresses. For the IPSETs that are created using the DNSMASQ method, I am using the ipset-save command to save the backup/restore file. It is responsible for the "add IPSET_NAME ip_address" format. I use the ipset-restore command at boot to restore the list.

The IP addresses that are created using the AMAZON, NETFLIX and MANUAL scripts come from online sources. The code loads the ipset list from the text file containing the IP addresses. The code adds the required syntax to follow the ipset-restore format it expects. It also loads the ipset list super fast and is the most efficient method.

awk '{print "add '"$IPSET_NAME"' " $1}' "$DIR/$IPSET_NAME" | ipset restore -!

Thanks, I'll play it some more. But for the 3 ranges for HBO Now that I am using for my location, I am able to stream video from a device that sits behind the VPN, so I know at least one of those ranges is right. I'll try to narrow it down later.

 

[Xentrk]

Thanks, I'll play it some more. But for the 3 ranges for HBO Now that I am using for my location, I am able to stream video from a device that sits behind the VPN, so I know at least one of those ranges is right. I'll try to narrow it down later.

Does HBO block known shared VPN servers like Netflix does?