x3mRouting x3mRouting ~ Selective Routing for Asuswrt-Merlin Firmware

[Luizlp10]

This script is a blessing from the divine. I fought really hard trying to send some streaming-latency-sensitive traffic through my WAN outside my VPN tunnel, and for days I was unable to do so. A lot of reading from this great forum and Github finally got things going for me. All I needed to do was lookout for the ASN codes, set the ipset straight and it all worked like magic from netflix, to some local media companies. Thank you, sir!

Only question I have: it is normal to have "liststats" netflix number growing indefinetely?

 

[Xentrk]

This script is a blessing from the divine. I fought really hard trying to send some streaming-latency-sensitive traffic through my WAN outside my VPN tunnel, and for days I was unable to do so. A lot of reading from this great forum and Github finally got things going for me. All I needed to do was lookout for the ASN codes, set the ipset straight and it all worked like magic from netflix, to some local media companies. Thank you, sir!

Only question I have: it is normal to have "liststats" netflix number growing indefinetely?

The good news is that IPSET is super efficient and can handle large lists. The largest IPSET list I have is Skynet-Blacklist with 194,028 entries.

What size is it? It should stop growing eventually. I'm using the ASN method for NF e.g. AS2906 and my list has 154 entries. But ASN method uses IPv4 addresses in CIDR format (e.g. 192.168.77.0/24). Whereas the IPs collected using the dnsmasq method use individual IPv4 address and may result in hundreds of entries.

 

[Luizlp10]

The good news is that IPSET is super efficient and can handle large lists. The largest IPSET list I have is Skynet-Blacklist with 194,028 entries.

What size is it? It should stop growing eventually. I'm using the ASN method for NF e.g. AS2906 and my list has 154 entries. But ASN method uses IPv4 addresses in CIDR format (e.g. 192.168.77.0/24). Whereas the IPs collected using the dnsmasq method use individual IPv4 address and may result in hundreds of entries.

As of now it is 1453. I wonder if this number can vary from the region and how netflix uses peering servers.

 

[box4m]

I think only two to three ppl that have had to define a rule to route ISP to the WAN. Most ppl just put the router IP address in the Policy Routing GUI so services on the router, like NTP, can still work and get data even if the OpenVPN client goes down and one has "Block traffic if the tunnel goes down" button checked that is shown when Policy Rules is enabled.

The ISP is the backbone to the WWW and will direct traffic where you want it. It is not necessarily the endpoint.

One person had Xfinity Internet Service Provider and there was a feature that did work with the Xfinity unless they routed the ISP ASN to the WAN. This makes sense since Xfinity is the ISP and is expecting you to be coming fro the IP address they assign you.

The other reason that was provided for routing ISP ASN to the WAN is that many streaming services use a Content Delivery Service or CDN. They cache content locally to reduce buffering and provide faster response times. For reasons unknown, the person had issues with a streaming service unless they routed the ISP to the WAN. There are many variables involved such as VPN Provider, DNS Settings on the WAN and OpenVPN Client Page, Policy Rules or RPDB priorities that come into play. This is an outlier use case that only a few ppl have reported as being necessary.

See the Policy Rule Routing post for a basic overview of selective routing and use case examples.

Thanks.

Perhaps thats why i have to add ISP ASN, i read a post from you before mentioning many ppl add the router IP in policy routing to go to WAN, and rest of the LAN goes through VPN, like you say, so that the router can access internet even without VPN.

Can this be the reason i have to add ISP ASN? I only have 192.168.1.0/24 -> VPN in policy routing.
I cannot test this until later today

Thanks

 

[figorr]

Nice script.

I am seriously considering start using VPN for my home Network. I have been reading your posts about TorGuard,
https://www.snbforums.com/threads/t...for-asus-merlin-380-65-380-65_2-part-i.38281/
https://www.snbforums.com/threads/t...or-asus-merlin-380-65-380-65_2-part-ii.38282/
https://www.snbforums.com/threads/t...r-asus-merlin-380-65-380-65_2-part-iii.38283/

And also your blog article about TorGuard
https://x3mtek.com/why-i-use-torguard-as-my-vpn-provider/

And TorGuard I think it will be the final choice (using your coupon code, of course, thank you in advance, ).

But I have some previous questions.

1) I am from Spain. And when I am going to the TorGuard webpage ... there is some differences with respect your blog.

I can choose the Product ANONYMOUS VPN. And then select the BILLING CYCLE. But inside the ADDONS there is no Streaming IP for Spain or even Europe (just for US & UK)

Inside the CONFIGURABLE OPTIONS is available to choose a DEDICATED IP for Spain, but this doesn't look the same as the STREAMING IP ADDON. Am I right? It will work ok?

2) If finally I should choose the ANONYMOUS VPN and the DEDICATED IP for Spain ... How I can use the DEDICATED IP under the CLIENT VPN SETUP in under Merlin 384.16?
I have seen in the TorGuard forum https://forums.torguard.net/index.php?/topic/876-vpn-with-dedicated-ip-is-it-possible-on-a-router/
that someone for support says that it is possible to include the DEDICATED IP if you create a ovpn config file with your IP here >> https://torguard.net/tgconf.php?action=vpn-openvpnconfig

3) I am still reading the info about the script and also the Policy rules section,

https://github.com/Xentrk/x3mRouting#acknowledgements
https://github.com/RMerl/asuswrt-merlin.ng/wiki/Policy-based-routing

Best regards,

 

[Xentrk]

But I have some previous questions.

1) I am from Spain. And when I am going to the TorGuard webpage ... there is some differences with respect your blog.

I can choose the Product ANONYMOUS VPN. And then select the BILLING CYCLE. But inside the ADDONS there is no Streaming IP for Spain or even Europe (just for US & UK)

Inside the CONFIGURABLE OPTIONS is available to choose a DEDICATED IP for Spain, but this doesn't look the same as the STREAMING IP ADDON. Am I right? It will work ok?

You need a dedicated IP for each location. If you want to watch a service in US that blocks VPNs, you need a private IP from US. Similarly, If you want to watch BBC, you need a private IP from the UK. If there is a service you need in Spain or another location that blocks known VPN servers, you could ask TG if they can spin one up for you. Or look into using a dnsproxy service. It is normally the major streaming services who are doing the blocking of VPN servers.

2) If finally I should choose the ANONYMOUS VPN and the DEDICATED IP for Spain ... How I can use the DEDICATED IP under the CLIENT VPN SETUP in under Merlin 384.16?
I have seen in the TorGuard forum https://forums.torguard.net/index.php?/topic/876-vpn-with-dedicated-ip-is-it-possible-on-a-router/
that someone for support says that it is possible to include the DEDICATED IP if you create a ovpn config file with your IP here >> https://torguard.net/tgconf.php?action=vpn-openvpnconfig

Just create the ovpnc.config file using the link you reference is all that is required. After you import it, you then have to enter your userid and password and apply the settings.

3) I am still reading the info about the script and also the Policy rules section,

The new version will be easier to use. It should be ready by end of April. The setup is now automated so things will restore at boot time.

 

[figorr]

Thank you.

Now everything is more clear.

I will choose TorGuard then. And awaiting for the new script version. I am a noob, so the easiest way is always welcomed.

 

[TechTinkerer]

@Xentrk kinda stuck tried to capture, see log below

ASUSWRT-Merlin RT-AC86U 384.16_0 Sun Apr  5 17:38:01 UTC 2020
KBvUTgSItSQdjpCB@RT-AC86U-87E8:/tmp/home/root# cd /opt/var/log
KBvUTgSItSQdjpCB@RT-AC86U-87E8:/tmp/mnt/USBHDD/entware/var/log# tail -f dnsmasq.log > NETFLIX
tail: can't open 'dnsmasq.log': No such file or directory
tail: no files
KBvUTgSItSQdjpCB@RT-AC86U-87E8:/tmp/mnt/USBHDD/entware/var/log# cd /opt/var/log
KBvUTgSItSQdjpCB@RT-AC86U-87E8:/tmp/mnt/USBHDD/entware/var/log# ls
NETFLIX                    syslog-ng.log-20200322.gz  wlceventd.log              wlceventd.log-20200412
logrotate.log              syslog-ng.log-20200329.gz  wlceventd.log-20200322.gz
messages                   syslog-ng.log-20200405.gz  wlceventd.log-20200329.gz
syslog-ng.log              syslog-ng.log-20200412     wlceventd.log-20200405.gz
KBvUTgSItSQdjpCB@RT-AC86U-87E8:/tmp/mnt/USBHDD/entware/var/log#
KBvUTgSItSQdjpCB@RT-AC86U-87E8:/tmp/mnt/USBHDD/entware/var/log# cd /jffs/scripts
KBvUTgSItSQdjpCB@RT-AC86U-87E8:/jffs/scripts# sh getdomainnames.sh NETFLIX 192.168.10.13
Error! /opt/var/log/NETFLIX does not exist

I think I did it correct...any idea?

Thanks

 

[Xentrk]

@Xentrk kinda stuck tried to capture, see log below

ASUSWRT-Merlin RT-AC86U 384.16_0 Sun Apr  5 17:38:01 UTC 2020
KBvUTgSItSQdjpCB@RT-AC86U-87E8:/tmp/home/root# cd /opt/var/log
KBvUTgSItSQdjpCB@RT-AC86U-87E8:/tmp/mnt/USBHDD/entware/var/log# tail -f dnsmasq.log > NETFLIX
tail: can't open 'dnsmasq.log': No such file or directory
tail: no files
KBvUTgSItSQdjpCB@RT-AC86U-87E8:/tmp/mnt/USBHDD/entware/var/log# cd /opt/var/log
KBvUTgSItSQdjpCB@RT-AC86U-87E8:/tmp/mnt/USBHDD/entware/var/log# ls
NETFLIX                    syslog-ng.log-20200322.gz  wlceventd.log              wlceventd.log-20200412
logrotate.log              syslog-ng.log-20200329.gz  wlceventd.log-20200322.gz
messages                   syslog-ng.log-20200405.gz  wlceventd.log-20200329.gz
syslog-ng.log              syslog-ng.log-20200412     wlceventd.log-20200405.gz
KBvUTgSItSQdjpCB@RT-AC86U-87E8:/tmp/mnt/USBHDD/entware/var/log#
KBvUTgSItSQdjpCB@RT-AC86U-87E8:/tmp/mnt/USBHDD/entware/var/log# cd /jffs/scripts
KBvUTgSItSQdjpCB@RT-AC86U-87E8:/jffs/scripts# sh getdomainnames.sh NETFLIX 192.168.10.13
Error! /opt/var/log/NETFLIX does not exist

I think I did it correct...any idea?

Thanks

dnsmasq.log file is not in the /opt/var/log directory on your router. I'll add a check to the script to make sure that that /opt/var/log/dnsmasq.log file exists and provide a message if not found.

I use Diversion which has enhanced dnsmasq functionality and does all of the setup for dnsmasq. For those that don't use Diversion, the dnsmasq.log file will probably be located either in /tmp or in /var/lib. You can use this command to find the location.

find / -name dnsmasq.log

You will have to edit the script to reflect the default directory location or install Diversion to setup dnsmasq for you.

 

[TechTinkerer]

dnsmasq.log file is not in the /opt/var/log directory on your router. I'll add a check to the script to make sure that that /opt/var/log/dnsmasq.log file exists and provide a message if not found.

I use Diversion which has enhanced dnsmasq functionality and does all of the setup for dnsmasq. For those that don't use Diversion, the dnsmasq.log file will probably be located either in /tmp or in /var/lib. You can use this command to find the location.

find / -name dnsmasq.log

You will have to edit the script to reflect the default directory location or install Diversion to setup dnsmasq for you.

command didn't work drawing a blank at locating that dnsmasq.log

 

[Xentrk]

command didn't work drawing a blank at locating that dnsmasq.log

Appears that dnsmasq logging is not enabled.

I've never tried to set it up manually since I use Diversion. These steps should work though.

Use your favorite editor and edit the file /jffs/configs/dnsmasq.conf.add. Add the entry

log-facility=/opt/var/log/dnsmasq.log

Restart dnsmasq

service restart_dnsmasq

If you still have an issue, try some of the steps in the wiki.
https://github.com/RMerl/asuswrt-merlin.ng/wiki/Custom-domains-with-dnsmasq

 

[TechTinkerer]

Appears that dnsmasq logging is not enabled.

I've never tried to set it up manually since I use Diversion. These steps should work though.

Use your favorite editor and edit the file /jffs/configs/dnsmasq.conf.add. Add the entry

log-facility=/opt/var/log/dnsmasq.log

Restart dnsmasq

service restart_dnsmasq

If you still have an issue, try some of the steps in the wiki.
https://github.com/RMerl/asuswrt-merlin.ng/wiki/Custom-domains-with-dnsmasq

That did it, Thanks still waiting to test your new script maybe you should add that to the dnsmasq detection in the script if log doesn't exist or not using diversion, user gets prompted to automate the setup of the log as a prequisite with choice of just log or setup using diversion

 

[Luizlp10]

The good news is that IPSET is super efficient and can handle large lists. The largest IPSET list I have is Skynet-Blacklist with 194,028 entries.

What size is it? It should stop growing eventually. I'm using the ASN method for NF e.g. AS2906 and my list has 154 entries. But ASN method uses IPv4 addresses in CIDR format (e.g. 192.168.77.0/24). Whereas the IPs collected using the dnsmasq method use individual IPv4 address and may result in hundreds of entries.

I have a specific IP that i would like to add to the ipset NETFLIX list. An entry like this [ ipset add NETFLIX 50.237.33.212] should be effective or it's nonsense?

 

[Xentrk]

I have a specific IP that i would like to add to the ipset NETFLIX list. An entry like this [ ipset add NETFLIX 50.237.33.212] should be effective or it's nonsense?

That should work okay if you use the DNSMASQ method for NETFLIX. Once you add the entry to the IPSET list, the IP address you added will then get copied over to the backup/restore file when the nightly backup runs at 2 AM.

If you use the ASN method, the IP addresses are first downloaded to the backup/restore location. It is then loaded into the IPSET list from the backup/restore file. The current version of x3mRouting will write over any existing entries in the backup/restore file. So the entry will get wiped out. I changed the approach in the new version so entries are appended to the current entries in the backup/restore file. A unique sort is then performed on the the backup/restore file to eliminate duplicates before loading to the IPSET list. I changed how it is managed to provide the feature for ppl to append more than one ASN to an IPSET list. So, if you use the ASN method, create a separate list and route it to the same interface as NETFLIX. Once the new version is out, you can then manually add it to the backup/restore file and it won't get wiped out when a refresh occurs.

 

[Zacco]

I have two streaming services that both us Amazon AS 16509. I have tried to only us DNSMASQ for these services but Netflix wont work unless I use ASN and then my other service that I want to route through WAN will get geo blocked. Any ides how to fix this?

 

[Xentrk]

I have two streaming services that both us Amazon AS 16509. I have tried to only us DNSMASQ for these services but Netflix wont work unless I use ASN and then my other service that I want to route through WAN will get geo blocked. Any ides how to fix this?

I noticed Disney+ is using AS16509 too. ASN method can cast too wide of a net in some cases.

The dnsmasq method is probably the best method to use for these situations. What domains did you use?

First try the NF domains:

netflix.com,nflxext.com,nflximg.net,nflxso.net,nflxvideo.net

If you still have problems, add amazonaws.com

amazonaws.com,netflix.com,nflxext.com,nflximg.net,nflxso.net,nflxvideo.net

You should delete the current ipset list so you start with a fresh list.

 

[Luizlp10]

That should work okay if you use the DNSMASQ method for NETFLIX. Once you add the entry to the IPSET list, the IP address you added will then get copied over to the backup/restore file when the nightly backup runs at 2 AM.

If you use the ASN method, the IP addresses are first downloaded to the backup/restore location. It is then loaded into the IPSET list from the backup/restore file. The current version of x3mRouting will write over any existing entries in the backup/restore file. So the entry will get wiped out. I changed the approach in the new version so entries are appended to the current entries in the backup/restore file. A unique sort is then performed on the the backup/restore file to eliminate duplicates before loading to the IPSET list. I changed how it is managed to provide the feature for ppl to append more than one ASN to an IPSET list. So, if you use the ASN method, create a separate list and route it to the same interface as NETFLIX. Once the new version is out, you can then manually add it to the backup/restore file and it won't get wiped out when a refresh occurs.

This is very efficient. Did'nt know that IPSET lists were handled that way. I will learn more about your script to optimize my setup.

 

[Xentrk]

This is very efficient. Did'nt know that IPSET lists were handled that way. I will learn more about your script to optimize my setup.

The default location for the save/restore files is /opt/tmp. Sorry, I forgot to mention that.

 

[Zacco]

I noticed Disney+ is using AS16509 too. ASN method can cast too wide of a net in some cases.

The dnsmasq method is probably the best method to use for these situations. What domains did you use?

First try the NF domains:

netflix.com,nflxext.com,nflximg.net,nflxso.net,nflxvideo.net

If you still have problems, add amazonaws.com

amazonaws.com,netflix.com,nflxext.com,nflximg.net,nflxso.net,nflxvideo.net

You should delete the current ipset list so you start with a fresh list.

This is what I have tried

# *ipTV*
# sh /jffs/scripts/x3mRouting/load_ASN_ipset_iface.sh 2 GenIPTV-134512 A134512

# *VIAPLAY*
sh /jffs/scripts/x3mRouting/load_DNSMASQ_ipset_iface.sh 0 VIAPLAY viaplay.se,viaplay.tv

# *NETFLIX*
sh /jffs/scripts/x3mRouting/load_ASN_ipset_iface.sh 1 NETFLIX-812 AS812
sh /jffs/scripts/x3mRouting/load_ASN_ipset_iface.sh 1 NETFLIX-2906 AS2906
sh /jffs/scripts/x3mRouting/load_ASN_ipset_iface.sh 1 NETFLIX-14618 AS14618
sh /jffs/scripts/x3mRouting/load_ASN_ipset_iface.sh 1 NETFLIX-394406 AS394406
sh /jffs/scripts/x3mRouting/load_ASN_ipset_iface.sh 1 AMAZON-16509 AS16509

sh /jffs/scripts/x3mRouting/load_DNSMASQ_ipset_iface.sh 1 NETFLIX_WEB1 amazonaws.com,netflix.com,netflix.net,nflxext.com,nflximg.net,nflxso.net,nflxvideo.net,dvd.netflix.com

Missed deleting ipset list

So this might work.

# *NETFLIX*
sh /jffs/scripts/x3mRouting/load_DNSMASQ_ipset_iface.sh 1 NETFLIX_WEB1 amazonaws.com,netflix.com,netflix.net,nflxext.com,nflximg.net,nflxso.net,nflxvideo.net,dvd.netflix.com
# *VIAPLAY*
sh /jffs/scripts/x3mRouting/load_DNSMASQ_ipset_iface.sh 0 VIAPLAY viaplay.se,viaplay.tv

 

[Xentrk]

This is what I have tried

# *ipTV*
# sh /jffs/scripts/x3mRouting/load_ASN_ipset_iface.sh 2 GenIPTV-134512 A134512

# *VIAPLAY*
sh /jffs/scripts/x3mRouting/load_DNSMASQ_ipset_iface.sh 0 VIAPLAY viaplay.se,viaplay.tv

# *NETFLIX*
sh /jffs/scripts/x3mRouting/load_ASN_ipset_iface.sh 1 NETFLIX-812 AS812
sh /jffs/scripts/x3mRouting/load_ASN_ipset_iface.sh 1 NETFLIX-2906 AS2906
sh /jffs/scripts/x3mRouting/load_ASN_ipset_iface.sh 1 NETFLIX-14618 AS14618
sh /jffs/scripts/x3mRouting/load_ASN_ipset_iface.sh 1 NETFLIX-394406 AS394406
sh /jffs/scripts/x3mRouting/load_ASN_ipset_iface.sh 1 AMAZON-16509 AS16509

sh /jffs/scripts/x3mRouting/load_DNSMASQ_ipset_iface.sh 1 NETFLIX_WEB1 amazonaws.com,netflix.com,netflix.net,nflxext.com,nflximg.net,nflxso.net,nflxvideo.net,dvd.netflix.com

Missed deleting ipset list

So this might work.

# *NETFLIX*
sh /jffs/scripts/x3mRouting/load_DNSMASQ_ipset_iface.sh 1 NETFLIX_WEB1 amazonaws.com,netflix.com,netflix.net,nflxext.com,nflximg.net,nflxso.net,nflxvideo.net,dvd.netflix.com
# *VIAPLAY*
sh /jffs/scripts/x3mRouting/load_DNSMASQ_ipset_iface.sh 0 VIAPLAY viaplay.se,viaplay.tv

Keep in mind that using the "amazonaws.com" domain will force Amazon Prime streaming to get routed thru the tunnel. The "dvd.netflix.com" isn't reqired as the "netflix.com" entry will capture the IPv4 address for that domain.

Use the command below to see if packets are traversing the iptables chain.

iptables -nvL PREROUTING -t mangle --line