Selective Routing for Netflix

frustin · Jun 26, 2018

Xentrk said:
Those entries are from windows checking for web connecticity.

Do you see log entries from the ip address of the fire tv in dnsmasq.log? Do you assign static ip addresses to your lan clients?

No i dont see the log entries. Perhaps what @Martineau was suggesting? Though when i was using the Adblocker to get the logs at that point it was recording this:

frustin said:

Interesting this is what i get. Nothing in there about "netflix":

Code:

admin1@RT-AC86U:/tmp/mnt/sda1/adblocking/logs# cat netflix
Jun 25 18:10:28 dnsmasq[1585]: forwarded rgom10-en.url.trendmicro.com to 103.86.99.100
Jun 25 18:10:28 dnsmasq[1585]: forwarded rgom10-en.url.trendmicro.com to 103.86.96.100
Jun 25 18:10:28 dnsmasq[1585]: query[AAAA] rgom10-en.url.trendmicro.com from 127.0.0.1
Jun 25 18:10:28 dnsmasq[1585]: forwarded rgom10-en.url.trendmicro.com to 103.86.99.100
Jun 25 18:10:28 dnsmasq[1585]: reply rgom10-en.url.trendmicro.com is <CNAME>
Jun 25 18:10:28 dnsmasq[1585]: reply trendmicro.com.edgesuite.net is <CNAME>
Jun 25 18:10:28 dnsmasq[1585]: reply rgom10-en.url.trendmicro.com is <CNAME>
Jun 25 18:10:28 dnsmasq[1585]: reply trendmicro.com.edgesuite.net is <CNAME>
Jun 25 18:10:28 dnsmasq[1585]: reply a151.g.akamai.net is 42.99.128.161
Jun 25 18:10:28 dnsmasq[1585]: reply a151.g.akamai.net is 42.99.128.168

That trendmicro check happens each time i do a "check your network" on netflix on the Fire TV.

Am i allowing just a151.g.akamai.net?

FYI - I have added a151.g.akamai.net to the dnsmasq.config.add file. Since i posted that.

I use DHCP for all devices, although i lock one address to my QNAP RAID box. The Fire TV uses 10.0.0.252 or something (it has not changed since i begun this).

Would it be better for me to post copy&paste of all my /jffs/scripts and /jffs/config? Anything else?

I've been saying that Netflix is working on my laptop (which it is) but i think it's US netflix as i have The Office (US), which is not on the UK version.

Ted Danson · Jun 27, 2018

Xentrk said:
This is the screen to create the IPv4 list in pfBlockerNG.

View attachment 13383

If you harvested the domain names, create a Firewall Alias using the IP screen and copy/paste them into the Alias.

View attachment 13384

This is how to create the Firewall rule on the LAN page for the Alias. Place the Firewall Alias name or the IPv4 list name as the Destination.

View attachment 13386
View attachment 13385

Just got my pfsense box set up along with a few VPN clients and have selective routing working at a host IP level. So I have the basics there working.

I noticed you are using pfblocker for your alias entries. I tried using just firewall aliases. Only thing is I think I'm going to have to resort to mining domains as Blawk.net and Hurricane Electric are good for finding ASN's but it still needs more domains. Probably other CDN's etc I guess.

How do you go about mining them in pfsense or some other way? I used to try just running Wireshark but of course it won't decrypt HTTPS/SSL so I can't get all the information required.

Have you got any hints and tips for mining domains? I think I'm going to start a pastebin someplace that folks can maintain...

Xentrk · Jun 27, 2018

Ted Danson said:
Just got my pfsense box set up along with a few VPN clients and have selective routing working at a host IP level. So I have the basics there working.

I noticed you are using pfblocker for your alias entries. I tried using just firewall aliases. Only thing is I think I'm going to have to resort to mining domains as Blawk.net and Hurricane Electric are good for finding ASN's but it still needs more domains. Probably other CDN's etc I guess.

How do you go about mining them in pfsense or some other way? I used to try just running Wireshark but of course it won't decrypt HTTPS/SSL so I can't get all the information required.

Have you got any hints and tips for mining domains? I think I'm going to start a pastebin someplace that folks can maintain...

I also use Firewall Aliases for streaming media where I could not get the ASN method to work. Here are two:

I had already captured all of the domain names using my Asus router before moving over to pfS

ense. I use Unbound as the DNS on pfSense as it is tightly coupled with pfBlockerNG. I will update the post later today on how to capture domains from the Unbound log file or using the screen showing the log entries.

I just took a refresher on github over the weekend. I plan to post the scripts I have written for selective routing on github. I can also post the domains I have harvested as well.

Xentrk · Jun 27, 2018

frustin said:
No i dont see the log entries. Perhaps what @Martineau was suggesting? Though when i was using the Adblocker to get the logs at that point it was recording this:

FYI - I have added a151.g.akamai.net to the dnsmasq.config.add file. Since i posted that.

I use DHCP for all devices, although i lock one address to my QNAP RAID box. The Fire TV uses 10.0.0.252 or something (it has not changed since i begun this).

Would it be better for me to post copy&paste of all my /jffs/scripts and /jffs/config? Anything else?

I've been saying that Netflix is working on my laptop (which it is) but i think it's US netflix as i have The Office (US), which is not on the UK version.

I looked at my Amazon Fire TV and did not see any setting for configuring DNS on the device. Nothing popped up when I searched the web. With selective routing, I always recommend people define static IP addresses to LAN clients.

In this code, I have navigated to the location of dnsmasq.log file, which for me is /tmp/mnt/absolution/adblocking/logs.

I then issue the command

Code:

tail -f dnsmasq.log > temp_dnsmasqlog

I then navigate around Netflix or whatever streaming media service I want to capture domains for on my device or devices. When done, I issue a ctrl-C to stop outputting to temp_dnsmasqlog.

I then run the script to search for all query entries in the log file who source is the IP address of the device I was streaming from. The output is sent to /tmp/mnt/absolution/adblocking/logs/temp_dnsmasqlog_output

Code:

#!/bin/sh
#set -xo
# This script will format the output of tail -f dnsmasq.log > logfile
# where logfile is the output of tail -f dnsmasq.log
#  1. extract records whose contents contain the word "query"
#  2. output only the domain name
#  3. sort file for unique contents to elimnate duplicates
#  4. save to $1_output
#
# Parameters Passed
# $1 = provide the name of the source file when running the script
#     e.g. ./getdomainnames.sh logfile
#
source_file=/tmp/mnt/absolution/adblocking/logs/$1
output_file=$source_file"_output"
cat $source_file | grep query | grep 192.168.22.157 | awk '{ print $6 }' | sort -u > $output_file

The contents of the output file contains a sorted file of the domains referenced with duplicates removed.

When testing selective routing for Netflix, I use the browser to confirm region. If I type netflix.com in the browser and the URL does not change, then I am in USA. If I it changes to have the two character country acronym, then I am in my local country.

See if you have any OpenVPN client settings that may differ from this blog post https://x3mtek.com/torguard-openvpn-2-4-client-setup-for-asuswrt-merlin-firmware/
DNS and DNSMasq is handled differently when using selective routing with AB-Solution. This is covered in the DNSmasq and OpenVPN DNS section.

Xentrk · Jun 27, 2018

@thiggins

Can you please move posts 130 thru 141 and posts 162 -163 to a new thread called "Selective Routing using pfSense". The OP topic of this thread is dedicated to Selective Routing for Netflix on AsusWRT-Merlin. I will update the title to include AsusWRT-Merlin in the title after you have moved the posts.

Thank you for your help.

Joseph Douce · Jun 28, 2018

Am I correct this thinking this is the most upto date script to simply route Netflix to WAN. I'm running PIA on merlin and simply want to route Netflix to the WAN and bypass the VPN.

There are many versions spread across this thread maybe you could keep a upto date tutorial in post 1 with the most recent versions as it is a popular search and google brought me here.

Xentrk said:
Thank you for your patience. I was not able to work on this during my recent travels. I ran the 3.2 version the script. I did not get any errors. But the NETFLIX ipset list did not get populated. I then went to ipinfo.io/AS2906 to view the source code to see if changes were required to the sed or grep syntax. I then noticed the url for ipinfo.io had the https rather than the http in the URL. I was able to get the NETFLIX ipset list to populate once I changed the code to reference https://ipinfo.io/AS2906 rather than http://ipinfo.io/AS2906. Here is the updated code.

Code:

#!/bin/sh #################################################################################################### # Script: IPSET_Netflix.sh # Author: Xentrk # 12-Apr-2018 Version 3.3 # Collaborators: @Martineau, @thelonelycoder, @Adamm # # Thank you to @Martineau on snbforums.com for educating myself and others on Selective # Routing using Asuswrt-Merlin firmware. # ##################################################################################################### # Script Description: # # The purpose of this script is for selective routing of Netflix traffic using # Autonomous System Numbers (ASNs). ASNs are assigned to entities such as Internet # Service Providers and other large organizations that control blocks of IP addresses. # # Netflix and other services that use Amazon AWS servers are blocking VPN's. #

I have some updates planned for the script and want to post the next revisions on GitHub. Let me know if the change fixes your issue.

Joseph Douce · Jun 28, 2018

Any ideas

Code:

set add AMAZONAWS 52.144.194.0/26
Bad argument `add'
Try `ipset -H' or 'ipset --help' for more information.
+ ipset add AMAZONAWS 54.150.0.0/16
Bad argument `add'
Try `ipset -H' or 'ipset --help' for more information.
+ ipset add AMAZONAWS 18.200.0.0/16
Bad argument `add'
Try `ipset -H' or 'ipset --help' for more information.
+ ipset add AMAZONAWS 54.206.0.0/16
Bad argument `add'
Try `ipset -H' or 'ipset --help' for more information.
+ ipset add AMAZONAWS 52.95.255.96/28
Bad argument `add'
Try `ipset -H' or 'ipset --help' for more information.
+ ipset add AMAZONAWS 54.231.128.0/19
Bad argument `add'
Try `ipset -H' or 'ipset --help' for more information.
+ ipset add AMAZONAWS 54.226.0.0/15

Ted Danson · Jun 28, 2018

Xentrk said:
I can also post the domains I have harvested as well.

That would be great thank you!

Xentrk · Jun 28, 2018

Joseph Douce said:

Any ideas

Code:

set add AMAZONAWS 52.144.194.0/26
Bad argument `add'
Try `ipset -H' or 'ipset --help' for more information.
+ ipset add AMAZONAWS 54.150.0.0/16
Bad argument `add'
Try `ipset -H' or 'ipset --help' for more information.
+ ipset add AMAZONAWS 18.200.0.0/16
Bad argument `add'
Try `ipset -H' or 'ipset --help' for more information.
+ ipset add AMAZONAWS 54.206.0.0/16
Bad argument `add'
Try `ipset -H' or 'ipset --help' for more information.
+ ipset add AMAZONAWS 52.95.255.96/28
Bad argument `add'
Try `ipset -H' or 'ipset --help' for more information.
+ ipset add AMAZONAWS 54.231.128.0/19
Bad argument `add'
Try `ipset -H' or 'ipset --help' for more information.
+ ipset add AMAZONAWS 54.226.0.0/15

Version 3.3 is the latest version.

What router model and firmware version are you using? I suspect Note 1 applies to you. Run ipset -V to see what version of ipset you have installed. You may have to revised the script, replacing ipset add with ipset -N if you have ipset v4.5.

Code:

# Note 1: IPSET syntax differs between version 6 and 4.5
#             Syntax for ipset v6
#                ipset create WAN0 list:set
#                ipset add WAN0 setlist (e.g. SPEEDTEST)
#             for routers running ipset v4.5 (ipset -V)
#                ipset -N WAN0 setlist (e.g. SPEEDTEST)

Joseph Douce · Jun 29, 2018

RTN66U Merlin 380.70, sorry should've read the note

in 4.5 i changed add to -N but the note also says something about create? I'm still getting the following.

I will try to modify the script to check the version of ipset and use the relevent commands accordingly.

Code:

Try `ipset -H' or 'ipset --help' for more information.
+ ipset -N AMAZONAWS 54.231.253.0/24
ipset v4.5: Couldn't load settype `54.231.253.0/24':File not found
Try `ipset -H' or 'ipset --help' for more information.
+ ipset -N AMAZONAWS 54.231.0.0/17
ipset v4.5: Couldn't load settype `54.231.0.0/17':File not found
Try `ipset -H' or 'ipset --help' for more information.
+ ipset -N AMAZONAWS 52.219.20.0/22

Xentrk · Jun 29, 2018

Joseph Douce said:
RTN66U Merlin 380.70, sorry should've read the note in 4.5 i changed add to -N but the note also says something about create? I'm still getting the following.

I will try to modify the script to check the version of ipset and use the relevent commands accordingly.

Code:

Try `ipset -H' or 'ipset --help' for more information. + ipset -N AMAZONAWS 54.231.253.0/24 ipset v4.5: Couldn't load settype `54.231.253.0/24':File not found Try `ipset -H' or 'ipset --help' for more information. + ipset -N AMAZONAWS 54.231.0.0/17 ipset v4.5: Couldn't load settype `54.231.0.0/17':File not found Try `ipset -H' or 'ipset --help' for more information. + ipset -N AMAZONAWS 52.219.20.0/22

Sorry, I missed an example entry for adding an IP address to an ipset list on version 4.5. See the last line in the code snip below:

Code:

# Note 1: IPSET syntax differs between version 6 and 4.5
#             Syntax for ipset v6
#                ipset create WAN0 list:set
#                ipset add WAN0 ipv4addr
#                --match-set
#             for routers running ipset v4.5 (ipset -V)
#                create ipset list: ipset -N WAN0
#                add ipv4 addresses to ipset list: ipset -A WAN0 ipv4addr
#               --set

Changes need to be:

Code:

ipset -N WAN0 AMAZONAWS

#Pull all IPv4s listed for Amazon AWS

for IPv4 in `jq -r '.prefixes | .[].ip_prefix' < /jffs/scripts/ip-ranges.json`
do
  ipset -A AMAZONAWS $IPv4
done
unset IPv4

--match-set references in the code need to be changed to --set

I will double check for other areas. However, I can't test. If it works for you, I can add logic to check for ipset version and issue the correct syntax.

Joseph Douce · Jun 29, 2018

I added some ipset version checking in but I still get an error, also i don't know if it should be 6 or 6.0 as I don't have v6

Code:

#get ipset version info
IPSET_VER=$(ipset -v)
IPSET_VER=${IPSET_VER:7:3}

# Create IPSET lists
if [ "$IPSET_VER" == "6" ];then
  ipset create AMAZONAWS hash:net family inet hashsize 1024 maxelem 65536
elif [ "$IPSET_VER" == "4.5" ];then
  ipset -N WAN0 AMAZONAWS
else
  echo unsupported ipset v$IPSET_VER
fi

#Pull all IPv4s listed for Amazon AWS
for IPv4 in `jq -r '.prefixes | .[].ip_prefix' < /jffs/scripts/ip-ranges.json`
do
  if [ "$IPSET_VER" == "6" ];then
    ipset add AMAZONAWS $IPv4
  elif [ "$IPSET_VER" == "4.5" ];then
    ipset -A AMAZONAWS $IPv4
  else
    echo unsupported ipset v$IPSET_VER
  fi
done
unset IPv4

Code:

+ ipset -A AMAZONAWS 205.251.228.0/22
ipset v4.5: Unknown set

Xentrk · Jun 29, 2018

Joseph Douce said:

I added some ipset version checking in but I still get an error, also i don't know if it should be 6 or 6.0 as I don't have v6

Code:

#get ipset version info
IPSET_VER=$(ipset -v)
IPSET_VER=${IPSET_VER:7:3}

# Create IPSET lists
if [ "$IPSET_VER" == "6" ];then
  ipset create AMAZONAWS hash:net family inet hashsize 1024 maxelem 65536
elif [ "$IPSET_VER" == "4.5" ];then
  ipset -N WAN0 AMAZONAWS
else
  echo unsupported ipset v$IPSET_VER
fi

#Pull all IPv4s listed for Amazon AWS
for IPv4 in `jq -r '.prefixes | .[].ip_prefix' < /jffs/scripts/ip-ranges.json`
do
  if [ "$IPSET_VER" == "6" ];then
    ipset add AMAZONAWS $IPv4
  elif [ "$IPSET_VER" == "4.5" ];then
    ipset -A AMAZONAWS $IPv4
  else
    echo unsupported ipset v$IPSET_VER
  fi
done
unset IPv4

Code:

+ ipset -A AMAZONAWS 205.251.228.0/22
ipset v4.5: Unknown set

Looks like the AMZONAWS ipset list does not exist. Type ipset -L AMAZONAWS to see if it got created.

Here is a method @redhat27 used in one of his scripts

Code:

case $(ipset -v | grep -o "v[4,6]") in
  v6) MATCH_SET='--match-set'; CREATE='n'; DESTROY='destroy'; RESTORE='restore'; ADD='add'; SWAP='swap'; IPHASH='hash:ip'; NETHASH='hash:net'; ESL=7
      lsmod | grep -q "xt_set" || for module in ip_set ip_set_hash_net ip_set_hash_ip xt_set; do modprobe $module; done;;
  v4) MATCH_SET='--set'; CREATE='-N'; DESTROY='--destroy'; RESTORE='--restore'; ADD='-A'; SWAP='--swap'; IPHASH='iphash'; NETHASH='nethash'; ESL=6
      lsmod | grep -q "ipt_set" || for module in ip_set ip_set_nethash ip_set_iphash ipt_set; do modprobe $module; done;;

Joseph Douce · Jun 29, 2018

no it wasn't created

Code:

+ ipset -N WAN0 AMAZONAWS
ipset v4.5: Couldn't load settype `AMAZONAWS':File not found

Xentrk · Jun 29, 2018

Try to create each list on separate lines:
e.g.
ipset -N WAN0 hash:net
ipset -N AMAZONAWS hash:net

Actually, the script is not using the WAN0 ipset list. I only used WAN0 as an example in the comment section, and it was written poorly after I looked at it again.. I'll change it in the next revision.

Joseph Douce · Jun 29, 2018

no success

Code:

+ ipset -N WAN0 hash:net
ipset v4.5: Couldn't load settype `hash:net':File not found
Try `ipset -H' or 'ipset --help' for more information.
+ ipset -N NETFLIX hash:net
ipset v4.5: Couldn't load settype `hash:net':File not found
Try `ipset -H' or 'ipset --help' for more information.
+ ipset -N AMAZONAWS hash:net
ipset v4.5: Couldn't load settype `hash:net':File not found

Joseph Douce · Jun 29, 2018

almost there it needs to be nethash not hash:net for 4.5 as per @redhat27's script

Code:

# Create IPSET lists
ipset -N WAN0 nethash
ipset -N NETFLIX nethash
ipset -N AMAZONAWS nethash

Xentrk · Jun 29, 2018

Joseph Douce said:
got it it needs to be nethash not hash:net for 4.5 as per @redhat27's script

Code:

# Create IPSET lists ipset -N WAN0 nethash ipset -N NETFLIX nethash ipset -N AMAZONAWS nethash

Thanks for catching the issue. I have other changes for the script in mind. I'll add support for all versions of ipset on the roadmap.

Joseph Douce · Jun 29, 2018

Xentrk said:
Thanks for catching the issue. I have other changes for the script in mind. I'll add support for all versions of ipset on the roadmap.

final issue

Code:

+ ip rule del prio 9990
+ ip rule add from 0/0 fwmark 0x7000/0x7000 table main prio 9990
Error: argument "0x7000/0x7000" is wrong: fwmark value is invalid
+ iptables -t mangle -D PREROUTING -i br0 -p tcp -m set --set NETFLIX dst,dst -j MARK --set-mark 0x7000/0x7000
+ iptables -t mangle -A PREROUTING -i br0 -p tcp -m set --set NETFLIX dst,dst -j MARK --set-mark 0x7000/0x7000
+ iptables -t mangle -D PREROUTING -i br0 -p tcp -m set --set AMAZONAWS dst,dst -j MARK --set-mark 0x7000/0x7000
+ iptables -t mangle -A PREROUTING -i br0 -p tcp -m set --set AMAZONAWS dst,dst -j MARK --set-mark 0x7000/0x7000
+ basename IPSET_Netflix.sh
+ logger -t (IPSET_Netflix.sh) 28641 Ending IPSET_Netflix.sh... IPSET_Netflix.sh

Xentrk · Jun 29, 2018

Joseph Douce said:

final issue

Code:

+ ip rule del prio 9990
+ ip rule add from 0/0 fwmark 0x7000/0x7000 table main prio 9990
Error: argument "0x7000/0x7000" is wrong: fwmark value is invalid
+ iptables -t mangle -D PREROUTING -i br0 -p tcp -m set --set NETFLIX dst,dst -j MARK --set-mark 0x7000/0x7000
+ iptables -t mangle -A PREROUTING -i br0 -p tcp -m set --set NETFLIX dst,dst -j MARK --set-mark 0x7000/0x7000
+ iptables -t mangle -D PREROUTING -i br0 -p tcp -m set --set AMAZONAWS dst,dst -j MARK --set-mark 0x7000/0x7000
+ iptables -t mangle -A PREROUTING -i br0 -p tcp -m set --set AMAZONAWS dst,dst -j MARK --set-mark 0x7000/0x7000
+ basename IPSET_Netflix.sh
+ logger -t (IPSET_Netflix.sh) 28641 Ending IPSET_Netflix.sh... IPSET_Netflix.sh

Let's try an older version of the fwmark method. Change the 0x7000/0x7000 bitmask fwmark references to the number 8. e.g.

Code:

ip rule add from 0/0 fwmark 8 table main prio 9990

Selective Routing for Netflix

Occasional Visitor

Regular Contributor

Part of the Furniture

Part of the Furniture

Part of the Furniture

Occasional Visitor

Occasional Visitor

Regular Contributor

Part of the Furniture

Occasional Visitor

Part of the Furniture

Occasional Visitor

Part of the Furniture

Occasional Visitor

Part of the Furniture

Occasional Visitor

Occasional Visitor

Part of the Furniture

Occasional Visitor

Part of the Furniture

Similar threads

Similar threads

Support SNBForums w/ Amazon

Sign Up For SNBForums Daily Digest