x3mRouting x3mRouting ~ Selective Routing for Asuswrt-Merlin Firmware (1-Nov-2020)

[Xentrk]

I already tried making Custom 1 and Custom 2 vpn provider DNS addresses as you suggest, but no luck. I tried setting the router DNS to Nord and Express DNS addresses, too, and set DNS filter to router, but just the same result. It doesn't work. Tried Disabled, relaxed. strict....the only thing that works is Explicit. Haven't tried putting the DNS addresses in the policy routing section...I may reload x3m later and give it a try. I'm a bit exhausted by it all at the moment, to be honest. I assume this has been working at some time, with Nord and Express. Or not?
Also, I want to add some aliases into the ssh sessions.....is there a .bashrc file or similar where I can do that?
And what's TOR like. My current internet is 450-500mbps both ways, and I get 120-150mbps with VPN's on (Nord is faster that Express). Does TOR unblock all the streaming sites? I was advised against getting a dedicated IP from Nord about 2 years ago. Someone from Nord support told me they get found out eventually, and won't change the address for one that isn't known. Do you know if that's still an issue?
How does dedicated IP address work? You still VPN to it, or you need your own vpn server on it?

With the TG dedicated IP, you are the only one using it. As a result, streaming media companies don't flag it as a VPN as they do not see 1,000's of connections coming from the same IP. That is my theory anyway. TG recommends that you only use the private IP for streaming due to the hacks they do to make it work. They want you to use one of their shared VPN servers for non-streaming use. I don't believe TOR will be of help.

 

[Brainstorm]

You've confirmed what others have reported about Nord and Express here on the snbforums. Those are the two VPN providers that people always report issues with since they require exclusive use of their DNS to circumvent the VPN blocks. There was a recent thread where some forum users developed a script that may solve the issue. I searched for the thread two days ago but could not find it. I'll keep looking. I will add a note on the README that the dnsmasq method does not work with Nord and Express due to their requirement to exclusively use their DNS.

I don't have issues with TorGuard. I can use any DNS setting. I have recommended it to many of the expats here in the land of smiles and they always come back thanking me for the recommendation. Still, there is going to be the issue of doing the analysis and hoping you get it right. SlingTV is the easiest one I've done. movetv.com is the only domain needed. They have a 7 day free trial period if you want to try it. The dnsmasq method is the only one that requires the use of dnsmasq. When I was new to this, I would collect all of the domain names into a file. I then ran a script that would do an nslookup on each domain and load the IPv4 addresses to the IPSET list. I'll see if I can write a website scraper that will collect the domain names to see if we can apply this approach.

Hi X. thanks for the update. To be honest, doing the analysis was very interesting. I don't mind that kind of problem solving at all. It was not being able to get ANYTHING working that was so frustrating.
eibgrad has been very helpful in explaining some of the intricacies of DNS configuration, but I'm also using Express mediastreamer, which makes things even more difficult.
I'll wait and see if you can find that script.
Are you saying in your earlier post that a TOR private address will NOT be of any help with this issue?

 

[Xentrk]

Hi X. thanks for the update. To be honest, doing the analysis was very interesting. I don't mind that kind of problem solving at all. It was not being able to get ANYTHING working that was so frustrating.
eibgrad has been very helpful in explaining some of the intricacies of DNS configuration, but I'm also using Express mediastreamer, which makes things even more difficult.
I'll wait and see if you can find that script.
Are you saying in your earlier post that a TOR private address will NOT be of any help with this issue?

Glad you are still hanging in there. I first started this journey on DD-WRT. eibgrad has many posts and scripts on the DD-WRT forum that helped me learn when I first got into this. Glad to see him jump on snbforms as he has a lot of knowledge.

I have not heard of anyone using TOR for streaming media services that block known VPNs. I suspect that buffering may be a concern because of all of the hops involved.

I recently saw a solution advertised online that allowed one to choose the streaming service from a menu. I think it was a VPN solution. I will see if I can find it.

pfSense has different methods to handle DNS. All of my VPN tunnels use Cloudflare DNS in the same geo location as the VPN end point without any special configuration. I do have a rule to force clients to use DNS on router. WAN clients use Unbound DNS.

 

[Brainstorm]

A couple of points. It's Merlin, not x3mRouting that doesn't work with Express/Nord vpn's unless Exclusive is set.
I HAVE got both vpn's working by finding the DNS they push when you turn them up, putting that address in the router WAN DNS settings AND as a rule on the destination address side, forcing it down the VPN, and setting openvpn client to DNS "disabled" Thanks to you and eibgrad for that help.
Howver, I may be mistaken, but I think the vpn suppliers DNS addresses are dynamic. I'm pretty sure when I was testing with DNS Filtering, I got different addresses to the ones I got today and used to get the vpn's working with "Disabled". And it does appear that you must use the vpn suppliers DNS address to get them working. I tried with Cloudfare and Google DNS, VPN's connected but services did NOT get unblocked. I only tested the BBC.
So, several issues. Are the DNS addresses dynamic? That causes a problem....
I want to use both NORD and Express, That's both router WAN addresses used up.....
I also use Express Mediastreamer. That needs both router DNS WAN addresses to be set to their special address. Not sure how it works, it connects me to a DNS in the USA, but it unblocks ITV, Netflix, Disney, and it does it at very high speed. No buffering, super-fast connection times, and HDR and 4k with no issues at all. It's VERY good, but it leaks. Or it appears to. Talking to eibgrad, maybe it's not leaking, but vpn's don't unblock if I use it and set "Disabled"
It did unblock Amazon, NOW TV and BBC as well, but they caught them at it September last year and they haven't managed to get any of them working again. I'm not sure how hard they're trying.
Why doesn't Merlin work with Express and Nord?. What's the difference and why the difference between Exclusive and Strict and Relaxed and Disabled. Can I get into those particular scripts to try and alter them? Where are all the important scripts?
Lots of questions, I know. Apologies for that, but now I'm intrigued.....and once again, thanks for your time and effort.....

 

[Xentrk]

A couple of points. It's Merlin, not x3mRouting that doesn't work with Express/Nord vpn's unless Exclusive is set.
I HAVE got both vpn's working by finding the DNS they push when you turn them up, putting that address in the router WAN DNS settings AND as a rule on the destination address side, forcing it down the VPN, and setting openvpn client to DNS "disabled" Thanks to you and eibgrad for that help.
Howver, I may be mistaken, but I think the vpn suppliers DNS addresses are dynamic. I'm pretty sure when I was testing with DNS Filtering, I got different addresses to the ones I got today and used to get the vpn's working with "Disabled". And it does appear that you must use the vpn suppliers DNS address to get them working. I tried with Cloudfare and Google DNS, VPN's connected but services did NOT get unblocked. I only tested the BBC.
So, several issues. Are the DNS addresses dynamic? That causes a problem....
I want to use both NORD and Express, That's both router WAN addresses used up.....
I also use Express Mediastreamer. That needs both router DNS WAN addresses to be set to their special address. Not sure how it works, it connects me to a DNS in the USA, but it unblocks ITV, Netflix, Disney, and it does it at very high speed. No buffering, super-fast connection times, and HDR and 4k with no issues at all. It's VERY good, but it leaks. Or it appears to. Talking to eibgrad, maybe it's not leaking, but vpn's don't unblock if I use it and set "Disabled"
It did unblock Amazon, NOW TV and BBC as well, but they caught them at it September last year and they haven't managed to get any of them working again. I'm not sure how hard they're trying.
Why doesn't Merlin work with Express and Nord?. What's the difference and why the difference between Exclusive and Strict and Relaxed and Disabled. Can I get into those particular scripts to try and alter them? Where are all the important scripts?
Lots of questions, I know. Apologies for that, but now I'm intrigued.....and once again, thanks for your time and effort.....

You will have to ask NORD and Express if their DNS is dynamic. Maybe one of the other NORD and Express users know????? I've not used them myself. Perhaps it is like Cloudflare? I enter 1.1.1.1 in the config. It then uses a DNS close to my geo end point when I set Accept DNS Config to Disabled. When I do an ipleak test, DNS is a different IP than 1.1.1.1. So it may work that way. The issue with Nord and Express has come up often on the forum which is why I am aware of it. Like you, I tried several ways to work around it in the past but was never successful.

With TorGuard, I can use any DNS and I don't get blocked with the private IP. The services only care about my end point IPv4 address. On my pfSense box, I don't do any special config for DNS. I have one rule to force all clients to use the DNS of the router which is Unbound DNS. Clients assigned to the VPN use Cloudflare servers in the geo location of the end point. Sorry I can't help much with the DNS issue. But it sounds like you are closer in getting it to work.

 

[Brainstorm]

I CAN get it working, but it's frustrating I can't get it to do everything I want, and my requirement is restricted by the DNS issue. What I wanted was a configuration that worked without me having to think about it, and constantly make changes to achieve what I want to do. I think x3mRouting would have been perfect, but Merlin is holding it back, particularly, as you say, for Nord and Express users. To be honest, it feels like one step forwards, two steps back....
Can the Merlin stuff be accessed and tampered with, or is that not an option? I have a bit of unix experience.....

 

[Xentrk]

I CAN get it working, but it's frustrating I can't get it to do everything I want, and my requirement is restricted by the DNS issue. What I wanted was a configuration that worked without me having to think about it, and constantly make changes to achieve what I want to do. I think x3mRouting would have been perfect, but Merlin is holding it back, particularly, as you say, for Nord and Express users. To be honest, it feels like one step forwards, two steps back....
Can the Merlin stuff be accessed and tampered with, or is that not an option? I have a bit of unix experience.....

I would just give up on those two providers because of the DNS issue. The topic has come up multiple times on the forum and there is no work around that I am aware of. The only work around that may work is to do what you are doing to get it working with the manual intervention when using the dnsmasq method. After awhile, the IPSET will get populated with all of the IPv4 addresses from accessing all of the functions on the streaming device or web site surfining. The cron job will save the list in /opt/tmp/ directory once per day. You can then go back to Accept DNS Config = Exclusive with Policy Rules once you think you have collected all of the IPv4 addresse. The list will not get any further updates from dnsmasq. The ipset list will get populated from the backup/save file and the iptables rule will still work that controls the routing rule. You can force a save to /opt/tmp using the command

ipset save IPSETNAME > /opt/tmp/IPSETNAME

 

[Brainstorm]

I'm using a Roku streaming stick, with all the TV apps I want on it.
I tried your approach, and it seemed to work.
I created IPSET lists using autoscan for BBC, Britbox, BTSport, Nowtv. I can see these lists using the "liststats" command. I can see the number of entries growing as I access the apps.
Some of the apps on the stick need to go to WAN ie ITV.
So I set the VPN client 5 to push ALL traffic through the WAN (192.168.2.0/24 WAN)
ITV works perfectly. The VPN is connected, but ITV works fine. It's obviously going via the WAN. However, as soon as I create a rule to send, for example BTSPORT down the VPN (ALL 5 BTSPORT autoscan=bt) ITV stops working, it's obviously NOT going down the WAN anymore. If I delete the BTSPORT rule, ITV starts working again. What am I doing something wrong?
And when I create an IPSET list using " x3mRouting ipset_name=BTSPORT autoscan=bt", what have I actually created when to route BTSPORT I must use the command " ALL 5 BTSPORT autoscan=bt"
What exactly am I doing with those 2 commands?

 

[Brainstorm]

I've just tried something else:
I set VPN client 1 to route all traffic via the WAN (192.168.2.0/24 WAN)
I put the Roku stick in VPN 1
ITV works, but although I have a rule for BTSPORT to use VPN 5 (ALL 5 BTSPORT autoscan=bt), as soon as I turn on VPN 1, BTSPORT stops working

 

[Brainstorm]

And I've just tried this......:

"x3mRouting 5 0 NETFLIX dnsmasq=netflix.com,nflxext.com,nflximg.net,nflxso.net,nflxvideo.net"

That's pushing the NETFLIX traffic down the WAN of a NORD vpn set for Exclusive DNS. Or trying to. That doesn't work either.
I'm guessing x3mRouting just doesn't work with NORD or Express, whatever the vpnclient settings are...

 

[Suresh]

@Xentrk i have setup a new pihole this week which effectively broke my selective vpn routing. I have the pihole ip set as DNS server. What could it be?

 

[Brainstorm]

I would just give up on those two providers because of the DNS issue. The topic has come up multiple times on the forum and there is no work around that I am aware of. The only work around that may work is to do what you are doing to get it working with the manual intervention when using the dnsmasq method. After awhile, the IPSET will get populated with all of the IPv4 addresses from accessing all of the functions on the streaming device or web site surfining. The cron job will save the list in /opt/tmp/ directory once per day. You can then go back to Accept DNS Config = Exclusive with Policy Rules once you think you have collected all of the IPv4 addresse. The list will not get any further updates from dnsmasq. The ipset list will get populated from the backup/save file and the iptables rule will still work that controls the routing rule. You can force a save to /opt/tmp using the command

ipset save IPSETNAME > /opt/tmp/IPSETNAME

Just to confirm what you're suggesting here.
If I set, for example, Express vpn to "Disabled", put the express vpn pushed dns address in the WAN settings and in the client rules so the vpn will work when set to disabled, create IPSETS with this configuration, then, eventually, when I think I've collected all the necessary ip addresses, set the VPN settings back to Exclusive, it might work?
Is it the case that x3mRouting WILL work with "Exclusive", it's just that dnsmasq gets ignored so it doesn't collect site data for IPSET, and therefore doesn't build valid IPSET lists, or does x3mRouting simply NOT work when the vpn is set to Exclusive.
I ask because I'm considering having one more go to get it working.....

 

[kernol]

@Xentrk - can I ask you to consider editing your first post in this thread ADD links to ...

Your GitHub page ... https://github.com/Xentrk/x3mRouting

And perhaps to your original version 2 thread .... https://www.snbforums.com/threads/x3mrouting-selective-routing-for-asuswrt-merlin-firmware.57793/

As a noob to the project I had to do a fair amount of digging to find them - must easier for us if the pointers are there

 

[thebatfink]

Hey @Xentrk is the VPN Director functionality going to break this do you think?

 

[Brainstorm]

What's VPN Director?

 

[Kingp1n]

What's VPN Director?

Beta - VPN Director testing

386.3 will introduce a complete overhaul of the OpenVPN routing handling, serving as the platform on top of which is implemented the new VPN Director. Backend changes: Starting with 386.3, all OpenVPN routing will now be directly configured by the firmware itself, rather than having the OpenVPN...

www.snbforums.com

 

[thebatfink]

I obviously have no real understanding what either are doing, but VPN director sounds like it might be doing something similar, maybe they will both coexist OK. But he says it won’t support ipsets and judging by the size of the ip lists I end up with for BBC, Netflix and Amazon, using it I imagine isn’t really going to be feasible.

 

[Brainstorm]

Thanks Kingp1n

 

[jata]

A recent update caused some changes to BBC which required some analysis. So far, I have it working with the method below. I am routing ALL BBC traffic to the "source destination" VPN Client 4. So adjust accordingly. The other item is the AWS method may not be required. I noticed many reply records resolving to AWS EU server and added it during the initial analysis. It may work without it but I have not confirmed.

Install x3mRouting option 3. Then, in a SSH command line copy/paste the following after adjusting the "to destination 4":

sh /jffs/scripts/x3mRouting/x3mRouting.sh ALL 4 AWS_EU aws_region=EU
sh /jffs/scripts/x3mRouting/x3mRouting.sh ALL 4 BBC_ASN asnum=AS2818,AS31459
sh /jffs/scripts/x3mRouting/x3mRouting.sh ALL 4 BBC_WEB1 dnsmasq=2cnt.net,at-o.net,bbc.com,bbcverticals.com,co.uk,dotmetrics.net,net.uk

BBC traffic will have a higher priority than the clients specified in the policy table.

For the dnsmasq method to work, you have to have dnsmasq logging enabled.

@Xentrk - thanks again for all your help a couple of months ago. I have come back to look at this again as I am struggling to get BBC iplayer to work with my existing setup (had your script set to option 2).

So using the setup you propose above, I have reinstalled the script using option 3.

Now I need to turn on dnsmasq logging. How do I check this?
[edit - used dnsmasq -q to start logging]

I also only want to send BBC iplayer traffic through the VPN with everything else normal (no VPN). So how do I setup the VPN so that only iplayer goes to VPN and everything else is not through the VPN?
[edit - have set VPN client with exclusive DNS and strict policy for forcing traffic through tunnel]

Also i have set the routing using the sh commands above (but changed to correct VPN client.

Below are my logs whan trying to play bbc video that is geolocked...

Any help appreciated

Jun 29 13:09:09 dnsmasq[28421]: query[A] www.bbc.co.uk from 192.168.100.249
Jun 29 13:09:09 dnsmasq[28421]: forwarded www.bbc.co.uk to 127.0.0.1
Jun 29 13:09:09 dnsmasq[28421]: query[A] iplayer-web.files.bbci.co.uk from 192.168.100.249
Jun 29 13:09:09 dnsmasq[28421]: forwarded iplayer-web.files.bbci.co.uk to 127.0.0.1
Jun 29 13:09:09 dnsmasq[28421]: query[A] www.bbc.co.uk from 192.168.100.249
Jun 29 13:09:09 dnsmasq[28421]: forwarded www.bbc.co.uk to 127.0.0.1
Jun 29 13:09:09 dnsmasq[28421]: query[A] iplayer-web.files.bbci.co.uk from 192.168.100.249
Jun 29 13:09:09 dnsmasq[28421]: forwarded iplayer-web.files.bbci.co.uk to 127.0.0.1
Jun 29 13:09:10 dnsmasq[28421]: reply iplayer-web.files.bbci.co.uk is <CNAME>
Jun 29 13:09:10 dnsmasq[28421]: reply iplayer-web.files.bbci.co.uk.edgekey.net is <CNAME>
Jun 29 13:09:10 dnsmasq[28421]: ipset add BBC_WEB1 104.74.37.53 e3891.dscf.akamaiedge.net
Jun 29 13:09:10 dnsmasq[28421]: reply e3891.dscf.akamaiedge.net is 104.74.37.53
Jun 29 13:09:10 dnsmasq[28421]: reply www.bbc.co.uk is <CNAME>
Jun 29 13:09:10 dnsmasq[28421]: reply www.bbc.co.uk.pri.bbc.co.uk is <CNAME>
Jun 29 13:09:10 dnsmasq[28421]: ipset add BBC_WEB1 212.58.233.251 uk.www.bbc.co.uk.pri.bbc.co.uk
Jun 29 13:09:10 dnsmasq[28421]: reply uk.www.bbc.co.uk.pri.bbc.co.uk is 212.58.233.251
Jun 29 13:09:10 dnsmasq[28421]: ipset add BBC_WEB1 212.58.237.251 uk.www.bbc.co.uk.pri.bbc.co.uk
Jun 29 13:09:10 dnsmasq[28421]: reply uk.www.bbc.co.uk.pri.bbc.co.uk is 212.58.237.251
Jun 29 13:09:11 dnsmasq[28421]: query[A] europe-west1-bbc-otg-traf-mgr-bq-prod-4591.cloudfunctions.net from 192.168.100.249
Jun 29 13:09:11 dnsmasq[28421]: forwarded europe-west1-bbc-otg-traf-mgr-bq-prod-4591.cloudfunctions.net to 127.0.0.1
Jun 29 13:09:11 dnsmasq[28421]: query[A] europe-west1-bbc-otg-traf-mgr-bq-prod-4591.cloudfunctions.net from 192.168.100.249
Jun 29 13:09:11 dnsmasq[28421]: forwarded europe-west1-bbc-otg-traf-mgr-bq-prod-4591.cloudfunctions.net to 127.0.0.1
Jun 29 13:09:11 dnsmasq[28421]: reply europe-west1-bbc-otg-traf-mgr-bq-prod-4591.cloudfunctions.net is 216.239.36.54

 

[ToasterPC]

Hello there!

Been using x3mRouting successfully for about a week now, but I can't seem to be able to wrap my head around something.

Tunneling through two VPN clients pointing to PIA in the US and Spain I'm able to route different services through the respectively desired routes, but after either rebooting the router or seemingly by chance, the router's IP address will begin to report as if tunneling all traffic through either of the connected tunnels.

I was wondering if there would be a way to route everything to the WAN and just use the IPSET rules as exceptions (say, route all Amazon-owned addresses to the US OVPN Client, and leave everything else to be routed by the WAN).

So far I've attempted to set a Policy Rule to route both SRC 0.0.0.0 and DST 0.0.0.0 to WAN, then setting the IPSET rules as asnum and calling them from the OVPN client GUI, but sites still report being able to see my WAN's IP address when done this way.

Is the solution I'm looking for out of scope, or am I doing something wrong perhaps?

Thanks for the help!