x3mRouting x3mRouting ~ Selective Routing for Asuswrt-Merlin Firmware (1-Nov-2020)

[Xentrk]

Once again, thanks for your time....
So the records I acquired by NOT going through the tunnel are valid, but won't work without DNS filtering, is that correct?
And it's not obvious which DNS server to configure in the filtering section. Can I put anything I like in there and it will work? The same DNS that I use for my wan connections? Google DNS?
How does using filtering stop the leaks?

LAN Client devices often have a default DNS configured which takes precedence over router DNS settings. DNS Filter prevents this and forces all LAN clients to use the DNS specified on the WAN page, other DNS providers, or a custom DNS (VPN provider DNS) by device MAC address. You can experiment with the options and use the ipleak.net or dnsleak.com website to get familiar with the settings. Below is the basic setting that forces all LAN clients to use DNS specified on WAN page.

I think an iptables rule gets created for DNS Filter. It has been awhile I looked at it. There was a recent thread where some of the x3mRouting users put together a script for some additional controls. I will go back and look at it tomorrow. It was a feature I was thinking of adding.

Accept DNS Configuration=Strict mode does not create any iptable rules. It merely appends the DNS servers to dnsmasq, and sets the "strict" parameter in dnsmasq which determines the order used when querying servers.

 

[Brainstorm]

LAN Client devices often have a default DNS configured which takes precedence over router DNS settings. DNS Filter prevents this and forces all LAN clients to use the DNS specified on the WAN page, other DNS providers, or a custom DNS (VPN provider DNS) by device MAC address. You can experiment with the options and use the ipleak.net or dnsleak.com website to get familiar with the settings. Below is the basic setting that forces all LAN clients to use DNS specified on WAN page.


View attachment 33973

I think an iptables rule gets created for DNS Filter. It has been awhile I looked at it. There was a recent thread where some of the x3mRouting users put together a script for some additional controls. I will go back and look at it tomorrow. It was a feature I was thinking of adding.

Accept DNS Configuration=Strict mode does not create any iptable rules. It merely appends the DNS servers to dnsmasq, and sets the "strict" parameter in dnsmasq which determines the order used when querying servers.

Thanks X. Again.
I think I'm making SOME progress.
What about traffic I want to go straight to the WAN. How do I deal with that?
If I switch on a vpn, all my traffic seems to go down it. I've looked at "VPN Client Bypass Routing", but nothing seems to fit. I've got NOWTV working down client 4, and that's configured NOT to block traffic if the vpn is down. If I shut the vpn, traffic goes straight to the WAN, and NOWTV reports a geolocation error, which is what I'd expect, but I'm having lots of trouble routing 2 other services direct to the WAN. Any advice for that?

 

[Background]

I posted OpenVPN Client instructions that may help on my blog site. I need to update it but it will give you an overview.

Thank you! Will check it out now. Trying to use NoLagVPN for PS5. I noticed Diversion mentioned so will check that out too. I'm a noob so easy guides help! LOL.

 

[Brainstorm]

LAN Client devices often have a default DNS configured which takes precedence over router DNS settings. DNS Filter prevents this and forces all LAN clients to use the DNS specified on the WAN page, other DNS providers, or a custom DNS (VPN provider DNS) by device MAC address. You can experiment with the options and use the ipleak.net or dnsleak.com website to get familiar with the settings. Below is the basic setting that forces all LAN clients to use DNS specified on WAN page.


View attachment 33973

I think an iptables rule gets created for DNS Filter. It has been awhile I looked at it. There was a recent thread where some of the x3mRouting users put together a script for some additional controls. I will go back and look at it tomorrow. It was a feature I was thinking of adding.

Accept DNS Configuration=Strict mode does not create any iptable rules. It merely appends the DNS servers to dnsmasq, and sets the "strict" parameter in dnsmasq which determines the order used when querying servers.

No. Turns out none of the routing is working. I've deleted all the IPSETS I've created, I've got the vpn working with STRICT, not EXPLICIT, no DNS leaks. NOW TV works fine across the vpn. I use the command "x3mRouting ALL 3 NOWTV autoscan=nowtv" I can see the IPSET file using "liststats". I can see 3 IPSET entries and 12 FQDN entries in the using "sh autoscan.sh scan=nowtv". NOWTV still works, but if I shut the vpn 3 and open vpn 2, NOWTV works again, obviously down vpn 2, so it looks like the routing is NOT forcing NOWTV down VPN 3. Am I missing something? I have static ip addresses configured under DHCP for TV and Roku stick, both of these are in both VPN2 and VPN3 client rules. Is THAT correct?

 

[Brainstorm]

Thank you! Will check it out now. Trying to use NoLagVPN for PS5. I noticed Diversion mentioned so will check that out too. I'm a noob so easy guides hel

Thanks X. Again.
I think I'm making SOME progress.
What about traffic I want to go straight to the WAN. How do I deal with that?
If I switch on a vpn, all my traffic seems to go down it. I've looked at "VPN Client Bypass Routing", but nothing seems to fit. I've got NOWTV working down client 4, and that's configured NOT to block traffic if the vpn is down. If I shut the vpn, traffic goes straight to the WAN, and NOWTV reports a geolocation error, which is what I'd expect, but I'm having lots of trouble routing 2 other services direct to the WAN. Any advice for that?

I took a look at your blog, but now I'm not sure if John's fork is the answer to my problems, or the "Stubby" fix you mention underneath it.

 

[Xentrk]

Thanks X. Again.
I think I'm making SOME progress.
What about traffic I want to go straight to the WAN. How do I deal with that?
If I switch on a vpn, all my traffic seems to go down it. I've looked at "VPN Client Bypass Routing", but nothing seems to fit. I've got NOWTV working down client 4, and that's configured NOT to block traffic if the vpn is down. If I shut the vpn, traffic goes straight to the WAN, and NOWTV reports a geolocation error, which is what I'd expect, but I'm having lots of trouble routing 2 other services direct to the WAN. Any advice for that?

What traffic do you want to route to WAN? VPN Client Bypass Routing is the correct method to use. You can specify any VPN client and the source 0 for WAN as the destination. Most people use it to bypass the VPN for exceptions. For example, they route all LAN traffic thru the VPN client using CIDR notation (192.168.1.0/24) but need to make an exception for Netflix.

This will report the WAN IP when using the site whatismyip.com:

x3mRouting 1 0 WMYIP dnsmasq=whatismyip.com

For the other services, you have to perform the analysis as to what domains the website is using. Sometimes it is very easy and other times it takes effort. It took me a week of on and off analysis to figure out BBC iPlayer. I used the utility scripts in x3mRouting to figure it out. For SlingTV, all of the domain names used the word "movetv.com". So that one was easy. If I had tried to use sling.com, it never would have worked. The utility scripts report back what is going on in dnsmasq.log to help with the analysis. I also follow the dnsmasq.log file as I perform the analysis to locate other domain name search terms. You can use the follow the log file in diversion or issue the command tail -f /opt/var/log/dnsmasq.log

Netflix is another good example. netflix.com only gets you some of the high level domain names. I had to follow the log file and watch it to see that nflx.com was also being queried.

I did a high level look at the domains used by nowtv.com. I do see references for sky.com. So you may need to include it as well.

 

[Xentrk]

No. Turns out none of the routing is working. I've deleted all the IPSETS I've created, I've got the vpn working with STRICT, not EXPLICIT, no DNS leaks. NOW TV works fine across the vpn. I use the command "x3mRouting ALL 3 NOWTV autoscan=nowtv" I can see the IPSET file using "liststats". I can see 3 IPSET entries and 12 FQDN entries in the using "sh autoscan.sh scan=nowtv". NOWTV still works, but if I shut the vpn 3 and open vpn 2, NOWTV works again, obviously down vpn 2, so it looks like the routing is NOT forcing NOWTV down VPN 3. Am I missing something? I have static ip addresses configured under DHCP for TV and Roku stick, both of these are in both VPN2 and VPN3 client rules. Is THAT correct?

I think the error is there are more domain names that you need to specify in addition to nowtv. I saw sky.com listed in the web page source. Please see my comments in the prior post about performing the analysis.

 

[Brainstorm]

What traffic do you want to route to WAN? VPN Client Bypass Routing is the correct method to use. You can specify any VPN client and the source 0 for WAN as the destination. Most people use it to bypass the VPN for exceptions. For example, they route all LAN traffic thru the VPN client using CIDR notation (192.168.1.0/24) but need to make an exception for Netflix.

This will report the WAN IP when using the site whatismyip.com:

x3mRouting 1 0 WMYIP dnsmasq=whatismyip.com

For the other services, you have to perform the analysis as to what domains the website is using. Sometimes it is very easy and other times it takes effort. It took me a week of on and off analysis to figure out BBC iPlayer. I used the utility scripts in x3mRouting to figure it out. For SlingTV, all of the domain names used the word "movetv.com". So that one was easy. If I had tried to use sling.com, it never would have worked. The utility scripts report back what is going on in dnsmasq.log to help with the analysis. I also follow the dnsmasq.log file as I perform the analysis to locate other domain name search terms. You can use the follow the log file in diversion or issue the command tail -f /opt/var/log/dnsmasq.log

Netflix is another good example. netflix.com only gets you some of the high level domain names. I had to follow the log file and watch it to see that nflx.com was also being queried.

I did a high level look at the domains used by nowtv.com. I do see references for sky.com. So you may need to include it as well.

So, just to clarify, I'm only looking for high-level domain names, not FQDN entries. Is that correct?
And I've found that if I use the DNS filter to stop the DNS leaks when the vpn is set to anything other than explicit, nothing goes to the dnsmasq log. I think that may be a part of my problem.
So NOWTV won't connect and work, but I DO see output in the file.
If I analyse with DNS filter off, get my high-level domains from the log output, put them in the IPSET file and turn ON DNS filtering so the apps can connect, will it work?

 

[Brainstorm]

This is part of the output from the dnsmasq log when I connect to Britbox. Which entries should I be using, which can be ignored?
Should I be using only xxx.yyy domain names? Or aaa.bbb.ccc.ddd names?

 

[Brainstorm]

This is part of the output from the dnsmasq log when I connect to Britbox. Which entries should I be using, which can be ignored?
Should I be using only xxx.yyy domain names? Or aaa.bbb.ccc.ddd names?

 

[Xentrk]

This is part of the output from the dnsmasq log when I connect to Britbox. Which entries should I be using, which can be ignored?
Should I be using only xxx.yyy domain names? Or aaa.bbb.ccc.ddd names?

You want to use the high level names under the IPSET Format heading. These are "query" records. The domains return by the reply records don't need to be included. The IPSET feature of dnsmasq handles that.

The reason for listing the FQDN in the report is for a sanity check. For example, I was recently performing some analysis and an unrelated domain name matched the 3 characters I was searching for. Seeing the FQDN helped me notice it did not belong. Here is an example for HBO.

sh autoscan.sh scan=hbo

IPSET Format
-------------------------------------
hbo.com
hbomax.com
warnermediacdn.com

FQDN Format
-------------------------------------
artist.api.cdn.hbo.com
comet.api.hbo.com
commerce.api.hbo.com
dash.pro42.akm.cdn.hbomax.com
dash.pro42.cf.cdn.hbomax.com
hbomax-images.warnermediacdn.com
hbomax.com
markers.api.hbo.com
media.fly.cdn.hbomax.com
telegraph.api.hbo.com
www.hbomax.com

Following is what I ended up with for BBC iPlayer. I combined several methods. When I did the analysis, I noticed that many of the reply records were AWS domains in the EU region. I also used the search feature on Hurricane Electric to see what ASN BBC belonged to and included that. The top level domain names were gathered using the getdomainnames.sh and autoscan.sh scripts.

sh /jffs/scripts/x3mRouting/x3mRouting.sh ALL 3 AWS-EU aws_region=EU
sh /jffs/scripts/x3mRouting/x3mRouting.sh ALL 3 BBC_ASN asnum=AS2818,AS31459
sh /jffs/scripts/x3mRouting/x3mRouting.sh ALL 3 BBC_WEB1 dnsmasq=2cnt.net,at-o.net,bbc.com,bbcverticals.com,co.uk,dotmetrics.net,net.uk

cdn.britbox.co.uk would be entered as co.uk when using the dnsmasq method e.g. dnsmasq=co.uk

 

[Laxarus]

Does anyone having a similar issue as mine?
I use dnsmasq_file method.
Everything works fine. However, after reboot my custom dnsmasq entries disappear.

I think x3mRouting is rewriting the dnsmasq.conf.add file and my own custom dns entries get deleted.

Here is my dnsmasq.conf.add file and the last line is the custom dns entries ı manually entered.

log-async
log-queries
log-facility=/opt/var/log/dnsmasq.log
ipset=/domain1/domain2/domain3/GULIBU
address=/domain1/domain2/domain3/192.168.1.45

 

[Xentrk]

Does anyone having a similar issue as mine?
I use dnsmasq_file method.
Everything works fine. However, after reboot my custom dnsmasq entries disappear.

I think x3mRouting is rewriting the dnsmasq.conf.add file and my own custom dns entries get deleted.

Here is my dnsmasq.conf.add file and the last line is the custom dns entries ı manually entered.

log-async
log-queries
log-facility=/opt/var/log/dnsmasq.log
ipset=/domain1/domain2/domain3/GULIBU
address=/domain1/domain2/domain3/192.168.1.45

I added your entries to my dnsmasq.conf.add followed by a reboot. No changed after the reboot.

I suspect you have another user script or postconf script (dnsmasq.posconf) that is modifying the file.

This will list out the files that contain the word dnsmasq.conf.add in the /jffs/scripts directory

grep -l dnsmasq.conf.add /jffs/scripts/*.*

 

[Laxarus]

I added your entries to my dnsmasq.conf.add followed by a reboot. No changed after the reboot.

I suspect you have another user script or postconf script (dnsmasq.posconf) that is modifying the file.

This will list out the files that contain the word dnsmasq.conf.add in the /jffs/scripts directory

grep -l dnsmasq.conf.add /jffs/scripts/*.*

There is no other script.
I updated the script and tried again. There is no issue now. I guess the previous version was causing this issue.

 

[Xentrk]

There is no other script.
I updated the script and tried again. There is no issue now. I guess the previous version was causing this issue.

Nothing changed in the new version in how dnsmasq.conf.add gets updated. No other reports of the issue have been reported since the major rewrite last summer. If it happens again, look in the system log for clues.

 

[Brainstorm]

I've spent literally hours and hours trying to get this working. Nothing works.
So, I've deleted x3mRouting.
I've got one express vpn running. I've got my Roku stick connected to that vpn. In the "Rules for routing client traffic through the tunnel" window, I've set it so ALL Roku traffic goes via the WAN. "Accept DNS Configuration" is set to Explicit. It works as expected. All Roku traffic goes via the WAN.
However, x3mRouting will not work with DNS set to explicit.
As soon as I set DNS to strict, and turn on DNS filtering in the LAN configuration, everything stops working. Looks like all traffic goes down the vpn. That's without x3mRouting installed. I've tried all the DNS servers offered in that window. None of them work.
In custom configuration, I've tried the nordvpn dns ip address. That sort of works, but traffic is still obviously going to the vpn, not the wan.
What am I missing here? It's not the x3mRouting, it's more fundamental than that.
What setting should I be using in the DNS filtering section to get "DNS Strict" working? Just basically, with the one vpn and Roku stick using the WAN? I think once I have THAT working, x3mRouting will also work.

 

[Xentrk]

I've spent literally hours and hours trying to get this working. Nothing works.
So, I've deleted x3mRouting.
I've got one express vpn running. I've got my Roku stick connected to that vpn. In the "Rules for routing client traffic through the tunnel" window, I've set it so ALL Roku traffic goes via the WAN. "Accept DNS Configuration" is set to Explicit. It works as expected. All Roku traffic goes via the WAN.
However, x3mRouting will not work with DNS set to explicit.
As soon as I set DNS to strict, and turn on DNS filtering in the LAN configuration, everything stops working. Looks like all traffic goes down the vpn. That's without x3mRouting installed. I've tried all the DNS servers offered in that window. None of them work.
In custom configuration, I've tried the nordvpn dns ip address. That sort of works, but traffic is still obviously going to the vpn, not the wan.
What am I missing here? It's not the x3mRouting, it's more fundamental than that.
What setting should I be using in the DNS filtering section to get "DNS Strict" working? Just basically, with the one vpn and Roku stick using the WAN? I think once I have THAT working, x3mRouting will also work.

If it makes you feel better, I have not been able to figure out Paramount+ (formely CBS All Access) despite my best effort. Lot's of CDN stuff going on. I'm going to switch streaming devices to see if I can get a better handle on where it is failing.

NordVPN and ExpressVPN have the most issue with selective routing as it appears that those providers require that you use their DNS to get around the VPN blocks. I don't have that issue with TorGuard dedicated IP. I can use any DNS.

For the DNS Filter, you can configure Custom 1 and Custom 2 DNS to be the DNS of VPN provider. In the section below, enter the device mac address and add an entry for Custom DNS 1. Repeat for Custom DNS2. Another idea is to put the DNS IP addresses in the Policy Routing section and assign it to use the VPN tunnel. May have to experiment with the Accept DNS Configuration settings. Maybe set to Disabled? Been awhile since I tried this.

 

[Brainstorm]

I already tried making Custom 1 and Custom 2 vpn provider DNS addresses as you suggest, but no luck. I tried setting the router DNS to Nord and Express DNS addresses, too, and set DNS filter to router, but just the same result. It doesn't work. Tried Disabled, relaxed. strict....the only thing that works is Explicit. Haven't tried putting the DNS addresses in the policy routing section...I may reload x3m later and give it a try. I'm a bit exhausted by it all at the moment, to be honest. I assume this has been working at some time, with Nord and Express. Or not?
Also, I want to add some aliases into the ssh sessions.....is there a .bashrc file or similar where I can do that?
And what's TOR like. My current internet is 450-500mbps both ways, and I get 120-150mbps with VPN's on (Nord is faster that Express). Does TOR unblock all the streaming sites? I was advised against getting a dedicated IP from Nord about 2 years ago. Someone from Nord support told me they get found out eventually, and won't change the address for one that isn't known. Do you know if that's still an issue?
How does dedicated IP address work? You still VPN to it, or you need your own vpn server on it?

 

[Brainstorm]

If it makes you feel better, I have not been able to figure out Paramount+ (formely CBS All Access) despite my best effort. Lot's of CDN stuff going on. I'm going to switch streaming devices to see if I can get a better handle on where it is failing.

NordVPN and ExpressVPN have the most issue with selective routing as it appears that those providers require that you use their DNS to get around the VPN blocks. I don't have that issue with TorGuard dedicated IP. I can use any DNS.

For the DNS Filter, you can configure Custom 1 and Custom 2 DNS to be the DNS of VPN provider. In the section below, enter the device mac address and add an entry for Custom DNS 1. Repeat for Custom DNS2. Another idea is to put the DNS IP addresses in the Policy Routing section and assign it to use the VPN tunnel. May have to experiment with the Accept DNS Configuration settings. Maybe set to Disabled? Been awhile since I tried this.

OK, so I've pretty much tried everything, and nothing works. Only "Exclusive" works with Nord and Express. Nothing else. DNS reverts to whatever the router is configured for, and leaks.
Why does x3m NOT work with Exclusive? The help files say "Strict, Add to list but use in the order specified" How does the order get specified? Obviously the VPN provider pushed DNS is NOT top of the list. Can that be changed? Can your scripts be changed to accomodate any of this and make it work?
I've spent a LONG time talking to Express VPN support, with little result, unfortunately. They're pointing the finger at Merlin.
You know that most people that want to use x3mRouting are using are going to use NORD or Express. There surely MUST be a way to make this work correctly.
I asked NORD (again) about a permanent ip address. They say they don't recommend it for streaming, and won't support it if it gets blocked.

 

[Xentrk]

OK, so I've pretty much tried everything, and nothing works. Only "Exclusive" works with Nord and Express. Nothing else. DNS reverts to whatever the router is configured for, and leaks.
Why does x3m NOT work with Exclusive? The help files say "Strict, Add to list but use in the order specified" How does the order get specified? Obviously the VPN provider pushed DNS is NOT top of the list. Can that be changed? Can your scripts be changed to accomodate any of this and make it work?
I've spent a LONG time talking to Express VPN support, with little result, unfortunately. They're pointing the finger at Merlin.
You know that most people that want to use x3mRouting are using are going to use NORD or Express. There surely MUST be a way to make this work correctly.
I asked NORD (again) about a permanent ip address. They say they don't recommend it for streaming, and won't support it if it gets blocked.

You've confirmed what others have reported about Nord and Express here on the snbforums. Those are the two VPN providers that people always report issues with since they require exclusive use of their DNS to circumvent the VPN blocks. There was a recent thread where some forum users developed a script that may solve the issue. I searched for the thread two days ago but could not find it. I'll keep looking. I will add a note on the README that the dnsmasq method does not work with Nord and Express due to their requirement to exclusively use their DNS.

I don't have issues with TorGuard. I can use any DNS setting. I have recommended it to many of the expats here in the land of smiles and they always come back thanking me for the recommendation. Still, there is going to be the issue of doing the analysis and hoping you get it right. SlingTV is the easiest one I've done. movetv.com is the only domain needed. They have a 7 day free trial period if you want to try it. The dnsmasq method is the only one that requires the use of dnsmasq. When I was new to this, I would collect all of the domain names into a file. I then ran a script that would do an nslookup on each domain and load the IPv4 addresses to the IPSET list. I'll see if I can write a website scraper that will collect the domain names to see if we can apply this approach.