Exclude website from being routed to VPN (policy based routing)

[machinist]

@eibgrad Absolutely incredible work, and what a writeup. Very detailed. This looks to be exactly what I've been searching for.

Couple of questions:

1)

I've loaded in the script in the router and added my own domains to the list. Does it still work with the VPN Director? I'm currently passing all traffic from a local IP/device thru OVPN1 - is that still going to work properly when I'm also entering that IP into:

# source ip(s)/network(s) to be routed to destination domains
SOURCES='
{my_device}
'

?

Or would the VPN need to be enabled "globally"?

2)

The way I understand the use of IPSET is that I enter the top level domain, e.g. netflix.com and it also graps all subdomain IP's? Is that right?

I ask because s1.netflix.com may be actually serving the content and not "netflix.com", which may have different IP's -- so do I need to input both of these, or will netflix.com suffice? How does it know about the unlimited possibilities of available content-serving subdomains?

3)

How do you find the content serving domains -- is it still fine to use @Xentrk 's "getdomainnames.sh" or is there a more appropriate way for what we're trying to do? Mainly asking because the domains I've tested with still doesn't pass the "your IP is foreign" test.

Thank you man, I really appreciate your time.

 

[eibgrad]

1)

I've loaded in the script in the router and added my own domains to the list. Does it still work with the VPN Director? I'm currently passing all traffic from a local IP/device thru OVPN1 - is that still going to work properly when I'm also entering that IP into:

# source ip(s)/network(s) to be routed to destination domains
SOURCES='
{my_device}
'

?

Or would the VPN need to be enabled "globally"?

The script is compatible w/ the VPN Director, even dual WAN. The script creates its own ip rule that has higher priority than any other IP rules, whether they come from any of the OpenVPN clients, or even routing policy on the dual WAN. Once installed and operational, you can dump the ip rules database and see for yourself (those listed first (lower numbered) have higher priority).

ip rule show

2)

The way I understand the use of IPSET is that I enter the top level domain, e.g. netflix.com and it also graps all subdomain IP's? Is that right?

I ask because s1.netflix.com may be actually serving the content and not "netflix.com", which may have different IP's -- so do I need to input both of these, or will netflix.com suffice? How does it know about the unlimited possibilities of available content-serving subdomains?

I use IPSET to create a hash table that stores the IP addresses. I then configure DNSMasq to use that hash table to store the name resolution from those domains you've specified in the script. How those domain names are interpreted is NOT a function of the script, but DNSMasq itself.

DNSMasq only matches based on the entire subpart of a domain name, reading from right to left. For example, if you specify somedomain.com, then that domain and ALL its subdomains (e.g., xyz.somedomain.com) will be included. However, xyzsomedomain.com will NOT. Nor will it match somedomain.com.us.

3)

How do you find the content serving domains -- is it still fine to use @Xentrk 's "getdomainnames.sh" or is there a more appropriate way for what we're trying to do? Mainly asking because the domains I've tested with still doesn't pass the "your IP is foreign" test.

That's one problem I leave up to YOU. It wasn't my intention to create the kind of broad features offered by x3mRouting. I'm only trying to provide a limited amount of functionality to get the primary job done. That makes it easier to support and prevent bugs. And (hopefully) easier for the end-user to understand and debug (if necessary). So use whatever tools you find available. I know others w/ the same needs have created these kinds of lists, and the default domain names I provide in the script are just ones I know are common to Netflix, based on what others have told me, and my own testing.

 

[machinist]

Amazing, thanks so much for this, it now works for me.

I had to manually execute the firewall-start script -- no idea if that's obvious, but it didn't work otherwise.

That's one problem I leave up to YOU. It wasn't my intention to create the kind of broad features offered by x3mRouting. I'm only trying to provide a limited amount of functionality to get the primary job done. That makes it easier to support and prevent bugs. And (hopefully) easier for the end-user to understand and debug (if necessary). So use whatever tools you find available. I know others w/ the same needs have created these kinds of lists, and the default domain names I provide in the script are just ones I know are common to Netflix, based on what others have told me, and my own testing.

Got it. I was only asking in case you had some other way of doing it, but I got it. The domains I've gathered seem to work -- at least not getting the "your location is foreign" message any longer.

 

[eibgrad]

@machinist

One other thing worth mentioning (I had forgotten about it, it's been some time since I developed the script).

If you uncomment the following line in the script ...

#LOG_QUERIES= # uncomment/comment to enable/disable logging of dns queries

.. then DNSMasq wil dump details about the individual queries being made to DNSMasq to the syslog, including who made the request, the domain name, etc. In that sense, you can use the script to help you find the relevant domains being used by your apps.

 

[eibgrad]

I had to manually execute the firewall-start script -- no idea if that's obvious, but it didn't work otherwise.

Not sure what that's about. It should work. Unless you're NOT using a routed config, but a bridged config (e.g., AP mode). Bridged configs do NOT call the firewall script because the firewall is only relevant when you have a WAN.

 

[machinist]

I'm not

 

[machinist]

@machinist

One other thing worth mentioning (I had forgotten about it, it's been some time since I developed the script).

If you uncomment the following line in the script ...

#LOG_QUERIES= # uncomment/comment to enable/disable logging of dns queries

.. then DNSMasq wil dump details about the individual queries being made to DNSMasq to the syslog, including who made the request, the domain name, etc. In that sense, you can use the script to help you find the relevant domains being used by your apps.

Thanks, I'll play around with it.

 

[eibgrad]

As a sanity check, I installed the script on 386.4 (just to make sure something didn't change since I last used it w/ 386.3_2), and I can clearly see the firewall-start script being called at bootup (at least if you have DEBUG enabled in the script, which is the default).

cat /tmp/syslog.log | grep firewall-start

Are you using any other scripts that might use the same firewall-start event (Diversion, Skynet, YazFi, whatever). Sometimes if you install those *after* mine, they'll end up preventing my script from getting called. My script won't even install if the file (firewall-start) already exists.

 

[machinist]

Yes I am, I have Skynet installed. I actually just came back to the thread to say this, as I was looking over the various config files.

Sometimes if you install those *after* mine, they'll end up preventing my script from getting called. My script won't even install if the file (firewall-start) already exists.

I removed it before installing yours, as, as you point out, it complained that it already existed. I had tried installing yesterday before installing Skynet, so just thought it was remnant.

So how do I get it to work WITH Skynet, is that not possible? Or do I have to manually input Skynet config into firewall-start after installing yours?

 

[machinist]

As a sanity check, I installed the script on 386.4 (just to make sure something didn't change since I last used it w/ 386.3_2), and I can clearly see the firewall-start script being called at bootup (at least if you have DEBUG enabled in the script, which is the default).

cat /tmp/syslog.log | grep firewall-start

Are you using any other scripts that might use the same firewall-start event (Diversion, Skynet, YazFi, whatever). Sometimes if you install those *after* mine, they'll end up preventing my script from getting called. My script won't even install if the file (firewall-start) already exists.

@eibgrad Sorry for beating a dead horse, but just want to confirm:

Are you saying your script won't work if Diversion/Skynet is already installed?

I had it working so-so before. A bunch of smaller glitches, so after flashing 384.5 just now, I did the factory reset according to L&LD's guide. Really just want the clean slate, so now I'm just trying to get the best setup possible, and that includes your script and Skynet, at least.

Thank you

 

[eibgrad]

@eibgrad Sorry for beating a dead horse, but just want to confirm:

Are you saying your script won't work if Diversion/Skynet is already installed?

I had it working so-so before. A bunch of smaller glitches, so after flashing 384.5 just now, I did the factory reset according to L&LD's guide. Really just want the clean slate, so now I'm just trying to get the best setup possible, and that includes your script and Skynet, at least.

Thank you

I'm NOT saying it won't work. I'm saying that it will NOT automatically install itself if there's something already installed that's using the same event script. That's a common problem w/ the event handling. If two or more scripts want to use a given event script, you then need to resolve the differences *manually*. I can't do it automatically w/o the risk of corrupting the other script that's using the same event(s). I know some authors will do so anyway at some risk, but *I* refuse to do it.

 

[machinist]

Ahh OK, got it. I'll try and edit manually. Thank you again for the swift reply, really do appreciate it.

 

[eibgrad]

Ahh OK, got it. I'll try and edit manually. Thank you again for the swift reply, really do appreciate it.

P.S. TIP: If it was *me*, I'd be inclined to create an third script based on the event, then have the other scripts called from it, rather than trying to merge one script into the other. In the latter case, that can quickly degenerate as more scripts vie for access to the same event.

In fact, I wish we could get the various authors to agree on such a convention regarding ALL user scripting so we could actually automate the process of allowing access to the same events.

 

[machinist]

P.S. TIP: If it was *me*, I'd be inclined to create an third script based on the event, then have the other scripts called from it, rather than trying to merge one script into the other. In the latter case, that can quickly degenerate as more scripts vie for access to the same event.

Sadly above my pay grade, mate. Merging will have to do

 

[machinist]

@eibgrad

I've now gotten it up and running and seems to work.

Have a few follow ups if youre up for it:

- What I ended up doing was backing up 2 of the 3 files that already existed, your script installs (dnsmasq.conf.add and firewall-start). These already existed because of Skynet and x3mRouting scripts.

I then deleted these to allow your script to install. After it installed and added the files, I pasted in the content of the two files from the backups at the very top of the files. Then rebooted from GUI.

Again, everything seems to work, including Skynet and yours, but my question is:

Is this an appropriate way of installing? Will this work, or is there something I haven't considered, like any special permissions incorrectly setup when pasting in like this, or something else?

- As for the IPSETs it produces, it lists then like this:

ipset=/ipchicken.com/netflix.com/nflxext.com/nflximg.net/nflxso.net/lan2wan

It always lists them in 5's, then a new row, and 5 again. Why is that?

When I edit the file going forward, is that how I should do it, or is it fine to make it one big line?

 

[eibgrad]

- What I ended up doing was backing up 2 of the 3 files that already existed, your script installs (dnsmasq.conf.add and firewall-start). These already existed because of Skynet and x3mRouting scripts.

I then deleted these to allow your script to install. After it installed and added the files, I pasted in the content of the two files from the backups at the very top of the files. Then rebooted from GUI.

Again, everything seems to work, including Skynet and yours, but my question is:

Is this an appropriate way of installing? Will this work, or is there something I haven't considered, like any special permissions incorrectly setup when pasting in like this, or something else?

This gets back to the point I was trying to make previously.

It can sometimes be tricky to merge scripts. It just depends on the particular scripts. That's why I *prefer* to take a different approach that minimizes the possibility of mistakes. For example, in the case of a pre-existing firewall-start script, it would be *better* to simply rename that file (e.g., firewall-start1), install the second script and rename it too (e.g., firewall-start2), then create the firewall-start script and have it call the other scripts.

Once installed and renamed, a copy/paste of the following into the window of an SSH session would create the appropriate firewall-start script.

cat << 'EOF' > /jffs/scripts/firewall-start
#!/bin/sh
/jffs/scripts/firewall-start1
/jffs/scripts/firewall-start2
EOF
chmod +x /jffs/scripts/firewall-start
:

In this way, you're not forced to merge the scripts and perhaps make a mistake. The scripts continue to run independently and unchanged. You could easily repeat this process for as many other scripts that want to use the same event.

But at the end of the day, even if you merge them, as long as you do so correctly, it will work. But if just makes it difficult for me or anyone else to provide assurance you actually did it correctly.

As far as dnsmasq.conf.add, that's just a simple non-executable text file containing a list of additional directives. So merging them is usually a simple process. Not unless you end up w/ contradictory/conflicting directives (e.g., each is insisting on the sole use of its own preferred DNS server(s)). In a case like that, you'd have to resolve it somehow (maybe it wouldn't even be resolvable, and thus deemed incompatible).

- As for the IPSETs it produces, it lists then like this:

ipset=/ipchicken.com/netflix.com/nflxext.com/nflximg.net/nflxso.net/lan2wan

It always lists them in 5's, then a new row, and 5 again. Why is that?

When I edit the file going forward, is that how I should do it, or is it fine to make it one big line?

The choice of 5 per line was arbitrary. I know there are usually practical limits when it comes to the length of a line, or even number of lines in a config file. I just thought it was safer to limit the ipset directive to 5 domains per line. But you can change it to whatever you like, even place all of them on the same line, as long as DNSMasq doesn't choke on it.

IOW, it's just my was of playing it safe and assuming there are unstated, practical limits that it would be best not to exceed.

 

[machinist]

@eibgrad You're a gentleman and a scholar. Thanks for always providing thourough and in-depth explanations. Makes it easy for even the noobs (hi) to understand.

This gets back to the point I was trying to make previously.

Totally. It was more a question of how to do it *specifically*, which you fully answered in the next bit.

I wasn't aware that it was OK to setup several of the firewall-start scripts, but thinking about it now, why wouldn't it. Makes sense.

If I ever have to set it up again, I'll go your preferred route. I can see why it's simpler to maintain.

But at the end of the day, even if you merge them, as long as you do so correctly, it will work. But if just makes it difficult for me or anyone else to provide assurance you actually did it correctly.

Pretty sure I merged them correctly (after all, just a simple copy/paste job), so I'll leave it alone for now.

IOW, it's just my was of playing it safe and assuming there are unstated, practical limits that it would be best not to exceed.

Got it, wasn't sure if it was your doing or some standard.

 

[eibgrad]

UPDATE: v3.0.0 is now available.

I've updated the script. Among other changes, I've revamped how the files are created and managed.

merlin-ovpn-lan2wan-by-domain.sh - Pastebin.com

Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.

pastebin.com

You can use the following command to download and install it using my new installer.

curl -kLs bit.ly/merlin-installer|tr -d '\r'|sh -s 9jHRA6DG

It will create seven (7) files.

/jffs/configs/merlin-ovpn-lan2wan-by-domain.options
/jffs/scripts/merlin-ovpn-lan2wan-by-domain.dnsmasq
/jffs/scripts/merlin-ovpn-lan2wan-by-domain.firewall
/jffs/scripts/merlin-ovpn-lan2wan-by-domain.openvpn
/jffs/scripts/dnsmasq.postconf
/jffs/scripts/firewall-start
/jffs/scripts/openvpn-event

I've been updating all my scripts to make it easier to resolve conflicts when other scripts (whether from me or others) need to use the same event scripts. I was previously burying the meat and potatoes of my scripts in the event scripts, making that process too difficult. Now the meat and potatoes are kept in my own files, and the Merlin user scripts are just shells that call my scripts. So even though there are more files, it's a good thing in the long run.

For the purposes of changing options, you just need to concern yourself w/ the *.options file.

If you need to uninstall it, just delete those same files.

rm -f /jffs/configs/merlin-ovpn-lan2wan-by-domain.options
rm -f /jffs/scripts/merlin-ovpn-lan2wan-by-domain.dnsmasq
rm -f /jffs/scripts/merlin-ovpn-lan2wan-by-domain.firewall
rm -f /jffs/scripts/merlin-ovpn-lan2wan-by-domain.openvpn
rm -f /jffs/scripts/dnsmasq.postconf
rm -f /jffs/scripts/firewall-start
rm -f /jffs/scripts/openvpn-event
:

So what's changed beyond that?

- Obviously the name. I changed it to more accurately reflect its actions.

- Fixed a bug that was ignoring the SOURCES variable for bypassing DNS. Probably most ppl are using *all* SOURCES anyway, so not a big deal.

- I removed the ipset hash table persistence logic. Didn't seem worth the trouble. All domains will simply be re-resolved on reboot. Persistence would have made more sense if ipset was being used for blocking. Eliminating it avoids two (2) more additional files.

- The prior version used ipchicken.com to verify the script was working (it always route to the WAN). I still retain it, but it's no longer in the DOMAINS variable, but in a separate variable called VERIFY_DOMAIN that always bound to the WAN, even if you specified nothing for DOMAINS. This prevents accidental deletion.

- Debugging mode is now OFF in all my scripts (it used to be a mix of ON and OFF). I tended to leave it ON assuming more ppl would have problems, but that's proven a rarity. And unless you're having problems, leaving it ON just fouls the syslog. If you ever need/or want to enable debugging mode, the easiest way is to pass the --debug option w/ the installer.

curl -kLs bit.ly/merlin-installer|tr -d '\r'|sh -s -- --debug 9jHRA6DG

The other option is to download the file w/ the --noexec option, and use the nano editor to uncomment the DEBUG variable at the top of the script.

script=/tmp/tmpfile.$$.sh
curl -kLs bit.ly/merlin-installer|tr -d '\r'|sh -s -- --noexec 9jHRA6DG > $script
chmod +x $script
nano $script
:

Finally, execute it.

$script

As I point out in the script's comments, realize the VPN can only be reliably bypassed for DNS purposes if you use either Exclusive or Disabled for Accept DNS configuration, since otherwise DNSMasq could possibly choose the WAN's DNS servers w/ the other options (Relaxed and Strict). As I'm sure most of you know, even if access to the domain is routed to the WAN, name resolution over the VPN may still be enough for the service to detect the VPN.

Enjoy!

 

[machinist]

Hey @eibgrad - and thanks for the update!

As I was clean flashing 386.7_2 I figured I might as well upgrade yours too. Now I'm having issues. The issue is no connection to outside internet at all. I can still SSH into the router.

Followed instructions in your latest post above.

Quick note, having first installed Skynet, firewall-start already existed so naturally there was a conflict. I manually pasted in Skynet's snippet:

sh /jffs/scripts/firewall start skynetloc=/tmp/mnt/usb/skynet # Skynet

...above yours.

Under DOMAINS='' I put the gathered hostnames on the same format as your netflix examples - one on each line.

In the options files under SOURCES='' I put 192.168.1.0/24 as I want all my internal IP's to route all the "DOMAINS" traffic thru WAN.

I have a OVPN profile set up in GUI that is working. I also have rules setup in VPN Director for the specific internal IPs I want routed thru the VPN tunnel.

Removing your script/files restores internet access. VPN still works.

Think that's it.

What am I doing wrong?

 

[eibgrad]

Hey @eibgrad - and thanks for the update!

As I was clean flashing 386.7_2 I figured I might as well upgrade yours too. Now I'm having issues. The issue is no connection to outside internet at all. I can still SSH into the router.

Followed instructions in your latest post above.

Quick note, having first installed Skynet, firewall-start already existed so naturally there was a conflict. I manually pasted in Skynet's snippet:

sh /jffs/scripts/firewall start skynetloc=/tmp/mnt/usb/skynet # Skynet

...above yours.

Under DOMAINS='' I put the gathered hostnames on the same format as your netflix examples - one on each line.

In the options files under SOURCES='' I put 192.168.1.0/24 as I want all my internal IP's to route all the "DOMAINS" traffic thru WAN.

I have a OVPN profile set up in GUI that is working. I also have rules setup in VPN Director for the specific internal IPs I want routed thru the VPN tunnel.

Removing your script/files restores internet access. VPN still works.

Think that's it.

What am I doing wrong?

It would help if you just posted the script as modified to PasteBin, just in case you made some sort of error.

And btw, the default in the options file applies ALL sources (0.0.0.0/0) to the rerouting. There's really no need for specifying 192.168.1.0/24 if in fact that's the only possible sources coming from your router. None the less, I'd still like to make sure there isn't a scripting error of my own.