Exclude website from being routed to VPN (policy based routing)

[machinist]

It would help if you just posted the script as modified to PasteBin, just in case you made some sort of error.

To simplify, I just tried to reinstall completely vanilla, except for the one line from Skynet that I pasted in above yours, in firewall-start like so:

#!/bin/sh
sh /jffs/scripts/firewall start skynetloc=/tmp/mnt/usb/skynet # Skynet
set -x # comment/uncomment to disable/enable debug mode
{
/jffs/scripts/merlin-ovpn-lan2wan-by-domain.firewall
} 2>&1 | logger -t $(basename $0)[$$]

I have not even touched the config file, so by default it should just route the netflix hostnames for all internal IP's.

As soon as I reboot, outside internet access completely disconnects. Both by cable and wifi. When I uninstall per your instructions by removing those 7 files and reinstating the Skynet "firewall-start" file and reboot, connection is back.

---

I install using your installer

curl -kLs bit.ly/merlin-installer|tr -d '\r'|sh -s 9jHRA6DG

I've also tried

$script

...after that , whether needed or not. Same result.

 

[eibgrad]

Took some effort, but I did find "a" problem, but I don't know if it's "the" problem.

I created a temporary installer and updated script that I'd like you to try. If it works, I'll then update the production script w/ the changes. I'll also explain the issue in more detail later.

curl -kLs pastebin.com/raw/PDXffDCs|tr -d '\r'|sh -s 0yRRsA82

One of the reasons I'm NOT so sure this is YOUR issue is because you're running Skynet too (I'm NOT). And so there's always the risk of incompatibility between the scripts, esp. when my script is marking packets, which as I explained earlier in this thread, is always a bit dicey proposition since it's far too easy to have the firmware and scripts clobber each other's marks. So I'm always on the lookout for that possibility when users report problems. But as I said, I found a different issue that's affecting me too (although in my case it was only preventing access to the domains being routed over the WAN, NOT the VPN), and it has NOTHING to do w/ marking packets.

 

[machinist]

@eibgrad

Thanks for the effort.

I can report that it still doesn't work. Connection gets cut, however there's strange behavior. "router.asus.com" no longer works, but 192.168.1.1 does. On the "activity" dashboard, I see activity. It also says "Internet status: connected"

Cable connection as well as wifi is still cut, tho. Tried from several devices. Removing the files immediately restores connection like before.

One of the reasons I'm NOT so sure this is YOUR issue is because you're running Skynet too (I'm NOT)

Since it worked with the previous version, should I just go back?

 

[eibgrad]

@eibgrad

Thanks for the effort.

I can report that it still doesn't work. Connection gets cut, however there's strange behavior. "router.asus.com" no longer works, but 192.168.1.1 does. On the "activity" dashboard, I see activity. It also says "Internet status: connected"

Cable connection as well as wifi is still cut, tho. Tried from several devices. Removing the files immediately restores connection like before.



Since it worked with the previous version, should I just go back?

Not quite sure what could have changed. In the process of revamping my scripts, I was pretty careful to NOT to make significant changes. It was complex enough as it was to just get things reorganized, and I feared breaking something. But looking through the script in detail, I just don't see what it could be. That's why I tend to believe it's a conflict w/ the other script.

Did you try the script w/o Skynet? Because after the change I made, it's working fine here. Just as it was before I reorganized the files. If there is a problem, I'd obviously like to find it. Before going back, it would help if you dumped some relevant data structure just to see if there's some obvious conflict(s).

ifconfig
ip route show table main
ip route show table ovpnc1
ip route show table ovpnc2
ip route show table ovpnc3
ip route show table ovpnc4
ip route show table ovpnc5
ip rule
iptables -t nat -vnL --line-numbers
iptables -t mangle -vnL --line-numbers
iptables -vnL --line-numbers
cat /tmp/etc/openvpn/client1/config.ovpn
cat /tmp/etc/openvpn/client2/config.ovpn
cat /tmp/etc/openvpn/client3/config.ovpn
cat /tmp/etc/openvpn/client4/config.ovpn
cat /tmp/etc/openvpn/client5/config.ovpn
cat /jffs/openvpn/vpndirector_rulelist

I know it's a lot, but then I don't know what I'm looking for as yet. Maybe I'll just get lucky and notice something.

You could also reinstall the script using the --debug options, which will dump the scripts activity to the syslog, and then PM (or PasteBin) the syslog back to me.

curl -kLs pastebin.com/raw/PDXffDCs|tr -d '\r'|sh -s -- --debug 0yRRsA82

 

[eibgrad]

P.S. I just add the following to that list as well.

iptables -t mangle -vnL --line-numbers

 

[machinist]

Tried your latest curl suggestion, after having uninstalled Skynet - and same result. Connection is completely cut.

I'm now starting completely fresh, following L&LD's factory reset guide. Including reflashing 386.7_2 to make sure no funny business is going on.

Now, with only Merlin installed, otherwise completely vanilla, I reinstalled your script from the latest installer.

I now have internet access. So it would indeed appear to be a conflict with either Skynet, connmon, spdMerlin or scMerlin. Don't know which one.

So I set up an OVPN profile and configured VPN Director. On the specified internal IP, the VPN is now running.

I went to update your config file with my domains. One is hostnames for my local streaming service and one is Disney plus. I got them using Xentrk's getdomainnames.sh script -- which worked previously.

Now it doesn't -- neither of the services. Both services complaining I'm from out of country, so they are not routed through WAN.

One step forward, two steps back.

You could also reinstall the script using the --debug options, which will dump the scripts activity to the syslog, and then PM (or PasteBin) the syslog back to me.

I'll do that, thanks.

 

[machinist]

Update: I think it's working!

Perhaps it was the reflashing that did the trick, but my connection is now fully working and the hostnames appear to be working as well! Both services working, yay!


And this after having reinstalled all the scripts in my sig, too.

One thing I did differently was to paste the Skynet snippet in after lan2wan in firewall-start, so it now looks like:

#!/bin/sh
set -x # comment/uncomment to disable/enable debug mode
{
/jffs/scripts/merlin-ovpn-lan2wan-by-domain.firewall
} 2>&1 | logger -t $(basename $0)[$$]
sh /jffs/scripts/firewall start skynetloc=/tmp/mnt/usb/skynet # Skynet

Will monitor if that has a negative effect on Skynet.

 

[eibgrad]

Update: I think it's working!

Perhaps it was the reflashing that did the trick, but my connection is now fully working and the hostnames appear to be working as well! Both services working, yay!


And this after having reinstalled all the scripts in my sig, too.

One thing I did differently was to paste the Skynet snippet in after lan2wan in firewall-start, so it now looks like:

#!/bin/sh
set -x # comment/uncomment to disable/enable debug mode
{
/jffs/scripts/merlin-ovpn-lan2wan-by-domain.firewall
} 2>&1 | logger -t $(basename $0)[$$]
sh /jffs/scripts/firewall start skynetloc=/tmp/mnt/usb/skynet # Skynet

Will monitor if that has a negative effect on Skynet.

That's why merging scripts this way can be problematic. The order of execution might matter. Just depends if one corrupts the other, and perhaps in only one particular order. It's a roll of the dice since there is no means to coordinate changes among authors. It's why when there are problems, you need to test them in isolation to make sure they do in fact work normally. Then when merged and things go awry, you can assume there are probably conflicts.

Thankfully my recent changes at least made it much easier to merge scripts that depend on the same events. But it can't in and of itself prevent conflicts.

 

[eibgrad]

BTW, the change I made to the script was to disable RPF (reverse-path filtering). For some reason, that became an issue for me during recent testing. It wasn't something I had seen any time prior, so it caught me off-guard.

RPF is what the routing system uses to force connections to always use the same network interface for both incoming and outgoing traffic on a given connection. What I suspect was happening is that at some point in the processing of packets originally intended for the VPN, my changes that forced it to the WAN were being seen as an attempt to have the connection used w/ both the WAN and VPN. Of course, that's NOT the case. But I think the change in the routing happens so late, that sometimes the routing system believes that's the case, and so it blocks it. So I had to disable RPF to keep the routing system happy.

A very obscure problem that most ppl don't need to concern themselves with. It might even be something unique w/ my setup (again, I didn't see any of this behavior during prior testing). So I will likely update the official script w/ these changes just in case someone else runs into the same problem. I might even make it a configurable option, since I'd prefer to keep RPF enabled unless there's no other choice.

 

[machinist]

The order of execution might matter. Just depends if one corrupts the other, and perhaps in only one particular order.

That was my thinking, and why I tried.

the change I made to the script was to disable RPF (reverse-path filtering)

Thanks for the explanation, and once again, your time and effort. It's appreciated.

Q: So I have the version with the debug flag installed now. How do I deactivate that, so it doesn't fill up logs needlessly?

 

[eibgrad]

Q: So I have the version with the debug flag installed now. How do I deactivate that, so it doesn't fill up logs needlessly?

You either need to install again w/o the --debug option, or go into each file and comment out the second line w/ set -x.

#set -x # comment/uncomment to disable/enable debug mode

That's normally how it's configured. But when you use the --debug option on installation, it removes the # to activate it.

 

[eibgrad]

P.S. And easy way to edit all the file at once is to simply cd to /jffs/scripts and issue the following command.

nano *

Makes quick work of it.

 

[ZebMcKayhan]

RPF is what the routing system uses to force connections to always use the same network interface for both incoming and outgoing traffic on a given connection. What I suspect was happening is that at some point in the processing of packets originally intended for the VPN, my changes that forced it to the WAN were being seen as an attempt to have the connection used w/ both the WAN and VPN. Of course, that's NOT the case. But I think the change in the routing happens so late, that sometimes the routing system believes that's the case, and so it blocks it. So I had to disable RPF to keep the routing system happy.

Using firewall to mark packages and use the mark to decide routing will always require you to atleast set rp_filter to loose (2). Appearantly the rp_filter dont understand that the reply package will recieve a mark by the firewall and be routed differently because of this mark. However loose is better than disabled.

https://github.com/ZebMcKayhan/WireguardManager#managesetup-ipsets-for-policy-based-routing

 

[eibgrad]

Using firewall to mark packages and use the mark to decide routing will always require you to atleast set rp_filter to loose (2). Appearantly the rp_filter dont understand that the reply package will recieve a mark by the firewall and be routed differently because of this mark. However loose is better than disabled.

https://github.com/ZebMcKayhan/WireguardManager#managesetup-ipsets-for-policy-based-routing

Thanks. Very helpful!

It's just that's it weird when it gave me no problems all the time I was testing it w/ the original release, then NOW it decides to become an issue. I don't understand the inconsistency. And it's NOT the first time I've seen such inconsistency.

 

[ZebMcKayhan]

Thanks. Very helpful!

It's just that's it weird when it gave me no problems all the time I was testing it w/ the original release, then NOW it decides to become an issue. I don't understand the inconsistency. And it's NOT the first time I've seen such inconsistency.

I never did get around to check if

echo 2 > /proc/sys/net/ipv4/conf/eth0/rp_filter

Is static. If it is then maybe it was disabled for you before and something decided to change it back? Just a thought. I'll set it at each nat-start and it's been behaving.

Edit: if the reply packets were carrying the same mark it may work with rp_filter turned on, but I have always assumed rp_filter to be before all netfilter tables but really dont know.

 

[ZebMcKayhan]

A very obscure problem that most ppl don't need to concern themselves with. It might even be something unique w/ my setup

Did some digging. Appearantly Wireguard requires src_valid_mark to be set to '1' (very common that wireguard uses fwmark to route packages):
https://www.kernel.org/doc/html/latest/networking/ip-sysctl.html

src_valid_mark - BOOLEAN

0 - The fwmark of the packet is not included in reverse path route lookup. This allows for asymmetric routing configurations utilizing the fwmark in only one direction, e.g., transparent proxying.

1 - The fwmark of the packet is included in reverse path route lookup. This permits rp_filter to function when the fwmark is used for routing traffic in both directions.

This setting also affects the utilization of fmwark when performing source address selection for ICMP replies, or determining addresses stored for the IPOPT_TS_TSANDADDR and IPOPT_RR IP options.

The max value from conf/{all,interface}/src_valid_mark is used.

Default value is 0.

this variable is indeed included in our firmware:

admin@RT-AC86U-D7D8:/# find -name src_valid_mark
./proc/sys/net/ipv4/conf/all/src_valid_mark
./proc/sys/net/ipv4/conf/bcmsw/src_valid_mark
./proc/sys/net/ipv4/conf/br0/src_valid_mark
./proc/sys/net/ipv4/conf/default/src_valid_mark
./proc/sys/net/ipv4/conf/dpsta/src_valid_mark
./proc/sys/net/ipv4/conf/eth0/src_valid_mark
...

maybee this would allow to keep rp_filter enabled?? Unfortunately I can't experiment with my router right now, my wife is already angry with me, but perhaps you could give it a go?

Edit: also found an interesting note to store fwmark to connmark and then restore it, might be useful if the internet reply needs the same mark?

iptables -I PREROUTING -t mangle -m state --state new -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask 0xffffffff
iptables -I PREROUTING -t mangle -j CONNMARK --restore-mark --nfmask 0xffffffff --ctmask 0xffffffff

Sorry to provide such half-baked solution but I too feel just disable/cripple rp_filter is not the best idea (it does perform some ddos protection). Perhaps you could puzzle it together or I'll look more into it as I have time.

 

[eibgrad]

Edit: also found an interesting note to store fwmark to connmark and then restore it, might be useful if the internet reply needs the same mark?

iptables -I PREROUTING -t mangle -m state --state new -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask 0xffffffff
iptables -I PREROUTING -t mangle -j CONNMARK --restore-mark --nfmask 0xffffffff --ctmask 0xffffffff

I've been doing that for years now w/ my DD-WRT/FreshTomato advanced routing policy scripts.

ddwrt-ovpn-split-advanced.sh - Pastebin.com

Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.

pastebin.com

tomato-ovpn-split-advanced.sh - Pastebin.com

Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.

pastebin.com

One of the advantages of this technique is that I can now mark packets from any connections initiated from either the WAN or VPN so that the replies are always sent out the same network interface, and w/o the user having to create those rules specifically. That would normally be a problem, for example, if the target of remote access over the WAN was bound to the router's local OpenVPN client. Without the script, those packets would be directed out the VPN, but now go back out the WAN.

IOW, you don't even need to add any rules of your own to benefit from the scripts. Merely installing them solves that one issue, w/ no further actions required.

 

[ZebMcKayhan]

I've been doing that for years now w/ my DD-WRT/FreshTomato advanced routing policy scripts.

ddwrt-ovpn-split-advanced.sh - Pastebin.com

Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.

pastebin.com

tomato-ovpn-split-advanced.sh - Pastebin.com

Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.

pastebin.com

One of the advantages of this technique is that I can now mark packets from any connections initiated from either the WAN or VPN so that the replies are always sent out the same network interface, and w/o the user having to create those rules specifically. That would normally be a problem, for example, if the target of remote access over the WAN was bound to the router's local OpenVPN client. Without the script, those packets would be directed out the VPN, but now go back out the WAN.

IOW, you don't even need to add any rules of your own to benefit from the scripts. Merely installing them solves that one issue, w/ no further actions required.

Cool! But you still disable rp_filter in your scripts despite symetric routing and symetric marking. Would this mean that my intention with symetric marking/routing and src_valid_mark=1 is not the solution for keeping rp_filter enabled?

 

[hung2k]

Hi, @eibgrad
Is it possible to make a alternative script to include specific website only being routed to vpn?
Thank You!

 

[eibgrad]

Update: v3.1.0 is now available.

I made a single change to enable RPF (reverse path filtering) on the WAN. This might prove necessary in some cases. Without it, you may find it impossible to reach any domains bound to the WAN.