Dual-Wan Force vpn client to WAN1

[hike]

Hi all i need some help.

Dual-wan in fail over mode does not have wan0 and wan1, however if you disable and enable the secondary connection it goes from cold-standby to hot-standby.
So i see on eth4 it recieved ip and gateway from my secondary ISP.
This connection is idle all the time and do nothing, but i would like to push some data over it, so i followed instructions from :

https://www.snbforums.com/threads/s...-to-choose-from-which-to-wan-to-go-out.38146/

#!/bin/sh
CONFIG=$1
source /usr/sbin/helper.sh
pc_replace "nobind" "local xxx.xxx.xxx.xxx" $CONFIG

So now my openvpnclient 1 bind nicely on my secondary ISP even when in failover mode.
Openvpn client log shows the right binding ip from secondary ISP.

ISP1 = 300 Mbit / 100 with openvpn
ISP2 = 20 Mbit / 20 with openvpn

I add machines to the openvpn1 (policy based routing) but when i do a speedtest i get full speed from my primary ISP , and when i look at the eth4 interface stats i see no data going over that interface.
So my traffic still goes out over primary ISP.

I use x3mrouting, skynet, and diversion, maybe one of these could brake something here.

For me is no option to go to loadbalance mode because of the big difference in connection speed , amd losing adaptive qos.
Does anybody have a clue what this could be ?

Thanks already !

Hike

 

[Martineau]

I add machines to the openvpn1 (policy based routing) but when i do a speedtest i get full speed from my primary ISP , and when i look at the eth4 interface stats i see no data going over that interface.
So my traffic still goes out over primary ISP.

Does anybody have a clue what this could be ?

You will need to examine the RPDB rules and VPN Client 1 and main routing tables

ip rule

ip route show table 111

ip route show table main

but I suspect you will need to investigate this script patch as a solution.


P.S. I'm not 100% sure your hack to put WAN1 into 'Hot-Standby' has actually worked.

 

[hike]

thanks for the reply
ip rule :

0: from all lookup local
9990: from all fwmark 0x8000/0x8000 lookup main
10101: from 192.168.1.160 lookup ovpnc1
10102: from 192.168.1.240 lookup ovpnc1
10103: from 192.168.1.27 lookup ovpnc1
10104: from 192.168.1.183 lookup ovpnc1
10105: from 192.168.1.19 lookup ovpnc1
10106: from 192.168.1.154 lookup ovpnc1
10107: from 192.168.1.120 lookup ovpnc1
10108: from 192.168.1.21 lookup ovpnc1
10109: from 192.168.1.128 lookup ovpnc1
10110: from 192.168.1.91 lookup ovpnc1
10111: from 192.168.1.203 lookup ovpnc1
10112: from 192.168.1.229 lookup ovpnc1
10113: from 192.168.1.159 lookup ovpnc1
10114: from 192.168.1.181 lookup ovpnc1
10115: from 192.168.1.209 lookup ovpnc1
10201: from 192.168.1.31 lookup main
10202: from 192.168.1.26 lookup main
10203: from 192.168.1.235 lookup main
10204: from 192.168.1.163 lookup main
10205: from 192.168.1.215 lookup main
10206: from 192.168.1.87 lookup main
10207: from 192.168.1.50 lookup main
10208: from 192.168.1.86 lookup main
10209: from 192.168.1.242 lookup main
10210: from 192.168.1.54 lookup main
10211: from 192.168.1.177 lookup main
10301: from 192.168.1.0/24 lookup ovpnc2
10501: from all to ***** lookup ovpnc3
10701: from 192.168.1.41 lookup ovpnc4
10702: from 192.168.1.18 lookup ovpnc4
10703: from 192.168.1.164 lookup ovpnc4
10704: from 192.168.1.92 lookup ovpnc4
10705: from 192.168.1.124 lookup ovpnc4
10706: from 192.168.1.83 lookup ovpnc4
32766: from all lookup main
32767: from all lookup default

table 111:

default via 10.8.1.1 dev tun11
10.8.1.0/24 dev tun11 proto kernel scope link src 10.8.1.3
192.168.1.0/24 dev br0 proto kernel scope link src 192.168.1.1

table main :

default via Main ISP dev eth0
10.8.1.0/24 dev tun11 proto kernel scope link src 10.8.1.3
10.8.3.0/24 dev tun12 proto kernel scope link src 10.8.3.31
Secondary ISP dev eth4 proto kernel scope link src ip from secondary isp
** dev tun13 proto kernel scope link src **
** dev eth0 proto kernel scope link src **
** dev eth0 proto kernel scope link
127.0.0.0/8 dev lo scope link
172.21.94.0/23 dev tun14 proto kernel scope link src 172.21.94.190
192.168.1.0/24 dev br0 proto kernel scope link src 192.168.1.1

eth4 shows all information from secondary ISP
default goes to Main ISP ip

Thanks !

 

[hike]

I checked the script you mentioned, i do not have WAN1 as a interface at the moment is hot standbye. (fail over)
However i have a eth4 with a working connection to my secondary ISP as you can see in my previous message.

 

[Martineau]

I checked the script you mentioned, i do not have WAN1 as a interface at the moment is hot standbye. (fail over)
However i have a eth4 with a working connection to my secondary ISP as you can see in my previous message.

I have only used USB as the Secondary WAN, and when enabling the GUI Dual-WAN Fail-Over (FO) mode, hacking the USB 'Cold-Standby' to 'Hot-Standby' isn't something I have successfully tried.

If you say using a LAN port as the secondary WAN, and a simple command sequence

ifconfig ethX down
ifconfig ethX up

suddenly allows the VPN Client to bind to ethX, then I suggest you manually initiate a command line data transfer via the VPN tunnel bound to switch interface ethX and validate the response.

NOTE: If you have IPv6 enabled, then this will override the Selective routing which is IPv4 only.

 

[hike]

Hi,

What i do to get the Secondary Wan from cold-standbye to hot-standbye is just disable and enable the connection. (sometimes i need to do this a few times because of DHCP fail)

https://gyazo.com/678b08fc251cb16ed88d7805a9b7ffb5

After that i can bind openvpn1 to the interface, and that goes fine.
Sep 19 10:03:48 ovpn-client1[23231]: UDP link local (bound): [AF_INET]Secondary ISP IP:1194

But now comes the weird part, if i add my laptop to openvpn1 based on policy routing, i get speeds what that connection does not have.
My secondary ISP is max 20mbit download and 2 mbit upload.
When i do a speedtest i get 100+ down and 1.5 mbit up.

So i think i send packets to openvpn1 , but on a strange way the traffic comes back true my primary ISP.
If i disable my primary ISP (plug out the cable) i get the max speed from ISP2 max 20 mbit.
As soon as i put ISP1 back i get 100+ mbit again.

Any idea how openvpn can bind to the right interface ISP2 but traffic comes back to the ISP 1 ?
I only have one openvpn client running in this test.

 

[Martineau]

Hi,

What i do to get the Secondary Wan from cold-standbye to hot-standbye is just disable and enable the connection. (sometimes i need to do this a few times because of DHCP fail)

https://gyazo.com/678b08fc251cb16ed88d7805a9b7ffb5

After that i can bind openvpn1 to the interface, and that goes fine.
Sep 19 10:03:48 ovpn-client1[23231]: UDP link local (bound): [AF_INET]Secondary ISP IP:1194

But now comes the weird part, if i add my laptop to openvpn1 based on policy routing, i get speeds what that connection does not have.
My secondary ISP is max 20mbit download and 2 mbit upload.
When i do a speedtest i get 100+ down and 1.5 mbit up.

So i think i send packets to openvpn1 , but on a strange way the traffic comes back true my primary ISP.
If i disable my primary ISP (plug out the cable) i get the max speed from ISP2 max 20 mbit.
As soon as i put ISP1 back i get 100+ mbit again.

Any idea how openvpn can bind to the right interface ISP2 but traffic comes back to the ISP 1 ?
I only have one openvpn client running in this test.

TL;DR

IMHO the designations 'Cold-Standby' and 'Hot-standby' describes the difference in the actual physical actions (and associated wall-clock delay) required to effect the state change to 'Connected'.

i.e. a crude analogy is my HP laptop which obviously includes a trackpad, but I prefer an external Bluetooth mouse with physical scroll-wheel.

Dual-Mouse Load-Balance:

Both the trackpad and the external mouse may be used concurrently.
​

Dual-Mouse Hot-Standby:

When I boot the laptop, I normally double tap on the top left-hand corner of the trackpad which illuminates an LED that indicates the trackpad is physically DISABLED (and usefully prevents accidental trackpad mouse movement when I am typing)

So if the external mouse isn't responsive, I can simply double tap the trackpad to ENABLE it and instantly restore alternate mouse functionality - although I lose the 'performance/convenience' of the scroll-wheel.
​

Dual-Mouse Cold-Standby:

If the external mouse isn't responsive and I still require the continued use of the scroll-wheel, I can go and find my old wired mouse and free-up an occupied USB port to attach the alternate external mouse.

So clearly I can restore the required mouse functionality, but the physical switching to the new mouse takes longer as it requires additional time-consuming actions.​

Any idea how openvpn can bind to the right interface ISP2 but traffic comes back to the ISP 1 ?

No, .....at least certainly not with the simple setup you have described - although it would be useful!

i.e. opening a website through the slow VPN ISP2 to browse say a HTML directory of available DVD ISOs to download, but when clicking on an item, magically it is physically retrieved through the fast ISP1 WAN interface!

However, in the case of your symptoms, manually switching the Secondary WAN to 'Hot-Standby' from its normal 'Cold-Standby' state, seemingly still leaves the Secondary WAN effectively in 'standby', probably meaning the Primary WAN is still preferred.

Consequently I suspect that the desired BINDing of the OpenVPN Client to the Secondary WAN isn't actually physically enforced if the (preferred) Primary WAN interface is UP.

i.e. Simply unplugging the Primary WAN cable sets off a chain of events that takes DOWN the Primary WAN, restarts the firewall etc. and restarts the VPN Client which now is physically able to BIND to the newly initialised connected Secondary WAN.
Re-inserting the Primary WAN cable, tears down the Secondary WAN (i.e. puts it back to 'Cold-Standby' state), and upon the firewall restart, the VPN Client is restarted and actually BINDs to the preferred Primary WAN.


P.S. If the loss of the Adaptive QoS feature can be tolerated, then perhaps configuring Dual-WAN Load-Balancing (with a forced ratio 1:0), and testing if BINDing the OpenVPN Client to the permanently 'connected' Secondary WAN may correctly always yields the expected VPN throughput rate.

 

[hike]

Thank you for taking the time and always respond in this forum on people's questions and come with solutions

I understand what you are saying about the Cold and Hot standby, i tried to do load-balance but i do not know how to change the ratio to 1:0 so that no traffic is going to the secondary WAN based on basic routing.

If i executed ( i hope you can help me with that ) the ratio to 1:0, and force openvpn client 2 to secondary wan with the openvpnclient2.postconf is there then still the posibility to let other clients use the secondary wan interface without using the VPN ?
I know i have seen a script from you because there is only the option to choose VPN or WAN (and not secondary WAN) but i do not know if that hack is still needed in the latest version of merlin.
What i have read is that openvpnclients are also going on preference, so client1 is prefered at client2 , that would give me troubles again because my openvpnclient1 has :

Openvpnclient 1

192.168.0.1/24 --> VPN
192.168.1.1/32 --> WAN
Machine 1 --> WAN
Machine 2 --> WAN

If i can bind openvpnclient2 to Secondary WAN i it would look like this :

Openvpn client 2

Machine 5 --> VPN
Machine 6 --> WAN (Secondary WAN)

Do you think this is possible ?

Again Thanks !

 

[Martineau]

i do not know how to change the ratio to 1:0 so that no traffic is going to the secondary WAN based on basic routing.

You can check the status of the current Dual-WAN Load-Balance (LB) ratio

iptables -nvL balance --line -t mangle

To override the Dual-WAN Load-Balance (LB) GUI ratio i.e. set it to 1:0

iptables -t mangle -R balance "$(iptables -nvL balance --line -t mangle | grep -F "xset 0x90000000" | cut -d' ' -f1)" -m connmark --mark 0x0 -j CONNMARK --set-xmark 0x80000000/0xf0000000

and to revert to the currently implemented Dual-WAN Load-Balance ratio i.e. remove the 1:0 ratio

iptables -t mangle -R balance "$(iptables -nvL balance --line -t mangle | grep -E "connmark match.*CONNMARK xset 0x80000000" | cut -d' ' -f1)"  -m connmark --mark 0x0 -j CONNMARK --set-xmark 0x90000000/0xf0000000

I suggest you first test that having verified that ALL traffic is now via WAN0, you can successfully BIND your VPN Client 2 only to WAN1.

If the BIND is successful, LAN client traffic that should be routed via a VPN will need to have the priority of the RPDB rules created by the Selective Routing GUI replicated/moved higher in the RPDB rule table

NOTE: The crude hack to '/usr/sbin/vpnrouting.sh' performs the necessary promoting/increasing of the priority of the Dual-WAN VPN Client RPDB rules, but you could manually create the rules in the appropriate openvpn event script 'vpn_client?-route-up'

e.g.

ip rule add from xxx.xxx.xxx.xxx table ovpnc? prio 5?

where ? is the VPN Client instance.​

If i can bind openvpnclient2 to Secondary WAN i it would look like this :
Openvpn client 2
Machine 5 --> VPN

ip rule add from xxx.xxx.xxx.5 table ovpnc2 prio 52
ip route flush cache

Machine 6 --> WAN1 (Secondary WAN)

Simply add 'Machine 6' in the Dual-WAN Routing GUI (but this initiates a reboot)

e.g.

Machine6   all   Secondary_WAN

What i have read is that openvpnclients are also going on preference, so client1 is prefered at client2 , that would give me troubles again because my openvpnclient1 has :
Openvpnclient 1
192.168.0.1/24 --> VPN
192.168.1.1/32 --> WAN
Machine 1 --> WAN
Machine 2 --> WAN

I'm not clear on exactly what you mean by the above?

Do you intend to route ALL of your LAN subnet (typo? 192.168.0.1/24) via VPN Client 1 except for 192.168.1.1/32,'Machine 1' and 'Machine 2' ?
If you do then you need to ensure that the rules for specific LAN devices are processed before CIDR ranges in the RPDB rules table.

i.e. if the VPN Client 1 table you propose is used, then 'Machine 5' will never be routed via VPN Client 2.

 

[hike]

Do you intend to route ALL of your LAN subnet (typo? 192.168.0.1/24) via VPN Client 1 except for 192.168.1.1/32,'Machine 1' and 'Machine 2' ?
If you do then you need to ensure that the rules for specific LAN devices are processed before CIDR ranges in the RPDB rules table.
i.e. if the VPN Client 1 table you propose is used, then 'Machine 5' will never be routed via VPN Client 2.

Yes that is correct , i have a manual entry where all my trafiic goes to the VPN (192.168.1.0/24) and then i tell what traffic needs to go to the WAN.
Because of the number of clients this is (lazy) easyier.
So i will play with the RPDB rules to make it work, when everything is up, maybe in my case it is better to use the client-up scripts on every openvpn interface with the same commands because i can't control what openvpnclient come up first.

One more question about Dual-Wan, if the secondary lose connection, does the ratio and the RPDB rules go to lower prio or are deleted ?
Because in that case the openvpn client to that secondary wan is disconnected and all traffic will be blocked ?
Is dual-wan also fail over in a way ?

I will set this up today and report how it goes, thanks for the detailed description.

 

[Martineau]

Yes that is correct , i have a manual entry where all my trafiic goes to the VPN (192.168.1.0/24) and then i tell what traffic needs to go to the WAN.
Because of the number of clients this is (lazy) easyier.
So i will play with the RPDB rules to make it work, when everything is up, maybe in my case it is better to use the client-up scripts on every openvpn interface with the same commands because i can't control what openvpnclient come up first.

For LAN devices that should not use a VPN tunnel, if they require WAN affinity (rather than honour the current Dual-WAN ratio) simply define them in the Dual-WAN Selective Routing GUI, but if the 64 rule limit is too restrictive then manually add them via a script.

Using the appropriate custom openvpn event scripts to manage the VPN RPDB rules is preferred - although if the VPN KILL-Switch is required then patching '/usr/sbin/vpnrouting.sh' may be the easier option.

However, ensure that the VPN RPDB rules are a higher priority than 150 and be aware priority 100 is used by the Dual-WAN GUI rules (if defined) - perhaps priority range 111-115? to indicate VPN related rules, and to ensure that the order in which the VPN Clients initialise is irrelevant.

e.g. Proposed/desired manual RPDB rules (if not using a patched '/usr/sbin/vpnrouting.sh')

# Selective Routing WAN rules
ip rule add from machine2 via table wan1     prio 100   # Machine2 via Secondary WAN
ip rule add from machine1 via table wan0     prio 100   # Machine1 via Primary   WAN

# Selective Routing VPN rules
ip rule add from machine5 via table ovpnc2   prio 111   # Machine5    via VPN Client 2
ip rule add from 192.168.1.0/24 table ovpnc1 prio 115   # LAN Default via VPN Client 1

One more question about Dual-Wan, if the secondary lose connection, does the ratio and the RPDB rules go to lower prio or are deleted ?
Because in that case the openvpn client to that secondary wan is disconnected and all traffic will be blocked

The Dual-WAN ratio in the GUI is retained, although it is possible for a script to display the true ratio.

e.g. The "9:1" ratio is still physically configured (to enable hot-fallback), but the script displays that in actual fact the real Dual-WAN ratio is "0:1"

./DualWAN_Selective.sh status

Chain balance (1 references)
num   pkts bytes target     prot opt in     out     source               destination        
<snip>
10      35  3445 CONNMARK   all  --  *      *       0.0.0.0/0            0.0.0.0/0            statistic mode random probability 0.89999999991 CONNMARK xset 0x90000000/0xf0000000
11       4   342 CONNMARK   all  --  *      *       0.0.0.0/0            0.0.0.0/0            connmark match  0x0 CONNMARK xset 0x90000000/0xf0000000

    Current Dual-WAN (eth0/ppp0) Load-Balance (LB) Ratio=0:1

Furthermore it is your responsibility to ensure custom VPN related RPDB rules are recreated if they go AWOL etc.

Is dual-wan also fail over in a way ?

Frustratingly, Dual-WAN Failover has been fatally borked by ASUS again
(Seriously, whoever thought it was a good idea to assume that an electrical NIC connection indicates the interface is capable of passing data is totally brain-dead IMHO. )

In order to eliminate 'false-positives', I wrote ChkWan.sh to personally control all aspects of the WAN Fail-Over process. My wrapper script Dual_WAN_Monitor.sh can now intelligently decide if it is appropriate to manually effect the logical WAN switch.

i.e. Having switched to only a working WAN, the wrapper script can patiently keep checking for the failed WAN to recover by itself, or can take appropriate action i.e. send notification e-mail before either requesting a manual restart of the failed WAN or initiating a full reboot if it is within an acceptable non-prime period of the day.

Whilst I may sometimes have better control of the Dual-WANs, if seamless WAN switching is critical, unfortunately, as others have commented, ASUS routers should not be used in mission critical environments - Synology Router appears to be recommended.

 

[Pak Kriss]

This morning I thought I need to write a little DualWAN checker script and now I found you did already. Thank you.

You mentioned the "Dual_WAN_Monitor.sh" script - where it's located for download?

This I found: https://github.com/MartineauUK/Chk-WAN

May I assume that crontab is best place for that?

For LAN devices that should not use a VPN tunnel, if they require WAN affinity (rather than honour the current Dual-WAN ratio) simply define them in the Dual-WAN Selective Routing GUI, but if the 64 rule limit is too restrictive then manually add them via a script.

Using the appropriate custom openvpn event scripts to manage the VPN RPDB rules is preferred - although if the VPN KILL-Switch is required then patching '/usr/sbin/vpnrouting.sh' may be the easier option.

However, ensure that the VPN RPDB rules are a higher priority than 150 and be aware priority 100 is used by the Dual-WAN GUI rules (if defined) - perhaps priority range 111-115? to indicate VPN related rules, and to ensure that the order in which the VPN Clients initialise is irrelevant.

e.g. Proposed/desired manual RPDB rules (if not using a patched '/usr/sbin/vpnrouting.sh')

# Selective Routing WAN rules
ip rule add from machine2 via table wan1     prio 100   # Machine2 via Secondary WAN
ip rule add from machine1 via table wan0     prio 100   # Machine1 via Primary   WAN

# Selective Routing VPN rules
ip rule add from machine5 via table ovpnc2   prio 111   # Machine5    via VPN Client 2
ip rule add from 192.168.1.0/24 table ovpnc1 prio 115   # LAN Default via VPN Client 1

The Dual-WAN ratio in the GUI is retained, although it is possible for a script to display the true ratio.

e.g. The "9:1" ratio is still physically configured (to enable hot-fallback), but the script displays that in actual fact the real Dual-WAN ratio is "0:1"

./DualWAN_Selective.sh status

Chain balance (1 references)
num   pkts bytes target     prot opt in     out     source               destination        
<snip>
10      35  3445 CONNMARK   all  --  *      *       0.0.0.0/0            0.0.0.0/0            statistic mode random probability 0.89999999991 CONNMARK xset 0x90000000/0xf0000000
11       4   342 CONNMARK   all  --  *      *       0.0.0.0/0            0.0.0.0/0            connmark match  0x0 CONNMARK xset 0x90000000/0xf0000000

    Current Dual-WAN (eth0/ppp0) Load-Balance (LB) Ratio=0:1

Furthermore it is your responsibility to ensure custom VPN related RPDB rules are recreated if they go AWOL etc.

Frustratingly, Dual-WAN Failover has been fatally borked by ASUS again
(Seriously, whoever thought it was a good idea to assume that an electrical NIC connection indicates the interface is capable of passing data is totally brain-dead IMHO. )

In order to eliminate 'false-positives', I wrote ChkWan.sh to personally control all aspects of the WAN Fail-Over process. My wrapper script Dual_WAN_Monitor.sh can now intelligently decide if it is appropriate to manually effect the logical WAN switch.

i.e. Having switched to only a working WAN, the wrapper script can patiently keep checking for the failed WAN to recover by itself, or can take appropriate action i.e. send notification e-mail before either requesting a manual restart of the failed WAN or initiating a full reboot if it is within an acceptable non-prime period of the day.

Whilst I may sometimes have better control of the Dual-WANs, if seamless WAN switching is critical, unfortunately, as others have commented, ASUS routers should not be used in mission critical environments - Synology Router appears to be recommended.

Sent from my GM1910 using Tapatalk

 

[hike]

Sorry for the late response.

I changed to Dual Wan Load balance and i can confirm everything you explained above at post #9 is working perfectly.
Very important to set the prio specified in post #11
I noticed that the openvpn clients also have prio by itself, so client 1 is prefered above client 5.

About the hot-standy, you are right the router get an IP but it is not possible to do anything with it, only when you switch to load balance.
Even when you bind the openvpn client to the WAN2 interface , traffic is not routed to the secondary WAN.
For me this is not a good solution because i lose , QOS and some other features i use if i need to be in load balance mode.

So for me i just leave my secondary connection idle, and use it when i have an outage.
If i unplug the coax cable from my cable modem (so ethernet stay alive to the router) it switch to the secondary wan and reconnect the vpn client(s)
If i plug it back my primary comes alive again and everything is back on primary WAN.
So for me this is working, it would just be nice to route something on a idle connection.

Thank you for all the info !