WANFailover Dual WAN Failover Script

[Ranger802004]

With ISP01 Primary WAN, and ISP02 Secondary WAN Hot-Standby, the 4th ip rule allowed me to ping 8.8.4.4 from eth4, but can not ping 8.8.4.4 from eth0

ASUSWRT-Merlin RT-AX88U 386.7_2 Sun Jul 24 21:39:14 UTC 2022
admin@XXAX88U:/tmp/home/root# ip rule add from all to 8.8.8.8 oif eth0 lookup wan0 priority 100
admin@XXAX88U:/tmp/home/root# ping -I eth4 8.8.4.4
PING 8.8.4.4 (8.8.4.4): 56 data bytes
^C
--- 8.8.4.4 ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss

admin@XXAX88U:/tmp/home/root# ip rule add from all to 8.8.4.4 oif eth4 lookup wan1 priority 100
admin@XXAX88U:/tmp/home/root# ping -I eth4 8.8.4.4
PING 8.8.4.4 (8.8.4.4): 56 data bytes
^C
--- 8.8.4.4 ping statistics ---
2 packets transmitted, 0 packets received, 100% packet loss

admin@XXAX88U:/tmp/home/root# ip rule add from all iif lo to 8.8.8.8 lookup wan0 priority 100
admin@XXAX88U:/tmp/home/root# ping -I eth4 8.8.4.4
PING 8.8.4.4 (8.8.4.4): 56 data bytes
^C
--- 8.8.4.4 ping statistics ---
2 packets transmitted, 0 packets received, 100% packet loss

admin@XXAX88U:/tmp/home/root# ip rule add from all iif lo to 8.8.4.4 lookup wan1 priority 100
admin@XXAX88U:/tmp/home/root# ping -I eth4 8.8.4.4
PING 8.8.4.4 (8.8.4.4): 56 data bytes
64 bytes from 8.8.4.4: seq=0 ttl=120 time=31.397 ms
64 bytes from 8.8.4.4: seq=1 ttl=120 time=30.258 ms
64 bytes from 8.8.4.4: seq=2 ttl=120 time=31.275 ms
^C
--- 8.8.4.4 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 30.258/30.976/31.397 ms

admin@XXAX88U:/tmp/home/root# ip -V
ip utility, iproute2-5.11.0
admin@XXAX88U:/tmp/home/root#

That is odd, your router doesn't appear to be properly using the IP rules correctly when you select an interface...that is definitely not right.

 

[JohnSmith]

That is odd, your router doesn't appear to be properly using the IP rules correctly when you select an interface...that is definitely not right.

These two rules seem to make the difference in my setup:

ip rule add from all iif lo to 8.8.8.8 lookup wan0 priority 100
ip rule add from all iif lo to 8.8.4.4 lookup wan1 priority 100

I will some more testing before switching back to 1.5.5 Prod, and see about some outputs from:

ip rule list
ip route show

 

[JohnSmith]

These two rules seem to make the difference in my setup:

ip rule add from all iif lo to 8.8.8.8 lookup wan0 priority 100
ip rule add from all iif lo to 8.8.4.4 lookup wan1 priority 100

I will some more testing before switching back to 1.5.5 Prod, and see about some outputs from:

ip rule list
ip route show

Still using V1.5.6-beta8, was on ISP01(Primary WAN), and ISP02(Secondary WAN - HotStandby) and then rebooted my router, and after the reboot, and all scripts settled down, noticed that I was on ISP02 on startup, and ISP01 was in HotStandBy.

Ran the command:

ping -I eth0 8.8.8.8 - no response

then ran

ip rule add from all iif lo to 8.8.8.8 lookup wan0 priority 100

and then

ping -I eth0 8.8.8.8 - ping response

Dual WAN Script got the response it needed from 8.8.8.8 via eth0, and flipped back over to ISP01(Primary WAN), and ISP02(Secondary WAN - HotStandby)

The only drawback to this is that once that ip rule was added, if you wanted to test switching back and forth using the command:

/jffs/scripts/wan-failover.sh switchwan

You can't do that as that ping response from eth0 will failback the ISP02 back to ISP01.

 

[JohnSmith]

V1.5.6-beta8, was on ISP01(Primary WAN), and ISP02(Secondary WAN - HotStandby) and then rebooted my router a second time to see if it was a coincedence or not, and after the reboot, and all scripts settled down, noticed that I was on ISP02 on startup, and ISP01 was in HotStandBy again.

 

[Ranger802004]

V1.5.6-beta8, was on ISP01(Primary WAN), and ISP02(Secondary WAN - HotStandby) and then rebooted my router a second time to see if it was a coincedence or not, and after the reboot, and all scripts settled down, noticed that I was on ISP02 on startup, and ISP01 was in HotStandBy again.

Yea the router firmware is going to load up on WAN1 by default and then once my script loads it should switch it back to WAN0 (When the pinging is working). For some reason your router is not liking the rules being added with the Outbound Interface being specified) That is the "oif eth0" "oif eth4", are you having the same behavior after a restart?

 

[JohnSmith]

Yea the router firmware is going to load up on WAN1 by default and then once my script loads it should switch it back to WAN0 (When the pinging is working). For some reason your router is not liking the rules being added with the Outbound Interface being specified) That is the "oif eth0" "oif eth4", are you having the same behavior after a restart?

Just rolled back to V1.5.5 Final and tested Failover and Failback and it works in my setup. YazFi is hit and miss, but the Failover and Failback are more important, and YazFi will check within 10 minutes to apply its settings

 

[Ranger802004]

Just rolled back to V1.5.5 Final and tested Failover and Failback and it works in my setup. YazFi is hit and miss, but the Failover and Failback are more important, and YazFi will check within 10 minutes to apply its settings

Yes the problem is the IP Rules on your router for some reason, they were changed in the script because they create conflicts if the outbound interface wasn't specified however your router for some reason is not using them properly and that's why I had you test with the other IP rules manually.

 

[JohnSmith]

Yes the problem is the IP Rules on your router for some reason, they were changed in the script because they create conflicts if the outbound interface wasn't specified however your router for some reason is not using them properly and that's why I had you test with the other IP rules manually.

V1.5.5 Final - ISP01 (Primary WAN), and ISP02(Secondary WAN - Hot-Standby), in V1.5.6-beta8, I could NOT ping eth4 at 8.8.4.4, but I can with this version after a FailBack and my ip rule list is:

admin@XXAX88U:/tmp/home/root# ip rule list
0:      from all lookup local
100:    from all to 8.8.8.8 iif lo lookup wan0
100:    from all to 8.8.4.4 iif lo lookup wan1
32766:  from all lookup main
32767:  from all lookup default

I did not have to apply any:

ip rule add from all iif lo to 8.8.8.8 lookup wan0 priority 100
ip rule add from all iif lo to 8.8.4.4 lookup wan1 priority 100

In order to get pings to either interface to work. I'm stumped unless somebody with an ASUS Merlin AX88U with V386.7_2 FW and the addons I use can run a test with your latest beta software.

 

[Ranger802004]

V1.5.5 Final - ISP01 (Primary WAN), and ISP02(Secondary WAN - Hot-Standby), in V1.5.6-beta8, I could NOT ping eth4 at 8.8.4.4, but I can with this version after a FailBack and my ip rule list is:

admin@XXAX88U:/tmp/home/root# ip rule list
0:      from all lookup local
100:    from all to 8.8.8.8 iif lo lookup wan0
100:    from all to 8.8.4.4 iif lo lookup wan1
32766:  from all lookup main
32767:  from all lookup default

I did not have to apply any:

ip rule add from all iif lo to 8.8.8.8 lookup wan0 priority 100
ip rule add from all iif lo to 8.8.4.4 lookup wan1 priority 100

In order to get pings to either interface to work. I'm stumped unless somebody with an ASUS Merlin AX88U with V386.7_2 FW and the addons I use can run a test with your latest beta software.

I can build in some compensating logic if necessary for this model but I’m hoping someone else has one to test for us

 

[Ranger802004]

v1.5.6-beta9 Release: ***Disclaimer: This is a beta release and has been untested***

Manually upgrade to this beta by running the following command" ***Allow for cronjob to relaunch the script***
Clean installation:

/usr/sbin/curl -s "https://raw.githubusercontent.com/Ranger802004/asusmerlin/main/wan-failover_v1.5.6-beta9.sh" -o "/jffs/scripts/wan-failover.sh" && chmod 755 /jffs/scripts/wan-failover.sh && sh /jffs/scripts/wan-failover.sh install

Upgrade from previous installation:

/usr/sbin/curl -s "https://raw.githubusercontent.com/Ranger802004/asusmerlin/main/wan-failover_v1.5.6-beta9.sh" -o "/jffs/scripts/wan-failover.sh" && chmod 755 /jffs/scripts/wan-failover.sh && sh /jffs/scripts/wan-failover.sh restart

To revert back to Production Release:

/jffs/scripts/wan-failover.sh update

Beta Readme

***WARNING*** If you are using an RT-AX88U, read release notes!

***HIGHLIGHT*** Script will now create an alias as "wan-failover", once script is updated and restarted. Consoles can now use the new alias instead of the full script path "/jffs/scripts/wan-failover.sh". Consoles open while the script is updated may need to be restarted or the following command executed.

source /jffs/configs/profile.add

Release Notes:
v1.5.6-beta9
- General optimization
- Added a confirmation prompt to Restart Mode.
- Fixed visual bugs when running Restart Mode.
- Load Balance Monitor now triggers Service Restart function during failover events.
- YazFi trigger during service restart will no longer run process in the background to prevent issues with script execution of YazFi.
- IP Rules should no longer create conflict with other scripts such as VPNMON.
- Target IPs for both interfaces can now be the same the Target IP.
- Added Recursive Ping Check feature. If packet loss is not 0% during a check, the Target IP Addresses will be checked again based on the number of iterations specified by this setting before determing a failure or packet loss. RECURSIVEPINGCHECK (Value is in # of iterations). Default: 1
- Resolved issues that prevented 4G USB Devices from properly working in Failover Mode.
- Moved WAN0_QOS_OVERHEAD, WAN1_QOS_OVERHEAD, WAN0_QOS_ATM, WAN1_QOS_ATM, BOOTDELAYTIMER, PACKETLOSSLOGGING and WANDISABLEDSLEEPTIMER to Optional Configuration and no longer are required to be set during Config or Installation. They will be given Default values that can be modified in the Configuration file.
- Created new Optional Configured Option to specify the ping packet size. PACKETSIZE specifes the packet size in Bytes, Default: 56 Bytes.
- Resolve issue where script would loop from WAN Status to Load Balance Monitor when an interface was disabled.
- Load Balance Mode will now dynamically update resolv.conf (DNS) for Disconnected WAN Interfaces.
- Fixed Cron Job deletion during Uninstallation.
- Corrected issue with Failure Detected log not logging if a device was unplugged or powered off from the Router while in Failover Mode.
- Modified Restart Mode logic to better detect PIDs of running instances of the script.
- Created Alias for script as wan-failover to shorten length of commands used in console.
- Fixed issue where if the USB Device is unplugged and plugged back in, script will now leave Disabled State to go back to WAN Status.
- Enhanced WAN Disabled Logging, will relog every 5 minutes the condition causing the script to be in the Disabled State.
- Added additional logging throughout script.
- Email function will check if DDNS is enabled before attempting to use saved DDNS Hostname
- Added cleanup function for when script exits to perform cleanup tasks.
- Service Restarts now include restarting enabled OpenVPN Server Instances.
- Target IP Rules will now compensate for the RT-AX88U however this can create conflicts if the Target IPs are the same or are used for other services/scripts.
- Fixed issue in DNS Switch in Load Balance Mode where WAN1 was using the Status of WAN0.
- Switch WAN Mode will now prompt for confirmation before switching.
- Fixed issue where Switch WAN Mode would fail due to missing Status parameters acquired in Run or Manual Mode.

 

[JohnSmith]

v1.5.6-beta9 Release: ***Disclaimer: This is a beta release and has been untested***

Manually upgrade to this beta by running the following command" ***Allow for cronjob to relaunch the script***
Clean installation:

/usr/sbin/curl -s "https://raw.githubusercontent.com/Ranger802004/asusmerlin/main/wan-failover_v1.5.6-beta9.sh" -o "/jffs/scripts/wan-failover.sh" && chmod 755 /jffs/scripts/wan-failover.sh && sh /jffs/scripts/wan-failover.sh install

Upgrade from previous installation:

/usr/sbin/curl -s "https://raw.githubusercontent.com/Ranger802004/asusmerlin/main/wan-failover_v1.5.6-beta9.sh" -o "/jffs/scripts/wan-failover.sh" && chmod 755 /jffs/scripts/wan-failover.sh && sh /jffs/scripts/wan-failover.sh restart

To revert back to Production Release:

/jffs/scripts/wan-failover.sh update

Beta Readme

***WARNING*** If you are using an RT-AX88U, read release notes!

***HIGHLIGHT*** Script will now create an alias as "wan-failover", once script is updated and restarted. Consoles can now use the new alias instead of the full script path "/jffs/scripts/wan-failover.sh". Consoles open while the script is updated may need to be restarted or the following command executed.

source /jffs/configs/profile.add

Release Notes:
v1.5.6-beta9
- General optimization
- Added a confirmation prompt to Restart Mode.
- Fixed visual bugs when running Restart Mode.
- Load Balance Monitor now triggers Service Restart function during failover events.
- YazFi trigger during service restart will no longer run process in the background to prevent issues with script execution of YazFi.
- IP Rules should no longer create conflict with other scripts such as VPNMON.
- Target IPs for both interfaces can now be the same the Target IP.
- Added Recursive Ping Check feature. If packet loss is not 0% during a check, the Target IP Addresses will be checked again based on the number of iterations specified by this setting before determing a failure or packet loss. RECURSIVEPINGCHECK (Value is in # of iterations). Default: 1
- Resolved issues that prevented 4G USB Devices from properly working in Failover Mode.
- Moved WAN0_QOS_OVERHEAD, WAN1_QOS_OVERHEAD, WAN0_QOS_ATM, WAN1_QOS_ATM, BOOTDELAYTIMER, PACKETLOSSLOGGING and WANDISABLEDSLEEPTIMER to Optional Configuration and no longer are required to be set during Config or Installation. They will be given Default values that can be modified in the Configuration file.
- Created new Optional Configured Option to specify the ping packet size. PACKETSIZE specifes the packet size in Bytes, Default: 56 Bytes.
- Resolve issue where script would loop from WAN Status to Load Balance Monitor when an interface was disabled.
- Load Balance Mode will now dynamically update resolv.conf (DNS) for Disconnected WAN Interfaces.
- Fixed Cron Job deletion during Uninstallation.
- Corrected issue with Failure Detected log not logging if a device was unplugged or powered off from the Router while in Failover Mode.
- Modified Restart Mode logic to better detect PIDs of running instances of the script.
- Created Alias for script as wan-failover to shorten length of commands used in console.
- Fixed issue where if the USB Device is unplugged and plugged back in, script will now leave Disabled State to go back to WAN Status.
- Enhanced WAN Disabled Logging, will relog every 5 minutes the condition causing the script to be in the Disabled State.
- Added additional logging throughout script.
- Email function will check if DDNS is enabled before attempting to use saved DDNS Hostname
- Added cleanup function for when script exits to perform cleanup tasks.
- Service Restarts now include restarting enabled OpenVPN Server Instances.
- Target IP Rules will now compensate for the RT-AX88U however this can create conflicts if the Target IPs are the same or are used for other services/scripts.
- Fixed issue in DNS Switch in Load Balance Mode where WAN1 was using the Status of WAN0.
- Switch WAN Mode will now prompt for confirmation before switching.
- Fixed issue where Switch WAN Mode would fail due to missing Status parameters acquired in Run or Manual Mode.

Initial Tests look promising, Failover shutting off ISP01, ISP02 kicks in, and YazFi works. Failback works powering ISP01 back up, and ISP01 takes back over. Still need to test Router power off and on, but looking good so far.

Thanks @Ranger802004 !

P.S. Tested the "switchwan" argument, and it switches from ISP01 to ISP02, but switches right back to ISP01, due to the ping count, timeout, and ip rule being able to ping eth0 8.8.8.8.

Any chance in the next beta version or even next revision alpha/beta, to create the following:

-"failover" argument which only works in failover mode (not LB), and always fails from Primary WAN to Secondary WAN (and removes the ip rule so that ping eth0 8.8.8.8 does not engage the failback automatically)

-"failback" argument which again only works in failover mode (not LB), and always fails from Secondary WAN to Primary WAN (and adds the ip rule back so that ping eth0 8.8.8.8 is setback up for proper failover mode again

 

[Ranger802004]

Initial Tests look promising, Failover shutting off ISP01, ISP02 kicks in, and YazFi works. Failback works powering ISP01 back up, and ISP01 takes back over. Still need to test Router power off and on, but looking good so far.

Thanks @Ranger802004 !

P.S. Tested the "switchwan" argument, and it switches from ISP01 to ISP02, but switches right back to ISP01, due to the ping count, timeout, and ip rule being able to ping eth0 8.8.8.8.

Any chance in the next beta version or even next revision alpha/beta, to create the following:

-"failover" argument which only works in failover mode (not LB), and always fails from Primary WAN to Secondary WAN (and removes the ip rule so that ping eth0 8.8.8.8 does not engage the failback automatically)

-"failback" argument which again only works in failover mode (not LB), and always fails from Secondary WAN to Primary WAN (and adds the ip rule back so that ping eth0 8.8.8.8 is setback up for proper failover mode again

Instead I would recommend killing the monitor with the kill command to manually Failover to Secondary WAN and when you are ready to actively monitor run the cron command to reschedule the script to launch.

 

[Ranger802004]

Instead I would recommend killing the monitor with the kill command to manually Failover to Secondary WAN and when you are ready to actively monitor run the cron command to reschedule the script to launch.

@JohnSmith can you send me debug logging when you get a moment, I'd like to deep dive into why your router is having issues with the IP Rules. You can DM them to me for privacy.

 

[JohnSmith]

@JohnSmith can you send me debug logging when you get a moment, I'd like to deep dive into why your router is having issues with the IP Rules. You can DM them to me for privacy.

Just REDACTING my ISP01 Down DEBUG Log for you. It's long and when ISP01 was powered off, the DEBUG system log took a lot longer to complete its process, not sure if that is the logic in your script, or a loop was created. I'll PM you when done.

 

[Ranger802004]

Just REDACTING my ISP01 Down DEBUG Log for you. It's long and when ISP01 was powered off, the DEBUG system log took a lot longer to complete its process, not sure if that is the logic in your script, or a loop was created. I'll PM you when done.

It’s probably just a side effect from the additional amount of data being gathered.

 

[Ranger802004]

Republished beta9 so you can reinstall with same link. I fixed a couple minor issues regarding quotation and the OpenVPN Server restart target.

 

[JohnSmith]

Republished beta9 so you can reinstall with same link. I fixed a couple minor issues regarding quotation and the OpenVPN Server restart target.

I'll give this a test today to see if it resolves those DEBUG log errors, and continues to Failover and Failback appropriately.

One thing I did notice, that I need somebody else to test is that last night my ISP02 (Secondary WAN - Hot Standby), either upgraded firmware on its router, rebooted, or went down early morning, and when it came back up, it was in Cold Standby, appearing to populate the "Network Map Tab | Secondary WAN". Once I clicked on "Secondary WAN - Cold Standby", and moved the "Internet Connection" slider to "OFF", and then "ON", it populated and then went into "Secondary WAN - Hot Standby". I logged into my ISP02 Router Modem to see if it was ready to go, and it was fully up with IP address, Gateway, Subnet Mask, etc.

I did this test yesterday not really thinking much about it, or reporting it for V1.5.6-beta9, but think this might need to be looked at.

I'll try testing this today as well, and get some logs if need be.

 

[JohnSmith]

I'll give this a test today to see if it resolves those DEBUG log errors, and continues to Failover and Failback appropriately.

One thing I did notice, that I need somebody else to test is that last night my ISP02 (Secondary WAN - Hot Standby), either upgraded firmware on its router, rebooted, or went down early morning, and when it came back up, it was in Cold Standby, appearing to populate the "Network Map Tab | Secondary WAN". Once I clicked on "Secondary WAN - Cold Standby", and moved the "Internet Connection" slider to "OFF", and then "ON", it populated and then went into "Secondary WAN - Hot Standby". I logged into my ISP02 Router Modem to see if it was ready to go, and it was fully up with IP address, Gateway, Subnet Mask, etc.

I did this test yesterday not really thinking much about it, or reporting it for V1.5.6-beta9, but think this might need to be looked at.

I'll try testing this today as well, and get some logs if need be.

Test#01:
ISP01 to ISP02 - Failover SUCCESS, YazFi SUCCESS
ISP02 to ISP01 - Failback SUCCESS, YazFi FAIL (waited 7 minutes for "sh /jffs/scripts/YazFi check" cron job to check and apply YazFi script)

Wishlist - as this version of your script in my setup finishes within 2 minutes, any way of having your script run "sh /jffs/scripts/YazFi check" after 3 minutes


Test#02:
ISP01 (Primary WAN), ISP02 (Secondary WAN - Hot Standby)

1. Power off ISP02 - No DEBUG Logs generated
2. Power up ISP02 - Secondary WAN is in Cold Standby

Aug  5 10:16:23 kernel: eth4 (Ext switch port: 3) (Logical Port: 11) (phyId: b) Link UP at 1000 mbps full duplex
Aug  5 10:16:26 kernel: eth4 (Ext switch port: 3) (Logical Port: 11) (phyId: b) Link DOWN.
Aug  5 10:16:33 kernel: eth4 (Ext switch port: 3) (Logical Port: 11) (phyId: b) Link UP at 1000 mbps full duplex

3. Under "Network Map TAB", clicked on "Secondary WAN - Cold Standby", and moved the "Internet Connection" slider to "OFF", and then "ON", it populated and then went into "Secondary WAN - Hot Standby"

Debug log being sent to you via PM @Ranger802004

 

[Ranger802004]

Test#01:
ISP01 to ISP02 - Failover SUCCESS, YazFi SUCCESS
ISP02 to ISP01 - Failback SUCCESS, YazFi FAIL (waited 7 minutes for "sh /jffs/scripts/YazFi check" cron job to check and apply YazFi script)

Wishlist - as this version of your script in my setup finishes within 2 minutes, any way of having your script run "sh /jffs/scripts/YazFi check" after 3 minutes


Test#02:
ISP01 (Primary WAN), ISP02 (Secondary WAN - Hot Standby)

1. Power off ISP02 - No DEBUG Logs generated
2. Power up ISP02 - Secondary WAN is in Cold Standby

Aug  5 10:16:23 kernel: eth4 (Ext switch port: 3) (Logical Port: 11) (phyId: b) Link UP at 1000 mbps full duplex
Aug  5 10:16:26 kernel: eth4 (Ext switch port: 3) (Logical Port: 11) (phyId: b) Link DOWN.
Aug  5 10:16:33 kernel: eth4 (Ext switch port: 3) (Logical Port: 11) (phyId: b) Link UP at 1000 mbps full duplex

3. Under "Network Map TAB", clicked on "Secondary WAN - Cold Standby", and moved the "Internet Connection" slider to "OFF", and then "ON", it populated and then went into "Secondary WAN - Hot Standby"

Debug log being sent to you via PM @Ranger802004

I got your debug logs and identified the problem, WAN1 is stuck in State 5 and I don't have that state included in my force restart loop for the interface which would bring it out of Cold Standby and back to Hot Standby. Will fix this in the last planned beta for v1.5.6 (Beta10)

 

[Ranger802004]

I got your debug logs and identified the problem, WAN1 is stuck in State 5 and I don't have that state included in my force restart loop for the interface which would bring it out of Cold Standby and back to Hot Standby. Will fix this in the last planned beta for v1.5.6 (Beta10)

@JohnSmith in the meantime you can use these commands to force them into Hot Standby while you wait on my update.
WAN0:

service "restart_wan_if 0"

WAN1:

service "restart_wan_if 1"