Is static route broken on RT-AX88U and Merlin 386.1?

[Johan Hansen]

Hi,

I'm a self taught network fiddler that has been lurking around here for a while and managed to solve most issues by feeding on the collective knowledge. Thank you all for that. However, this time I'm stuck and need help.

Been spending a lot of time setting up WireGuard VPN spanning multiple LANs in different locations. This is mainly based on Wireguard RPis present on the LAN side of respective site. At it's core all the Wireguard stuff works just fine. My issue is at home where my RT-AX88U refuses to route select remote subnets to my LAN RPi Wireguard client according to static routes stated in Asus Merlin. If I swap out the RT-AX88U with a RT-AC87U with the exact same settings, everything just works. Same thing if I "bypass" the RT-AX88U with a static route on a LAN client, ie laptop.

Attaching a traceroute and netstat -r from respective router in hope of someone being able to spot what's wrong.

Correct trace route on RT-AC87U
traceroute to 192.168.103.36 (192.168.103.36), 64 hops max, 52 byte packets
1 192.168.102.1 (192.168.102.1) 6.360 ms 0.896 ms 0.748 ms ##ROUTER @HOME##
2 192.168.102.180 (192.168.102.180) 1.193 ms 1.249 ms 1.186 ms ##WG GW @HOME##
3 10.6.0.1 (10.6.0.1) 3.496 ms 3.458 ms 3.681 ms ##WG GW @WORK##
4 192.168.101.1 (192.168.101.1) 3.988 ms 3.927 ms 3.710 ms ##ROUTER @WORK##
5 192.168.103.1 (192.168.103.1) 12.634 ms 12.503 ms 12.735 ms. ##ROUTER @REMOTESITE##
6 192.168.103.36 (192.168.103.36) 16.365 ms 13.564 ms 13.484 ms ##WG GW @REMOTESITE##

Broken traceroute on RT-AX88U
traceroute to 192.168.103.36 (192.168.103.36), 64 hops max, 52 byte packets
1 192.168.102.1 (192.168.102.1) 1.416 ms 0.959 ms 0.959 ms ##ROUTER @HOME##
2 192.168.102.1 (192.168.102.1) 3070.653 ms !H 3006.894 ms !H 3035.338 ms !H

Netstat -r on functional RT-AC87U
user@RT-AC87U-2B18:/tmp/home/root# netstat -r
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
default ISP GATEWAY 0.0.0.0 UG 0 0 0 eth0
10.6.0.0 192.168.102.180 255.255.255.0 UG 0 0 0 br0
ISP IP NET * 255.255.240.0 U 0 0 0 eth0
ISP IP NET * 255.255.255.255 UH 0 0 0 eth0
127.0.0.0 * 255.0.0.0 U 0 0 0 lo
169.254.39.0 * 255.255.255.0 U 0 0 0 br0
192.168.101.0 192.168.102.180 255.255.255.0 UG 0 0 0 br0
192.168.102.0 * 255.255.255.0 U 0 0 0 br0
192.168.103.0 192.168.102.180 255.255.255.0 UG 0 0 0 br0

Netstat -r on broken (?) RT-AX88U
user@RT-AX88U-45D0:/tmp/home/root# netstat -r
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
default ISP GATEWAY 0.0.0.0 UG 0 0 0 eth0
10.6.0.0 192.168.102.180 255.255.255.0 UG 0 0 0 br0
ISP IP NET * 255.255.240.0 U 0 0 0 eth0
ISP IP NET * 255.255.255.255 UH 0 0 0 eth0
127.0.0.0 * 255.0.0.0 U 0 0 0 lo
192.168.101.0 * 255.255.255.0 U 0 0 0 br1
192.168.101.0 192.168.102.180 255.255.255.0 UG 0 0 0 br0
192.168.102.0 * 255.255.255.0 U 0 0 0 br0
192.168.103.0 * 255.255.255.0 U 0 0 0 br2
192.168.103.0 192.168.102.180 255.255.255.0 UG 0 0 0 br0
239.0.0.0 * 255.0.0.0 U 0 0 0 br0

I really hope someone can find it in them to sift thru all these numbers and maybe give me a hint for what's going on here, Thanks.

 

[ColinTaylor]

I believe that 192.168.101.x and 192.168.102.x are now used by the guest WiFi networks in 386.1.

 

[Johan Hansen]

Thanks for your quick reply. Thats interesting and I would say the ultimate bad luck on my part. As you can see, I at least made the effort to choose "non standard" subnets. The chance of the guest networks overlapping this blows my mind. Quite some time has been put in to solve this, your explanation makes perfect sense on paper.

I'll investigate and report back. Thanks again.

 

[ColinTaylor]

Those subnets are only used by the first guest network on the 2.4 and 5 GHz band respectively. AFAIK guest networks 2 and 3 are as before.

 

[Johan Hansen]

Yup, everything works just fine now with the guest networks disabled. Without the "self taught" in OP I would probably have been more suspicious about the duplicate instances of the 192.168.101.x and 192.168.102.x networks in the routing tables. You live and you learn. On the bright side I figured out all the Wireguard and router shenanigans, but I did not see this coming.

Colin, thank you. Without these bits on information I would for sure gone mad by the end of the weekend. Cheers!

 

[eibgrad]

To be honest, I find your description of these static routes and how they fit into the bigger picture of your router and (apparently) multiple WG clients pretty vague. Might have helped if you have a lot more specific w/ the details, perhaps even a diagram. Very hard to work backwards from dumps of the routing table or traceroutes to figure it all out.

P.S. I see you've apparently fixed it, good to hear.

 

[Johan Hansen]

To be honest, I find your description of these static routes and how they fit into the bigger picture of your router and (apparently) multiple WG clients pretty vague. Might have helped if you have a lot more specific w/ the details, perhaps even a diagram. Very hard to work backwards from dumps of the routing table or traceroutes to figure it all out.

P.S. I see you've apparently fixed it, good to hear.

I agree, it's a bit vague. This is a fairly complicated setup with three Asus routers, one OPNsense router and bunch of RPis. Since I managed to boil it down to a static route issue on one of the routers, I figured I'll start with just that bit somewhat isolated. I hope this thread prevents some poor souls using 192.168.101.0/24 and 192.168.102.0/24 from going bonkers.