Asuswrt-Merlin 388.1 - Wireguard site-to-site on 2xAX88U

[GPSMapper]

Hi All,
I am having 2 x AX88U routers running Merlin FW, installed in 2 remote locations and I was using OVPN server+client on both sides with NAT to enable site to site connectivity for connected LAN devices and occasional routing of both segments to Internet via different ISPs connected to WANs. This configuration was working, but local-to-local performance was not great.
I know there was a comprehensive manual on SNB to configure site-to-site VPN without NAT, but it was requiring some extensive config via shell and it was not so critical so I have lived without it.

Now with 388.1 and Wireguard support being added, I have taken a try to migrate from Openvpn to Wireguard.

First observation:
I have followed this guide: https://www.asus.com/support/FAQ/1048281 (Scenario 3, two-way communication) but connected LANs routing was not working until I have added the VPN director rule on the client side to route local subnet to remote server local subnet over this WG tunnel.

On the server side, no rules were created, remote LAN segment appears to be routed to client without any extra rules.

I can see in the announcement that

Also, by default, no traffic is routed through WireGuard, you need to configure rules in VPNDirector. This is identical to the stock firmware which requires you to create VPN Fusion rules to handle WireGuard client traffic.

So I assume I did everything right and one VPN rule is required on the client side, but NOT required on the server side (actually there is no rule possible while there is no WG client running). Is that correct understanding or did I miss anything?

Second observation: NAT is disabled as per instructions but access to remote WAN of the server is still working (if routing is configured by the additional rule), so it seems remote server is doing NAT for local network clients. Is that a correct understanding?

Third observation: If I need to route some clients on both sides to the WAN interface of the opposite side - do I need to create another pair of Wireguard server + client with NAT enabled and add some rules in VPN director to route the traffic to remote (Scenario 2, one-way communication)?
When I do this, should I change the default settings of wireguard server from 10.6.0.1/32 to some other subnet to avoid same subnet running on WG servers which are interconnected?

 

[ZebMcKayhan]

(Scenario 3, two-way communication) but connected LANs routing was not working until I have added the VPN director rule on the client side to route local subnet to remote server local subnet over this WG tunnel.

You may be correct as this may be the only way to do it with fw implementation. In wgm we created 2 server peers that connect together (only lan 2 lan, no internet). In my opinion lan connection should not be policy routed as this only makes sense when there are more than one route to the same target.

Second observation: NAT is disabled as per instructions but access to remote WAN of the server is still working (if routing is configured by the additional rule), so it seems remote server is doing NAT for local network clients. Is that a correct understanding?

I would say: No. There is a firewall rule on all ASUS WAN connections so even though a packet is tunneled over Wireguard it will still be NAT:ed when leaving any WAN interface. The Wireguard could still be without NAT.

Third observation: If I need to route some clients on both sides to the WAN interface of the opposite side - do I need to create another pair of Wireguard server

Im pretty sure that wont work as you will experience issues with rp-filter.

The ideal setup would be to setup clients on both sides, without nat and enabled inbound firewall. Then adding the lan routes to main routing table and default route to policy table. Then lan would always be routable (as it should) and you could add director rules for internet how you see fit.
Without some scripting I believe that would not be possible.

 

[GPSMapper]

There is a firewall rule on all ASUS WAN connections so even though a packet is tunneled over Wireguard it will still be NAT:ed when leaving any WAN interface. The Wireguard could still be without NAT.

Interesting, so I have WGC1 with NAT disabled. Routing to remote lan works w/o nat (I have checked the logs of remote hosts), but routing of a connected client over the same WG tunnel to remote internet works as well, but NATed somehow in the middle? External IP is detected as remote WAN IP, so as desired.

Im pretty sure that wont work as you will experience issues with rp-filter.

I have tried and it seems working OK. Setup as per diagram in the initial post.
Primary WG server+client are configured as per Scenario 3 of Asus instructions + VPN director rule added for lan-2-lan routing on the client side + some client rules to redirect all traffic to remote (as per above screenshot)
Secondary WG server+ client are configured other way around with Scenario 2 instructions (NAT enabled) but with WG server/client IPs modified to differ from primary WG setup + VPN director rules added for clients on the other side to go through this tunnel in the opposite direction.

So far everything seems to be working fine and I have even stopped OpenVPN servers on both ends to make sure everything could go only over WG tunnels in either way.
But I have not tried restarting the routers yet

 

[ZebMcKayhan]

routing of a connected client over the same WG tunnel to remote internet works as well, but NATed somehow in the middle?

Not in the middle. Without knowing the exact implementation in the fw the normal way would be that communication arrives at server un-nat:ed and are forwarded to server wan interface (still un-nat:ed) but when reaching POSTROUTING chain of the firewall on the server, just before reaching the wan port on the server the source address is changed to match wan ip. Usually the packets will be dropped by the isp if it does not have correct source address.

I have tried and it seems working OK. Setup as per diagram in the initial post.

Alright, cool! So this would mean packets recieved by the server will be replied back over the client... guess that asus/merlin disables reverse path filtering for wireguard interfaces.

But it also mean all clients set to access remote lan will also access internet on remote side?

 

[GPSMapper]

Usually the packets will be dropped by the isp if it does not have correct source address.

this is already too much for me I understand that WG tunnel works over WANs on both ends and client traffic going over the tunnel is encapsulated somehow.
But I'll keep it simple and will call it a 'magic' as soon as it works

But it also mean all clients set to access remote lan will also access internet on remote side?

nope.
primary client side:

Xiaomi 12 (192.168.50.11) and all other clients on 192.168.50.0/24 can access remote lan clients (192.168.1.0/24) without NAT
Only Xiaomi 12 (192.168.50.11) will go to Internet via remote WAN. All other clients will take a default route (local WAN) instead

Secondary client side:

Only specified clients (FireTVs) will access internet over remote WAN
All other local clients (192.168.1.0/24) are taking a default route via local WAN.

But at the same time all local clients 192.168.1.0/24 can access remote lan clients 192.168.50.0/24 without NAT and without any additional rules configured (and without secondary WG tunnel in place).
Probably this routing is controlled by the primary WG server settings (scenario 3, two-way):

p.s. no WAN rules are configured on either side, only WGC

 

[ZebMcKayhan]

I understand that WG tunnel works over WANs on both ends and client traffic going over the tunnel is encapsulated somehow.

Yep, but as communication is recieved at the server end it pops out of the tunnel and when forwarded there (outside the tunnel) to wan there will be NAT as it goes out on the internet, just as magic!

nope

Ooh, I guess I didnt pay attention to all details of your initial post. Thats a clever way of doing it!
I would have proposed to add the lan route under LAN->Route instead but this works too I guess.
I would still think that the reverse path filter would have to be disabled (or set to loose) on all 4 Wireguard interfaces. But ofcource that is no issue for you. Nice work!

 

[Enrico85]

Hi guys, I have a very similar config with 2 router GT-AX6000
I got this bandwidth values using iperf:

VPN OpenVpn [Router server - Router client] (Site to site)
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 188 MBytes 158 Mbits/sec sender
[ 4] 0.00-10.00 sec 187 MBytes 157 Mbits/sec receiver

VPN WireGuard [Router server - win 10 software client]
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-30.00 sec 2.27 GBytes 650 Mbits/sec sender
[ 4] 0.00-30.00 sec 2.27 GBytes 650 Mbits/sec receiver

VPN WireGuard [Router server - Router client] (Site to site)
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-30.00 sec 1.60 GBytes 458 Mbits/sec
[ 4] 0.00-30.00 sec 1.60 GBytes 458 Mbits/sec

LAN (external web) no VPN
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-30.00 sec 3.24 GBytes 927 Mbits/sec sender
[ 4] 0.00-30.00 sec 3.24 GBytes 927 Mbits/sec receiver


Is there any way to get close to the speed get without vpn?

 

[L&LD]

Yes. Use an i3, i5, or i7 custom-built 'router'. You won't get this with the low-power chips used in consumer routers with the equally limited RAM they offer.

 

[ZebMcKayhan]

VPN WireGuard [Router server - Router client] (Site to site)
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-30.00 sec 1.60 GBytes 458 Mbits/sec
[ 4] 0.00-30.00 sec 1.60 GBytes 458 Mbits/sec

You may be able to squeeze out abit more if you disable nat on your wg client. But yea, @L&LD is correct.

 

[Enrico85]

thanks for the answers.
I have another question about the upload speed

this is my config: the router as server and the router as client WireGuard.

All the clients behind the client router go out with the asus router client ip (in this case X.X.27.120)
so the routing seems good but the upload speed is limited as if the web traffic goes through the vpn (about 500 mbit/s)

without vpn I have full bandwidth (about 950 mbit/s)

thanks

 

[L&LD]

Same answer.

 

[Enrico85]

I understand the performance issue, but the routing should work fine.
with openvpn I didn't have this problem

 

[ZebMcKayhan]

I understand the performance issue, but the routing should work fine.

Its not a routing issue there is no way you are having data going through different interfaces for different direction.

But Im curious as well, try to perform the test from a wired lan client instead. Testing on the router could give to low result if router processor cant keep up.

 

[Enrico85]

my config is:
GT-AX6000 as Router WireGuard with local ip 192.168.0.1
GT-AX6000 as Client WireGuard with local ip 192.168.2.1
all the clients behind the 192.168.0.1 came out to the web with the correct gateway public ip address X.X.41.131
all the clients behind the 192.168.2.1 came out to the web with the correct gateway public ip address X.X.27.120
only the traffic from 192.168.0.1 to 192.168.2.1 and from 192.168.2.1 to 192.168.0.1 go through to the vpn

so the speedtest should have the max speed (about 950 mbit/s)

in my case the client 192.168.2.122 (i7 16 GB RAM) wired conncted gets about 900/500 with the Wireguard Asus router client and about 900/900 with the WireGuard software client on win 10.
so the cpu is not the problem because on download I can get the max speed, the problem is only the upload test that I think is going out through the vpn instead of the gateway 192.168.2.1

 

[ZebMcKayhan]

upload test that I think is going out through the vpn instead of the gateway 192.168.2.1

So, you mean that the server you are running these test to recieves packets from one ip and responds to another ip, that does not make any sense. Even if so, there are heaps of protection against this, its just not the way it works.

Did you turn off nat on your wg client?

Even if so I wonder if this is a side-effect of the implemented fc bypass. So you get nat hw accelleration in one direction but not the other. But its just a guess.

 

[Enrico85]

Did you turn off nat on your wg client?
Even if so I wonder if this is a side-effect of the implemented fc bypass. So you get nat hw accelleration in one direction but not the other. But its just a guess.

yes the NAT is off, see attachment.
the upload speed is the same of the vpn tunnel speed, so it's a very strange effect

 

[ZebMcKayhan]

yes the NAT is off, see attachment.
the upload speed is the same of the vpn tunnel speed, so it's a very strange effect

What do you get if you execute this on your Wireguard client router in ssh:

cat /proc/blog/skip_wireguard_network

This may show if it is fc bypass that does this. If so we could try to remove it and see if it makes a difference.

 

[Enrico85]

What do you get if you execute this on your Wireguard client router in ssh:

cat /proc/blog/skip_wireguard_network

This may show if it is fc bypass that does this. If so we could try to remove it and see if it makes a difference.

cat /proc/blog/skip_wireguard_network
192.168.2.0/24

 

[ZebMcKayhan]

cat /proc/blog/skip_wireguard_network
192.168.2.0/24

Try to remove the entry by:

echo "del 192.168.2.0/24" >> /proc/blog/skip_wireguard_network

Make sure its gone by the previous command.
Then redo the speed test...

Note that if you experience issues, just restarting wireguard should set everything back to normal.

 

[Enrico85]

Try to remove the entry by:

echo "del 192.168.2.0/24" >> /proc/blog/skip_wireguard_network

Make sure its gone by the previous command.
Then redo the speed test...

Note that if you experience issues, just restarting wireguard should set everything back to normal.

TOP

but after restarting WG the issue come back