Splitting Wireguard between Router and client

[archiel]

So much closer. IPv4 now works but IPv6 does not, even if I add equivalent route for the ipv6 alias (I have assumed the command is otherwise the same).

Going back to basics and taking down wg13 and removing the aliases from unbound, and running ping from the router

ping -6 sunet.se -I <wan_ipv6>

All is good
However if I try to ping via the alias

ping -6 sunet.se -I <wan_ipv6_alias>

I get no response at all

The aliases are added from

Spoiler: wan-event

if [ "$1" = "0" ] && [ "$2" = "connected" ]; then
  ifconfig br0:1 192.168.3.1 netmask 255.255.255.255
  ip -6 address add dev eth5 fd36:7ef1:2add:aa88:100::1/128
  fi

and my initial thought was that maybe I could no longer use eth5 (on the RT-AX88U it is the bridge between ports 0-3 and4-7, so it is always up) so I removed the alias from eth5 and added directly to br0 (I do not do this at boot, due to the ipv6 components ''coming in late'). However I still could not ping via the alias.

All of which makes me think that although an ipv4 alias works, an ipv6 alias does not (or I need do do something else to get it to work).

If I do enable unbound alias and wg13, then check from my desktop (not the VM) in any dns check tool (e.g. on browserleaks.com), only the ipv4 DNS server shows up and if I run test-ipv6.com it get told

Your DNS server (possibly run by your ISP) appears to have no access to the IPv6 Internet, or is not configured to use it.

which would seem to corroborate this.

 

[ZebMcKayhan]

if I add equivalent route for the ipv6 alias (I have assumed the command is otherwise the same).

It would be:

ip -6 route add fd36:7ef1:2add:aa88:100::1 dev eth0 table main

So, yea, pretty much the same.

But ipv6 routing works differently with auto-discovery...

What if you use tcpdump and log pings again but ipv6... still only echo pings on eth0 and ping+reply on vpn if?

 

[archiel]

Updated the route but does not work

If I run an ipv6 ping on the router
ping -6 sunet.se -I <wan_ipv6>
I can see the traffic but nothing shows on the VM
tried with
tcpdump -i eth0 icmp
tcpdump -i eth0 icmp6
tcpdump -i eth0 ip6
and
tcpdump ip6

Separately, I am trying to add back passthru via the dummy wg13 (was previously on wg11, now deleted) but keeps failing

E:Option ==> peer wg21 passthru add wg13 pho21

        ***ERROR: Peer (wg13) must be defined in 'Policy mode' e.g. 'Auto=P'

Client  Auto  IP                                               Endpoint           DNS                                   MTU   Annotate
wg13    P     10.0.60.162/32,2a0e:1c80:1337:1:10:0:60:162/128  10.99.88.77:51880  10.50.60.1,fe80::aa5e:45ff:feae:5050  1384  # Dummy

        Selective Routing RPDB rules
ID  Peer  Interface  Source                      Destination  Description
14  wg13  VPN        fd36:7ef1:2add:aa88:100::1  Any          Unbound6VPN
13  wg13  VPN        192.168.3.1                 Any          Unbound4VPN


Server  Auto  Subnet                                       Port   Annotate
wg21    Y     10.50.1.1/24,aa36:7ef1:2add:aa88:100::1/120  11501  # RT-AX88U (IPv4/IPv6) Server 1


        Configuration rules for Peer wg21

Keeps saying wg13 needs policy mode, but also showing that it is in policy mode.

 

[ZebMcKayhan]

Updated the route but does not work

If I run an ipv6 ping on the router
ping -6 sunet.se -I <wan_ipv6>
I can see the traffic but nothing shows on the VM
tried with
tcpdump -i eth0 icmp
tcpdump -i eth0 icmp6
tcpdump -i eth0 ip6
and
tcpdump ip6

So, the packet probably doesn't reach ubuntu machine, so the problem is probably on the router. But what???

Separately, I am trying to add back passthru via the dummy wg13

Just a heads up, you may also need to add routes to wgm server network on the Ubuntu machine.

but keeps failing

Weird, I'll have to have a look at wgm code to understand what happens. Too tired tonight, maybe tomorrow...

 

[archiel]

So, the packet probably doesn't reach ubuntu machine, so the problem is probably on the router. But what???



Just a heads up, you may also need to add routes to wgm server network on the Ubuntu machine.



Weird, I'll have to have a look at wgm code to understand what happens. Too tired tonight, maybe tomorrow...

tomorrow is great - at least we are making progress. Thank you.

 

[archiel]

Even closer

I now have a working 'solution' for ipv6, in that if I run
ping -6 sunet.se -I <new_ipv6_alias> on the router, I can see the responses from tcpdump -i eth0 icmp6 on the VM
and where <new_ipv6_alias> = <wg21_tunnel_ipv6>::9
In addition any non-VPN lan client is now showing the VPN DNS servers for IPv4 and IPv6 and ipv6-test is ticking all the boxes

By trial and error I discovered that if I tried to ping from the router using -I <wg21_tunnel_ipv6>::1 it worked.
I then added an unused address from the tunnel subnet to eth5

ip -6 address add dev eth5 <wg21_tunnel_ipv6>::9/128

To wg13

peer wg13 rule add <wg21_tunnel_ipv6>::9 any

to unbound

outgoing-interface: <wg21_tunnel_ipv6>::9

I did not need to add any additional ipv6 routing on the VM.

As far as I have tested (I will do more over the weekend) I have not opened up any unwanted routes into the router/lan. I am sure there is a proper way of doing this, but for now this works, apart from working our why I cannot add the passthru.

EDIT

After rebooting the router and VM, connectivity was SNAFU. I needed to stop wg13 restart unbound rs nocache and restart wg13.

I will need to play around with when to add

ip -6 address add dev eth5 <wg21_tunnel_ipv6>::9/128

as this currently sits in wan-event. Will try moving to wg21-up.sh (or possibly) wg13-route-up.sh

In terms of speed, my 1SP is notionally 1GB up/down, in practice approx 940/930 direct and 600/540 via Wireguard on the VM.

 

[ZebMcKayhan]

By trial and error I discovered that if I tried to ping from the router using -I <wg21_tunnel_ipv6>::1 it worked.
I then added an unused address from the tunnel subnet to eth5

Yea, I guess you could add any AA-type address as this is global as far as the router concerned. Seems like you stumbled on the same issue once again.

I did not need to add any additional ipv6 routing on the VM

As I said, ipv6 routing works differently with auto-discovery so it may not be needed.

I am sure there is a proper way of doing this, but for now this works,

Hey, if it works, it works. If it routes, it routes...

I will need to play around with when to add

Probably better to select a unique AA address for Alias and unbound to use. It may make it easier.

In terms of speed, my 1SP is notionally 1GB up/down, in practice approx 940/930 direct and 600/540 via Wireguard on the VM.

That's great! And no syslog msg so far??

apart from working our why I cannot add the passthru.

Yea, I think I've found the issue but not sure. I'm not the best script reader/debugger.

As per your command:

peer wg21 passthru add wg13 pho21

As far I can see they end up in the following variables:

$ACTION=$1 (peer)
$WG_INTERFACE=$1 (wg21)
$CMD=$1 (add)
$IFACE=$1 (wg13)
$IP_SUBNET=$1 (pho1)

And at line 3100 - 1304:

if [ "$(sqlite3 $SQL_DATABASE "SELECT auto FROM clients WHERE peer='$WG_INTERFACE';")" != "P" ];then

    echo -e $cBRED"\a\n\t***ERROR: Peer (${cBMAG}$IFACE${cBRED}) must be defined in 'Policy mode' e.g. $cRESET'Auto=P'"$cRESET
    Show_Peer_Config_Entry "$IFACE"
  return 1
fi

as far as I can see, the variable used at line 3100 should be $IFACE and not $WG_INTERFACE
it queries the sql database for if wg21 is auto=P which of course its not and never have been.
Why it's used to work but not now I don't know. But this error string is only found in dev branch and not main branch so it appears new. Perhaps @Martineau could elaborate?

for now you could edit the script manually to get your rules added.

 

[archiel]

Re passthru, I will edit the script and let you know.

Regard my solution, i rebooted the pc with the VM and when it came back up everything broke - so I will need to go through each step to see what I missed / failed to save.

In regard to using 'any' aa address, I am not sure of this. I had tried this earlier (adding to eth5 / br0 and pinging on the router) and I did not get any replies other that with the wg21 tunnel address, so I think it may not be that simple - I will revisit everything in the morning.

 

[archiel]

As per your command:

peer wg21 passthru add wg13 pho21

As far I can see they end up in the following variables:

$ACTION=$1 (peer)
$WG_INTERFACE=$1 (wg21)
$CMD=$1 (add)
$IFACE=$1 (wg13)
$IP_SUBNET=$1 (pho1)

And at line 3100 - 1304:

if [ "$(sqlite3 $SQL_DATABASE "SELECT auto FROM clients WHERE peer='$WG_INTERFACE';")" != "P" ];then

echo -e $cBRED"\a\n\t***ERROR: Peer (${cBMAG}$IFACE${cBRED}) must be defined in 'Policy mode' e.g. $cRESET'Auto=P'"$cRESET
Show_Peer_Config_Entry "$IFACE"
return 1
fi

as far as I can see, the variable used at line 3100 should be $IFACE and not $WG_INTERFACE
it queries the sql database for if wg21 is auto=P which of course its not and never have been.
Why it's used to work but not now I don't know. But this error string is only found in dev branch and not main branch so it appears new. Perhaps @Martineau could elaborate?

The edit worked and passthru is enabled.

 

[archiel]

HI @ZebMcKayhan. All is up and running happily. While I will want to spend some time sorting out the best order for shutting down / starting up the services on the Router / VM when rebooting the VM host, this will be specific to my configuration and unlikely to be of use to anyone else.

However the main issues are solved: I wanted the ability to run WireGuard in dual stack mode which means WGM and I wanted the non-VPN router clients to be able to run at full speed*. In regard to the former, while @RMerlin may eventually add dual-stack support to VPNs, I would expect this be a long way down his list of priorities and even then I would expect OpenVPN to be the initial candidate. In regard to the latter, while the fc bypass appears to work on many of the newer routers, all the attempts I have run on my RT-AX88U (including stripping off all my scripts and reverting to IPv4 only) failed.

Running the WGM server component on the router, along with a WGM dummy client routed through to a live WG client running on a ubuntu VM gives me working server, including passthru, optimal speeds and unbound DNS queries being router through the VM VPN. Unless there is anything else you or anyone else (@Martineau ?) would like me to look at, I think this thread has run its course. Many thanks for all your help.

* And I wanted to do this with my existing hardware.

 

[ZebMcKayhan]

All is up and running happily. While I will want to spend some time sorting out the best order for shutting down / starting up the services on the Router / VM when rebooting the VM host, this will be specific to my configuration and unlikely to be of use to anyone else.

Glad we made it work!

Whenever you have the time, would you mind posting here everything you done to make this work, so others could benefit more easily? Both for ubuntu machine and router.

//Zeb

 

[archiel]

Splitting WireGuard between Router (RT-AX88U) and Client (Ubuntu VM running on Hyper-V)

Hi @ZebMcKayhan, sorry for the delay in getting back and thanks again for your invaluable support in getting this over the line

I wanted to

1. use my VPN setup (WireGuard) in dual stack mode with full support for IPv4 and IPv6.
2. run in server mode (for remote access to LAN, for "road warriors" e.g. phones, laptops, tablets)
3. run in client mode on some devices (for obfuscation)
4. route Unbound queries though the VPN client for the Ubuntu VM (for obfuscation)
5. allow remote clients to access the internet via the same VPN client (pass-through mode)
6. Use the routers hardware acceleration (Flow Cache enabled)

The Issue

1-5 above all work fine using @Martineau's WGM with Flow Cache off. However when I changed ISP and increased the notional WAN connection from 80Mb/20Mb to 1Gb/1Gb this limited my WAN speeds to 420/550 for all devices, as opposed to at least 930/930 with Flow Cache on. However with Flow Cache enabled, after 20-40 minutes the speed would drop back to 420/550 or less.

Although I was (and am) using v4.19b5 of WGM, which should allow the cache bypass for non-WireGuard clients, this does not work on my RT-AX88U. While I understand that the cache bypass may work on other models, I had previously tested on my router by removing all scripts other than Wireguard and disabling IPv6 - it did not help.

On testing it has been noted that running WGM in server mode with Flow Cache enabled does not cause any practical issues, the issue is with client mode.

The solution

To continue to run the VPN in server mode on the Router, run the client VPN on the Ubuntu VM and connect the two so that 'road-warrior' pass-through traffic still access the internet via the VPN on the VM and that unbound queries are similarly routed. The connection is a dummy VPN client that just points to the VM.

While I learned this the hard way (repeated failures) the easiest way to do this is to disable the WGM client on the router before proceeding with the setup.

In addition the VPN on the VM (or chosen device) should be up and running BEFORE enabling the dummy VPN. So, if I need to restart the VM or reboot its host, I will stop the dummy client on the router, reboot/restart the host/VM and then restart the dummy VPN. I also then restart Unbound and clear the cache.

On the VM

Firewall: As the default Ubuntu client VM for Hyper-V does not come with a firewall enabled, I installed ufw (uncomplicated firewall) following this guide (https://devtutorial.io/how-to-set-up-a-firewall-with-ufw-on-ubuntu-23-10-p3193.html) - there are plenty of others as well. I then added gufw (a gui for ufw)

Once the firewall was up I then added to necessary rules to allow access to its services (as necessary) for the other devices on the LAN that were using them and once I was happy that the LAN traffic was working as before (installing) the firewall), I moved onto WireGuard.

WireGuard: As I am not using a LTE for my setup I first chose to add the repository

sudo add-apt-repository pp:wireguard/wireguard

, then installed WireGuard itself

sudo apt install wireguard

and then ran the installer for my VPN provider (in my case AzireVPN).

In order to allow IP forwarding (for internet access)

sudo nano /etc/sysctl.conf
net.ipv4.ip_forward=1
net.ipv6.conf.all.forwarding=1

and to make it live

sudo sysctl -p

The next steps are to check the public network interface name

ip route list default

in my case

default via <router LAN ipv4> dev eth0 proto dhcp src <VM LAN ipv4> metric 100

so eth0 and then add the routing rules to the VPN as it comes up / is taken down

sudo nano /etc/wireguard/<wg_client>.conf

[Interface]
PrivateKey = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx=
Address = 10.0.xx.xx, 2a0e:xxxx:xxxx:1:10:0:xx:xx
DNS =  10.0.0.1, 2a0e:xxxx:xxxx:1:10:0:0:1, xxx.xxx.xxx.xx

PostUp = ufw route allow in on eth0 out on <wg_client>
PostUp = iptables -t nat -I POSTROUTING -o <wg_client> -j MASQUERADE
PostUp = ip6tables -t nat -I POSTROUTING -o <wg_client> -j MASQUERADE
Postup = ip route add <ip4_alias> dev eth0 table main
PreDown = ufw route delete allow in on eth0 out on <wg_client>
PreDown = iptables -t nat -D POSTROUTING -o <wg_client> -j MASQUERADE
PreDown = ip6tables -t nat -D POSTROUTING -o <wg_client> -j MASQUERADE
PreDown = ip route del <ip4_alias> dev eth0 table main

[Peer]
PublicKey = yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy=
Endpoint = xxxx.azirevpn.net:nnnnn
AllowedIPs = 0.0.0.0/0, ::/0

where <ip4_alias> is set on the router, see below.

In setting up the interface between the firewall and WireGuard, i found these pages helpful.

How To Set Up WireGuard on Ubuntu 20.04 | DigitalOcean

In this tutorial, you will set up WireGuard on an Ubuntu 20.04 server, and then configure another machine to connect to it as a peer using both IPv4 and IPv6…

www.digitalocean.com

https://www.cyberciti.biz/faq/how-to-set-up-wireguard-firewall-rules-in-linux/

On the Router

Where both server and client aspects of WireGuard can be run on the router, the method for routing unbound queries via a client instance is

1. create IPv4 and IPv6 aliases, <ipv4_alias> and <ipv6_alias>
2. add the to wan-event
3. add these aliases to unbound
4. add the modified IPv6 alias to the WireGuard server config <wg2x>
5. add the routing to WireGuard client <wg1x>

in wan-event

    if [ "$1" = "0" ] && [ "$2" = "connected" ]; then
      ifconfig br0:1  <ipv4_alias> netmask 255.255.255.255
      ip -6 address add dev eth5 <ipv6_alias>/128
      fi

then add these interfaces in unbound.conf under server:

outgoing-interface: <alias_4>         # routing to wan-event + wgm policy rules
outgoing-interface: <alias_6>         # routing to wan-event + wgm policy rules

and in WGM for wg21

peer wg21 ip=10.50.1.1/24 ipv6=aa36:7ef1:xxxx:aa88:100::1/120

where the ipv6 address is the <ipv6_alias> with the first two characters fd changed to aa and add

peer wg1x rule add <alias_4>  comment Unbound4VPN
peer wg1x rule add <alias_6>  comment Unbound6VPN

for the client routing


Aliases:

Aliases can be used to mask the router's IP address for DNS purposes by using additional link-local addresses on different subnet ranges to the default router subnets. Ideally these aliases would be added to br0 while the router is booting, using wan-event. However the IPv6 interfaces load slower than the IPv4 ones, so a more practical solutions is to add the ipv6 alias to another active interface. For the RT-AX88U eth5 can be used as it is always up, being the link to the secondary 4 port switch.

For <ipv4_alias>, this just needs on a separate subnet to the main LAN, but for ease or reading I use 10.xxx.xxx.xxx for my LAN network and then use 192.168.xxx.xxx for the IPv4 alias.

For dynamic IPv6 the starting point for the aliases is to use a ULA generator

1. create a dummy GUA, e.g. fd36:7ef1:2add:xxxx::/64
2. change fd to aa : aa36:7ef1:2add:xxxx::/64 # <wg2x_6_scope>
3. Remove :/64, add 100::1/120 : aa36:7ef1:2add:xxxx:100::1/120
4. Add to the server up/down scripts - see wg21-up.sh

Where
fd36:7ef1:2add:xxxx:100::1 is <ipv6_alias> and aa36:7ef1:2add:xxxx:100::1 is the modified IPv6 alias
and in wg21-up.sh

#!/bin/sh
###############################################################################
# Example for Wg21 ipv6 = aa00:aaaa:bbbb:cccc:100::1/120
# Change to your needs but keep formatting
Wg21Prefix=aa36:7ef1:2add:xxxx:: #Wg21 ULA prefix with aa instead of fd
Wg21Suffix=100::1  #Wg21 Device suffix (last 64 bits)
Wg21PrefixLength=120   #Wg21 Prefix Length (120 recommended)
WanInterface=eth0
# Changing below lines should not be needed:
WanIp6Prefix=$(nvram get ipv6_prefix)     #WanIp6Prefix=2001:1111:2222:3333::
Wg21_PrefIp=${Wg21Prefix%:*}${Wg21Suffix}/${Wg21PrefixLength}      #aa00:aaaa:bbbb:cccc:100::1/120
WanWg21_PrefIp=${WanIp6Prefix%:*}${Wg21Suffix}/${Wg21PrefixLength}   #2001:1111:2222:3333:100::1/120
ip6tables -t nat -I POSTROUTING -s ${Wg21_PrefIp} -o ${WanInterface} -j MASQUERADE -m comment --comment "WireGuard 'server'"

- the last line is modified from -I to -D in wg21-down.sh to remove the POSTROUTING rule.

END OF PART 1

 

[archiel]

PART 2:

Adding and configuring the dummy connection

When moving the VPN client from the Router to the VM (or other external device) changes are needed in wan-event and wg2x-up.sh

After some trial and error, I found that the old <ipv6_alias> would not work and that the new <ipv6_alias> needed to be on the same subnet as the modified IPV6 alias but using an unallocated address. As I have devices allocated to 1 through 6, I chose aa36:7ef1:2add:xxxx:100::9. In wan-event commented out

# ip -6 address add dev eth5 fd36:7ef1:2add:xxxx:100::1/128

and added a new last line to wg2x-up.sh

ip -6 address add dev eth5 aa36:7ef1:2add:xxxx:100::9/128

I have no idea why the alias now (appears) to need to be on the same subnet as the WGM server, but as is does it also needs to be added after the WGM server is up - hence the move to wg2x-up.sh

The new alias (e.g. aa36:7ef1:2add:xxxx:100::9) also need to be substituted in the unbound.conf and wg1x peer rules.

The dummy connection itself can be any client that points nowhere, so here the endpoint is 10:99:88:77 and in my case I created wg13 which looks like

E:Option ==> peer wg13

Client  Auto  IP                                               Endpoint           DNS                                   MTU   Annotate
wg13    P     10.0.60.162/32,2a0e:1c80:1337:1:10:0:60:162/128  10.99.88.77:51880  10.50.60.1,fe80::aa5e:45ff:feae:5050  1384  # Dummy

        Selective Routing RPDB rules
ID  Peer  Interface  Source        Destination  Description
16  wg13  VPN        <ipv6_alias>  Any          Unbound6VPN
13  wg13  VPN        <ipv4_alias>  Any          Unbound4VPN

Server  Client  Passthru
wg21    wg13    pho21
....

And the DNS should be the router's local LAN addresses.

To route the traffic from the dummy config (e.g. wg13) start the dummy and locate the routing table used

ip rule

admin@Router:/tmp/home/root# ip rule
0:      from all lookup local
9810:   from all fwmark 0xd2 lookup 210
9931:   from <ipv4_alias> lookup 123
9983:   from 10.50.1.2 lookup 123
32766:  from all lookup main
32767:  from all lookup default

In my case the table is 123 and the following routes need to be added to wg13-up.sh (and deleted in wg13-down.sh)

ip route add 0.0.0.0/2 via <ubuntu machine ipv4> dev br0 table 123
ip route add 64.0.0.0/2 via <ubuntu machine ipv4> dev br0 table 123
ip route add 128.0.0.0/2 via <ubuntu machine ipv4> dev br0 table 123
ip route add 192.0.0.0/2 via <ubuntu machine ipv4> dev br0 table 123
ip -6 route add 0000::/2 via <ubuntu link-local> dev br0 table 123
Ip -6 route add 4000::/2 via <ubuntu link-local> dev br0 table 123
ip -6 route add 8000::/2 via <ubuntu link-local> dev br0 table 123
ip -6 route add C000::/2 via <ubuntu link-local> dev br0 table 123

and finally testing was carried out by installing tcpdump on the VM and looking at the icmp interface

sudo tcpdump -i eth0 icmp

and from the router pinging a known site, e.g.

ping google.com -I 192.168.3.1

If all is well the router pings are echoed on the VM the equivalent tests should be carried out for IPv6 to check that it is also working.
After that it was a matter of sanity tests - is IPV6 working correctly, what DNS is showing on any DNS checking tool (e.g. browserleaks.com/ip) for a LAN device, from a road warrior device, port scans, DNSSec checks, etc.

Finally, thanks @ZebMcKayhan for getting me through this and would you let me know if the above is clear enough or needs any amendmemt.

 

[ZebMcKayhan]

Finally, thanks @ZebMcKayhan for getting me through this and would you let me know if the above is clear enough or needs any amendmemt.

Thank you for the comprehensive write-up. Quite the setup you got going on! I don't think anything is left out.

Maybe the most useful part of all this is to be able to perform policy routing on the router while using an external machine for Wireguard which could be a raspberry pi or similar to boost vpn performance.

This would be possible for firmware implementation and VPNDirector as well, but limited to ipv4 only (not sure where fw wg is on ipv6 at current time).