Splitting Wireguard between Router and client

[ZebMcKayhan]

do I also need to add table 123 after br0?

Yes, sorry I forgot! Fixed!

 

[archiel]

If I have understood this, the steps we want are as follows

          1                             2              3                 4
---------------------------     -----------------   ----------      ---------------
Traffic from LAN or Router | >  |Dummy WG Client| > |Ubuntu VM| >  |wg client on VM| > Internet
---------------------------     -----------------   ----------      ---------------

as WGM running in server mode does not appear to be a problem and traffic originating on the VM will simply use wg on ubuntu.

As such we are simply aiming to route

• Pass-thru traffic from external clients (phones, laptops, etc) in thru the Router and out through the ubuntu VM, and
• DNS requests via unbound (running on router) and out through the ubuntu VM

For Stage 2 we have now setup the Dummy WGM client (derived from the working WGM client but with a dummy port) and have the following routing tables
ip route show table 123

0.0.0.0/2 via 10.50.60.150 dev br0
0.0.0.0/1 dev wg13  scope link
10.50.60.0/24 dev br0  proto kernel  scope link  src 10.50.60.1
64.0.0.0/2 via 10.50.60.150 dev br0
128.0.0.0/2 via 10.50.60.150 dev br0
128.0.0.0/1 dev wg13  scope link
192.0.0.0/2 via 10.50.60.150 dev br0

and
ip -6 route show table 123

2a02:6b67:xxxx:yyyy::/56 dev br0  proto kernel  metric 256  pref medium
::/2 via fe80::eae5:3390:e03f:f055 dev br0  metric 1024  pref medium
4000::/2 via fe80::eae5:3390:e03f:f055 dev br0  metric 1024  pref medium
::/1 dev wg13  metric 1024  pref medium
8000::/2 via fe80::eae5:3390:e03f:f055 dev br0  metric 1024  pref medium
fe80::/64 dev br0  proto kernel  metric 256  pref medium
c000::/2 via fe80::eae5:3390:e03f:f055 dev br0  metric 1024  pref medium
8000::/1 dev wg13  metric 1024  pref medium

Question: do I need to change the metric for the

ip -6 route add 0000::/2 via <ubuntu link-local> dev br0 table 123, etc

rules?

If this is working it should now direct the output from the dummy WGM client (wg13) to the LAN connection on the Ubuntu VM (eth0) - is this correct?

I did notice that when I stop the dummy client I get

E:Option ==> stop wg13 debug

        Requesting WireGuard® VPN Peer stop (wg13)

[#] ip link del dev wg13
[+] wg13-down.sh
RTNETLINK answers: No such process
RTNETLINK answers: No such process
RTNETLINK answers: No such process
RTNETLINK answers: No such process
RTNETLINK answers: No such process
RTNETLINK answers: No such process
RTNETLINK answers: No such process
RTNETLINK answers: No such process

Question: do you think this is related to the metric question above or otherwise a problem?

Stage 3 Firstly the traffic from the WGM dummy needs to be accepted on the Lan port on the ubuntu VM (eth0) and then pass through the firewall (it is iptables and can be configured directly and/or with ufw)

Coming back to

As you enable the firewall you will also need to add accept -i <lan if> to -o <wg if> in FORWARD filter table and -i <wg if> -o <lan if> state RELATED, ESTABLISHED in the other way. And drop everything else. There are several tutorials on this online but I don't even know what firewall your machine is using. I only know iptables.

Running ifconfig on the VM it would appear that <lan if> and <wg if> are eth0 and azirevpn-nl-ams respectively. I found the following article how-to-set-up-wireguard-on-ubuntu-20-04#step-5-configuring-the-wireguard-server-s-firewall and while the article is looking at configuring a server instance, I think it may also work, with a few amends, for routing traffic though the client instance on the VM.

In the example given for setting up a server instance on ubuntu for the firewall I would add
sudo ufw allow <server_port>/udp

but in this setup there is also the <dummy_wgm_port> so to allow this traffic though the firewall I presumably need to run
sudo ufw allow <dummy_wgm_port>/udp

Question: do you think it would matter if I added both ports to the firewall?

In respect of adding / removing rules when the ubuntu VPN starts / stops, it appear that this is done by editing the azirevpn-nl-ams.conf file. In the example given for a server instance it shows

/etc/wireguard/wg0.conf
. . .
PostUp = ufw route allow in on wg0 out on eth0
PostUp = iptables -t nat -I POSTROUTING -o eth0 -j MASQUERADE
PostUp = ip6tables -t nat -I POSTROUTING -o eth0 -j MASQUERADE
PreDown = ufw route delete allow in on wg0 out on eth0
PreDown = iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE
PreDown = ip6tables -t nat -D POSTROUTING -o eth0 -j MASQUERADE

so do I invert this for a client instance

/etc/wireguard/azirevpn-nl-ams.conf
. . .
PostUp = ufw route allow in on eth0 out on azirevpn-nl-ams
PostUp = ....
PostUp = ....
PreDown = ufw route delete allow in on eth0 out on azirevpn-nl-ams
PreDown = .....
PreDown = ...

and do you think the PostUp / PreDown should be in the

iptables -t nat -I POSTROUTING -o azirevpn-nl-ams -j MASQUERADE as above or
iptables -t nat -I POSTROUTING ! -s eth0 -o azirevpn-nl-ams -j MASQUERADE as previously suggested.

I realise there is a lot here and it will need to be tested in chunks, but I thought setting out (what I think of as) the stages could be useful, as if they are wrong then they can be amended before digging a bigger hole proceeding further.

 

[ZebMcKayhan]

Question: do I need to change the metric for the

ip -6 route add 0000::/2 via <ubuntu link-local> dev br0 table 123, etc

rules?

Typically: no. As I have understood it, metric are only used when there are same route with same specificity. You don't.

Question: do you think this is related to the metric question above or otherwise a problem?

Not metric problem. Probably some typo or perhaps wgm removes the entirely routing table so you don't need to remove the routes???

Question: do you think it would matter if I added both ports to the firewall?

As ubuntu is running a client no ports needs forwarding or allowed. The tunnel should just work.

and do you think the PostUp / PreDown should be in the

iptables -t nat -I POSTROUTING -o azirevpn-nl-ams -j MASQUERADE as above or
iptables -t nat -I POSTROUTING ! -s eth0 -o azirevpn-nl-ams -j MASQUERADE as previously suggested.

Yea, that would work. Not sure about the ufw command though as I've never used them.

I realise there is a lot here and it will need to be tested in chunks, but I thought setting out (what I think of as) the stages could be useful, as if they are wrong then they can be amended before digging a bigger hole proceeding further.

Looks like you are making good progress, hang in there.

Try to figure out ways to test different parts so you can know where to focus.

 

[archiel]

Going right back to the start and

where do I start?

This could be done in so many ways though... to ease the burden of scripting too much I'm thinking something like setting up a dummy client in wgm with bogus data, then hijack that policy route table and point the default route to be to your Ubuntu machine instead. Then you could still use wgm for policy rules. Even passthrough rules if you wish... and unbound of course.
Not sure if wgm would give us trouble with the bogus client though.

It seems I was overly optimistic with the bogus client. What I did was to take the old working (azireVPN) client and just change the port. This allowed the client to load, but each time I did load it for any period of time it seemed to mess up the network, (unbound would report as working and I could see DNS lookups / responses) but the LAN clients were not resolving site names. the ubuntu VM 'network icon' was indicating the network was not functioning, etc.

So to go back to basics I modified the dummy client (wg13) to point to a completely bogus site

E:Option ==> 8 wg13

Client  Auto  IP                                               Endpoint                   DNS                                   MTU   Annotate
wg13    P     10.0.60.162/32,2a0e:1c80:1337:1:10:0:60:162/128  dummy.azire-vpn.net:51880  10.50.60.1,fe80::aa5e:45ff:feae:5050  1384  # Dummy

        Selective Routing RPDB rules
ID  Peer  Interface  Source                      Destination  Description
14  wg13  VPN        fd36:7ef1:2add:aa88:100::1  Any          Unbound6VPN
13  wg13  VPN        192.168.3.1                 Any          Unbound4VPN

but the immediate result was

start wg13

        Requesting WireGuard® VPN Peer start (wg13)

        wg_manager-clientwg13: Initialising WireGuard® VPN 'client' Peer (wg13) in Policy Mode to dummy.azire-vpn.net:51880 (# Dummy) DNS=10.50.60.1,fe80::aa5e:45ff:feae:5050
Name or service not known: `dummy.azire-vpn.net:51880'
Configuration parsing error
Cannot find device "wg13"

        ***ERROR Initialisation ABORTED - 'wg setconf wg13 /tmp/wg13.963 (/opt/etc/wireguard.d/wg13.conf)' FAILED

So running a bogus wgm client is not supported.

Any thoughts or ideas on what to try next?

 

[ZebMcKayhan]

So running a bogus wgm client is not supported.

Hmm, I would guess the problem is that wgm needs to resolve "dummy.azire-vpn.net" which I guess fails, so it won't start.
Try to set a different endpoint, perhaps some private ip address you don't use, like 10.99.88.77. Not sure if wgm barfs at this as well, altough I don't see why it should.

 

[archiel]

Hmm, I would guess the problem is that wgm needs to resolve "dummy.azire-vpn.net" which I guess fails, so it won't start.
Try to set a different endpoint, perhaps some private ip address you don't use, like 10.99.88.77. Not sure if wgm barfs at this as well, altough I don't see why it should.

Apparently 10.99.88.77 is just fine.

 start wg13

        Requesting WireGuard® VPN Peer start (wg13)

        wg_manager-clientwg13: Initialising WireGuard® VPN 'client' Peer (wg13) in Policy Mode to 10.99.88.77:51880 (# Dummy) DNS=10.50.60.1,fe80::aa5e:45ff:feae:5050
        wg_manager-clientwg13: Initialisation complete.

 

[ZebMcKayhan]

It seems I was overly optimistic with the bogus client. What I did was to take the old working (azireVPN) client and just change the port. This allowed the client to load, but each time I did load it for any period of time it seemed to mess up the network, (unbound would report as working and I could see DNS lookups / responses) but the LAN clients were not resolving site names. the ubuntu VM 'network icon' was indicating the network was not functioning, etc.

I don't get why your endpoint should cause this. Sounds like there is lan connectivity issue. The only thing that is happening is that wgm puts in a route to your endpoint to go out wan, that would not matter as the only one actually connecting to this is your Ubuntu machine, which should go the same way.

Unbound working but no lan lookup working, hmm... an issue with router to talk to lan???
I wonder if this is an ipv6 issue... does your ipv6 policy route table contain a route to br0?

ip -6 route show table 123

?

Edit: nvm... It does, saw your dump earlier.

 

[archiel]

I don't get why your endpoint should cause this. Sounds like there is lan connectivity issue. The only thing that is happening is that wgm puts in a route to your endpoint to go out wan, that would not matter as the only one actually connecting to this is your Ubuntu machine, which should go the same way.

Unbound working but no lan lookup working, hmm... an issue with router to talk to lan???
I wonder if this is an ipv6 issue... does your ipv6 policy route table contain a route to br0?

ip -6 route show table 123

?

Edit: nvm... It does, saw your dump earlier.

For now I was going to assume it was connected to the old dummy account -

ip -6 route show table 123
2a02:6b67:xxxx:yyyy::/56 dev br0  proto kernel  metric 256  pref medium
::/2 via fe80::eae5:3390:e03f:f055 dev br0  metric 1024  pref medium
4000::/2 via fe80::eae5:3390:e03f:f055 dev br0  metric 1024  pref medium
::/1 dev wg13  metric 1024  pref medium
8000::/2 via fe80::eae5:3390:e03f:f055 dev br0  metric 1024  pref medium
fe80::/64 dev br0  proto kernel  metric 256  pref medium
c000::/2 via fe80::eae5:3390:e03f:f055 dev br0  metric 1024  pref medium
8000::/1 dev wg13  metric 1024  pref medium

and then add /remove various bits until it breaks

The new dummy (10.99.88.77) has been up for a little over an hour with fc enabled and no problems, if is is all still good in the morning the next steps are see what happens if I re-add the ipset / point unbound at the spoofed WAN ipv4 / ipv6 addresses. (one at a time and then if still okay, both)

 

[ZebMcKayhan]

For now I was going to assume it was connected to the old dummy account -

ip -6 route show table 123
2a02:6b67:xxxx:yyyy::/56 dev br0  proto kernel  metric 256  pref medium
::/2 via fe80::eae5:3390:e03f:f055 dev br0  metric 1024  pref medium
4000::/2 via fe80::eae5:3390:e03f:f055 dev br0  metric 1024  pref medium
::/1 dev wg13  metric 1024  pref medium
8000::/2 via fe80::eae5:3390:e03f:f055 dev br0  metric 1024  pref medium
fe80::/64 dev br0  proto kernel  metric 256  pref medium
c000::/2 via fe80::eae5:3390:e03f:f055 dev br0  metric 1024  pref medium
8000::/1 dev wg13  metric 1024  pref medium

and then add /remove various bits until it breaks

The new dummy (10.99.88.77) has been up for a little over an hour with fc enabled and no problems, if is is all still good in the morning the next steps are see what happens if I re-add the ipset / point unbound at the spoofed WAN ipv4 / ipv6 addresses. (one at a time and then if still okay, both)

Alright! So it appears our trick with hijacking wgm policy table and redirect it to your Ubuntu machine and onwards to wg vpn, and back, works? Cool!

Would be interesting to hear what speeds you are getting!

 

[archiel]

Alright! So it appears our trick with hijacking wgm policy table and redirect it to your Ubuntu machine and onwards to wg vpn, and back, works? Cool!

Would be interesting to hear what speeds you are getting!

NOT YET..... All I have got to is that the wg client on the VM is happy and the dummy wgm client on the router is happy - what happens when I introduce them is yet to be seen) Until now the dummy wgm client looked happy, but was causing ructions, most probably caused by my decision to point it a 'wrong' port on an otherwise real endpoint.

 

[archiel]

Each time I run the dummy wg13 and then point unbound through the VPN then it (unbound) stops working, Looking at the logs I see the requests, but there are no replies.

Presumably this is either because

1. the DNS requests are not leaving the router, or
2. they are not getting through the firewall on the VM, or
3. they are getting though the firewall but not hitting the VM wg client
4. they are reaching the VM wg client but not being processed correctly

(if there are any other causes would you add them to the testing list)

I can see no reason for 1, given the routing rules added to wg13-up, but it seems the first thing to test. Any suggestions on how to do this, or even if this is this right place to start?

 

[ZebMcKayhan]

Each time I run the dummy wg13 and then point unbound through the VPN then it (unbound) stops working, Looking at the logs I see the requests, but there are no replies.

Presumably this is either because

1. the DNS requests are not leaving the router, or
2. they are not getting through the firewall on the VM, or
3. they are getting though the firewall but not hitting the VM wg client
4. they are reaching the VM wg client but not being processed correctly

(if there are any other causes would you add them to the testing list)

I can see no reason for 1, given the routing rules added to wg13-up, but it seems the first thing to test. Any suggestions on how to do this, or even if this is this right place to start?

I would probably install tcpdump on the Ubuntu machine and set it up to log, say icmp echo on its lan interface.

tcpdump -i eth0 icmp

Then on the router, ping some known destination (like google.com) using your alias ip as source address and see if tcpdump picks it up.

ping google.com -I 192.168.3.1

If that works, shift tcpdump to log on wireguard interface and redo.

 

[ZebMcKayhan]

Each time I run the dummy wg13 and then point unbound through the VPN then it (unbound) stops working, Looking at the logs I see the requests, but there are no replies.

Presumably this is either because

1. the DNS requests are not leaving the router, or
2. they are not getting through the firewall on the VM, or
3. they are getting though the firewall but not hitting the VM wg client
4. they are reaching the VM wg client but not being processed correctly

(if there are any other causes would you add them to the testing list)

I can see no reason for 1, given the routing rules added to wg13-up, but it seems the first thing to test. Any suggestions on how to do this, or even if this is this right place to start?

it just accrued to me that your ubuntu machines before VPN know of 1 network (10.50.60.0/24) and all other unknown destinations (internet) got sent to the same network.

now things have changed, when you added your VPN on it, it would still know to send lan data (10.50.60.0/24) to your router but all other unknown destinations (internet) would now be sent out to VPN. this could mean trouble (?)...

when you setup unbound, you make it use an alias ip (192.168.3.1) so this is where unbound outgoing packets appear to come from. so this arrives to ubuntu machine and if it ever gets a reply (to 192.168.3.1) your ubuntu machine would never know where to send this as it is an unknown destination for it, so it will end up in the unknown destination lump and gets sent back out on VPN again. rp filter will probably prevent this from ever happening if you have it on the ubuntu machine.

the same would be true for your VPN server clients I believe (not sure if they get masquaraded by router/wgm).

a quick way to test this would be to setup a wgm rule for some lan client temporarily (say, your phone, ipv4 only) and check your public ipv4 on that device. if lan clients work but not unbound, it could be a routing issue on the ubuntu machine and you may need to add manual routes for these.

 

[archiel]

I would probably install tcpdump on the Ubuntu machine and set it up to log, say icmp echo on its lan interface.

tcpdump -i eth0 icmp

Then on the router, ping some known destination (like google.com) using your alias ip as source address and see if tcpdump picks it up.

ping google.com -I 192.168.3.1

If that works, shift tcpdump to log on wireguard interface and redo.

With wg13 up and unbound not routed via the alias (192.168.3.1) both icmp echo pings from the router to 192.168.3.1 are picked up by the VM, on eth0 and azirevpn-nl-ams. and DNS for LAN clients is fine
With wg13 down and unbound using the alias

outgoing-interface: 192.168.3.1                       # routing to wan-event + wgm policy rules
outgoing-interface: fd36:7ef1:2add:aa88:100::1        # routing to wan-event + wgm policy rules

then both icmp echo pings are NOT picked up, as above DNS for LAN clients is fine

With wg13 up and unbound using the alias then both icmp echo pings are picked up, but DNS for LAN clients does not work (unbound is not showing any replies)

Does this tally with your second message and if so do we need to tell the VM to route incoming traffic for the aliases (192.168.3.1 and the ipv6 equivalent) back to the router?

 

[ZebMcKayhan]

With wg13 up and unbound not routed via the alias (192.168.3.1) both icmp echo pings from the router to 192.168.3.1 are picked up by the VM, on eth0 and azirevpn-nl-ams. and DNS for LAN clients is fine

I'm guessing it's a typo, you did not ping TO 192.168.3.1, you pinged from that ip, correct?
And you got a reply to your router, as usually when ping works?
So, if my guess is correct it means the ping went out via ubuntu to vpn and back, right?

With wg13 down and unbound using the alias

outgoing-interface: 192.168.3.1 # routing to wan-event + wgm policy rules
outgoing-interface: fd36:7ef1:2add:aa88:100::1 # routing to wan-event + wgm policy rules

then both icmp echo pings are NOT picked up, as above DNS for LAN clients is fine

Sounds as expected behavior...

With wg13 up and unbound using the alias then both icmp echo pings are picked up, but DNS for LAN clients does not work (unbound is not showing any repreplies

You mean back to lan, or did you check in Unbound??

Still wondering if it's a unbound<->lan issue with the policy route table. If wg13 is up and unbound are using the alias what do you get from a local lookup on the router directly to unbound?

dig @127.0.0.1 -p 53535 google.com

Guess unbound could still be serving you cached result, so try something you typically won't use, maybe "sunet.se" (swedish university network)

Does this tally with your second message and if so do we need to tell the VM to route incoming traffic for the aliases (192.168.3.1 and the ipv6 equivalent) back to the router?

If the ping is working then this would not be needed, altough I don't understand why your Ubuntu machine understands it should send this to lan... but we will see as we go.

 

[archiel]

I'm guessing it's a typo, you did not ping TO 192.168.3.1, you pinged from that ip, correct?
And you got a reply to your router, as usually when ping works?
So, if my guess is correct it means the ping went out via ubuntu to vpn and back, right?

Right - I was pinging from the router.

You mean back to lan, or did you check in Unbound??

Checking directly from unbound.

Still wondering if it's a unbound<->lan issue with the policy route table. If wg13 is up and unbound are using the alias what do you get from a local lookup on the router directly to unbound?
dig @127.0.0.1 -p 53535 google.com

Will run this much later. possibly tomorrow - my sons are having a gaming session and stopping and starting processes on the router will not be appreciated.

Guess unbound could still be serving you cached result, so try something you typically won't use, maybe "sunet.se" (swedish university network)

After I change each setup, I restart unbound and clear the cache (just in case)

If the ping is working then this would not be needed, although I don't understand why your Ubuntu machine understands it should send this to lan... but we will see as we go.

 

[ZebMcKayhan]

Checking directly from unbound.

Another thing to test is to remove the ipv6 outgoing-interface in Unbound.conf and only use ipv4. Then only ipv6... just to see if both have issues or just one.
I know Unbound is suppose to rotate between interfaces but if it alternates during the recursive part and one is not working it may fail the entire lookup...

 

[archiel]

Still wondering if it's a unbound<->lan issue with the policy route table. If wg13 is up and unbound are using the alias what do you get from a local lookup on the router directly to unbound?
dig @127.0.0.1 -p 53535 google.com

dig @127.0.0.1 -p 53535 sunet.se works fine with either wg13 up or unbound using aliases

Spoiler: normal dig output

admin@Router:/tmp/home/root# dig @127.0.0.1 -p 53535 sunet.se

; <<>> DiG 9.18.16 <<>> @127.0.0.1 -p 53535 sunet.se
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 17535
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;sunet.se.                      IN      A

;; ANSWER SECTION:
sunet.se.               14400   IN      A       37.156.192.51
sunet.se.               14400   IN      A       37.156.192.50

;; Query time: 816 msec
;; SERVER: 127.0.0.1#53535(127.0.0.1) (UDP)
;; WHEN: Thu Jan 25 12:04:51 GMT 2024
;; MSG SIZE  rcvd: 69

However if both are used then

admin@Router:/tmp/home/root# dig @127.0.0.1 -p 53535 sunet.se
;; communications error to 127.0.0.1#53535: timed out
;; communications error to 127.0.0.1#53535: timed out
;; communications error to 127.0.0.1#53535: timed out

; <<>> DiG 9.18.16 <<>> @127.0.0.1 -p 53535 sunet.se
; (1 server found)
;; global options: +cmd
;; no servers could be reached

Will try icmp echo ping tests with unbound using either ipv4 or ipv6 only next.

 

[archiel]

with ipv6 disabled on unbound, wg13 up and running

ping google.com -I 192.168.3.1

on the router and

tcpdump -i eth0 icmp

on the VM I get

i.e. no replies

and running

tcpdump -i azirevpn-nl-ams eth0 icmp

I see pairs like

This is the same as response with ipv4 and ipv6 enabled.

As far as I can see switching unbound to ipv6 only is not possible (or I don't know how) - setting do-ip4: no in unbound.conf simply results in no dns queries at all (regardless of whether do-ip6: is set to yes or no).

 

[ZebMcKayhan]

running
on the router and
on the VM I get

i.e. no replies

and running
I see pairs like

Aha, so the echo reply is received from vpn but it's not forwarded back to lan, which could confirm my initial suspicion that the ubuntu machine does not know where to send 192.168.3.1 destinations.

What if you give it a route by executing on the Ubuntu machine:

ip route add 192.168.3.1 dev eth0 table main

Would that make your ping reply find it's way back to router?

As far as I can see switching unbound to ipv6 only is not possible (or I don't know how) - setting do-ip4: no in unbound.conf simply results in no dns queries at all (regardless of whether do-ip6: is set to yes or no).

Yea, possibly some servers are ipv4 only so it wouldnt be able to do the lookup if ipv6 only. Let focus on ipv4 first and worry about ipv6 later.