3 way Site-to-site OpenVPN

[bortek]

I have setup two instances of S2S OpenVPN between 3 homes using AsusWRT routers just like I have drawing on dirgrams.net

There is one server and are two clients. The site with server side has two VPN instances running. Each Client sites connect to the server. Hosts on each client side can talk to the hosts on the server side Net and vice versa.

What I want to achieve is that hosts on the client side KL be able to talk to other hosts on side JÖ and vice versa. Please take a look at the diagram one more time to get the idea.

Now here are the route specific configs from the Server OpenVPN. Keep in mind that there are 2 OpenVPN instances running in it.

Server1
push "route 192.168.0.0 255.255.255.0 vpn_gateway 500"
route 192.168.1.0 255.255.255.0
push "route 192.168.1.0 255.255.255.0"

Server2
push "route 192.168.0.0 255.255.255.0 vpn_gateway 500"
route 192.168.2.0 255.255.255.0
push "route 192.168.2.0 255.255.255.0"

I have tried to play with push routes specifying cross network addresses but that did not make it work. I know this should be possible. But I think it is not enough to tell the client to route the traffic via the server router. I think the server needs to be able to route the packets between these two client nets in some way.



As an example routing table on KL client side router looks like this.

Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         hi.link         0.0.0.0         UG    0      0        0 eth7
10.8.0.0        *               255.255.255.0   U     0      0        0 tun13
127.0.0.0       *               255.0.0.0       U     0      0        0 lo
192.168.1.0     *               255.255.255.0   U     0      0        0 br0
192.168.8.0     *               255.255.255.0   U     0      0        0 eth7
192.168.8.1     *               255.255.255.255 UH    0      0        0 eth7

 

[eightiescalling]

Does all traffic between KL and JÖ have to be seen by JS?

Just wondering why not set KL and JÖ up as servers for direct connect rather than chewing up JS bandwidth with traffic between them. So each site running 1 server and 2 clients.

 

[bortek]

not sure what you mean by seen but hosts behind all 3 routes should be able to communicate with each other.

KL can only be set as client since it's behind 4G mobile net (No incoming traffic allowed). JS has a lot of bandwidth up to 1Gbit wherease KL has pretty limited 4G and JS like 10Gbit. So for JS it's not an issue.

But sure this would likely be one way to solve it. But I would really like to solve it with JS as a server.

 

[eightiescalling]

Makes sense on the reason - I was just curious.

What do the logs show when connecting the VPN client from KL - can you see it adding the route?

What does

ip route get <address at JÖ>

show?

I'm not particularly familiar with the internals of OpenVPN and routing setups but I see something similar here - the route doesn't show with a basic netstat -r / ip route list but I can see it added in the logs when the VPN client connects and it also reports correctly when I use that ip route get command to check.

 

[bortek]

Logs in KL router says its been added

Dec  1 16:32:29 ovpn-client3[2905]: ovpn-route-pre-down tun13 1500 1604 10.8.0.2 255.255.255.0 init
Dec  1 16:32:31 openvpn-routing: Add pushed route: /usr/sbin/ip route add 192.168.0.0/255.255.255.0 via 10.8.0.1 dev tun13 metric 500 table ovpnc3
Dec  1 16:35:34 ovpn-client3[2905]: PUSH: Received control message: 'PUSH_REPLY,route 192.168.0.0 255.255.255.0 vpn_gateway 500,route-gateway 10.8.0.1,topology subnet,ping 15,ping-restart 60,ifconfig 10.8.0.2 255.255.255.0,peer-id 0,cipher AES-256-CBC'
Dec  1 16:35:34 ovpn-client3[2905]: OPTIONS IMPORT: route options modified
Dec  1 16:35:34 ovpn-client3[2905]: OPTIONS IMPORT: route-related options modified

Now here is more stuff. Right now JÖ router is in the lab and is connected to Internet using 4G (via USB tethering) which should not make a difference compared to if it would have been connected via a fiber.

So here how the "ip a" and "ip route get look on both sides"

On JÖ router

admin@JO-AX86U-1:/tmp/home/root#ip a
.....CUT.....
46: usb0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 1000
    link/ether da:6c:8b:9b:48:8f brd ff:ff:ff:ff:ff:ff
    inet 192.168.143.67/24 brd 192.168.143.255 scope global usb0
       valid_lft forever preferred_lft forever
.....CUT.....

admin@JO-AX86U-1:/tmp/home/root#ip route get 192.168.1.1
192.168.1.1 via 192.168.143.236 dev usb0 src 192.168.143.67

where 192.168.143.0/24 is USB tethering net.


On KL router
admin@KL-AC86U-1:/tmp/home/root#ip a

.....CUT.....
20: eth7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 1000
    link/ether 0c:5b:8f:27:9a:64 brd ff:ff:ff:ff:ff:ff
    inet 192.168.8.100/24 brd 192.168.8.255 scope global eth7
       valid_lft forever preferred_lft forever
.....CUT.....

admin@KL-AC86U-1:/tmp/home/root#ip route get 192.168.2.1
192.168.2.1 via 192.168.8.1 dev eth7 src 192.168.8.100

where 192.168.8.0/24 is the net of a 4G dongle connected to the KL router.

Now these results show that the routes (from KL to JÖ and vice versa) point to the interfaces which are not of the VPN. I think they try to send that traffic right out to the Internet, not inside the VPN.

If I do the the similar test from the JS (server VPN) to the client sides KL and JÖ I get these results.

admin@JS-AC86U-1:/tmp/home/root# ip route get 192.168.2.1
192.168.1.1 via 10.8.0.2 dev tun21 src 10.8.0.1

admin@JS-AC86U-1:/tmp/home/root# ip route get 192.168.2.1
192.168.2.1 via 10.16.0.2 dev tun22 src 10.16.0.1

KL and JÖ need to send the traffic destined to other side/client using the VPN. Plus the server VPN also should be able to reroute those packets further into a second VPN but that is probably taken care of by the server routing table.

Something is missing here.

 

[eightiescalling]

OK, that's starting to make a bit of sense.

KL has a route pushed to get to JS but there isn't a route for JO (192.168.2/24) in that log - hence KL routes to JO via the default gateway.

Going back over your original post, the routes listed for server 1 (used for KL to connect) don't include anything for JO or vice versa. If you add this to the routes pushed for server 1, does KL now show as going via the VPN tunnel? (Worrying about JS routing tables once KL is behaving properly!)

push "route 192.168.2.0 255.255.255.0 vpn_gateway 500"

 

[elorimer]

If I were to do this, I would connect both clients to the same server, and use client specific configs to add the routes. There is a post by @eibgrad somewhere laying this out. EDIT: Perhaps here: https://www.snbforums.com/threads/h...swrt-merlin-openvpn-clients.79271/post-767572

EDIT: All 2 core 86Us here. I suppose, if you are CPU limited, and your central router had more than two cores, there might be an advantage with Merlin in using two VPN servers, so that core 0 would do most router stuff, core 1 would handle one site, and core 2 would handle the other. Wireguard might be worth investigating then, particularly if you had more sites. To be honest, I'd first investigate using a free Oracle server as the middleman with its symmetrical half gig connections. I'm in the US and limited by upload limits that makes it all irrelevant.

 

[bortek]

Thanks for the suggestions! Let me try both options and will update tomorrow.

 

[bortek]

A quick update. Unfortunately the above idea did having two vpn servers one for each client not work. I tried push routes in different ways but the traffic was not routed between the two separate vpn servers. Even when I added push routes using vpn_gateway 500.

So I followed the link and eibgrad suggestion to use one single vpn server and 2 clients. user/pass was not an option so I ended up creating new client cert using the same ca/keys as the first client (KL) was created with. I went the hard way and created csr/cert/signed using openssl commands. Thus having a second client cert called client-jo. Of course I needed to change the port on JÖ site to point to the now single VPN. Also needed to add push route for both nets.

Here is a full guide on how to generate openssl stuff (follow starting from "Next, you have to sign it with your CA.") You will need to create openssl-ca.cnf as described in the guide. You will need to fetch ca.key ca.crt and other files and specify then in the config file. Quite a work. Keep in mind this is NOT a self-signed cert signing but a signing by CA!

My Putty session disconnected and history got lost. So I have no commands left to share here by what you need to do is
1. Generate csr using "openssl req" command and specify the same client.key as the used for the first client. Read the fields of the Subject entry of the existing client cert and use the same ones except the CN=client-jo should be set to something else which is not "client"
2. Sign the client cert with ca keys using this command
openssl ca -config openssl-ca.cnf -policy signing_policy -extensions signing_req -out client-jo.crt -infiles client-jo.csr



Wow now I have a 3 site-to-site VPN. Awesome!

I was wondering about the ip route get command and why it does somehow show how the packet will be routed while netstat -nr or route get does not show that info.

And here comes screenshots of the config in case if someone wants to do the same thing.

Now this was not a quick update but a long one which was worth it! Enjoy!

 

[bortek]

Now this setup has a SPoF which is the JS router acting as a server. In the long run I think I will go for a different setup where KL has two client one to JS and one to JÖ. JS has one client to JÖ and one server for KL. JÖ has two servers one for KL and one for JS. If that will work it will be a more robust.

 

[elorimer]

Now this setup has a SPoF which is the JS router acting as a server.

Yes, and that isn't all. We're talking about 3 homes here, so some of this isn't so important.

1. You haven't said what the up/down speeds are for each connection. You don't want JS to be the hub in this hub-and-spoke setup if its upload speed is a constraint.
2. Looking at the diagram again, I realised JS is an AC86, while JO is an AX86 with greater processing power because of its higher clock speed. So AX86 would be better in the center. I think it may be that traffic from KL to JO gets decrypted at JS, and then re-encrypted to go to JO, so the AX86 would be better at the center. On the other hand, the AX86 is quad core, it it might be better to be two clients, one to JS and one to KL (with JS and KL each running a server), since the clients will start on different cores.
3. Why are you running site-to-site in the first place? If you have a NAS that everyone accesses, then that might be better in the hub network. If you want to RDP into a computer on a spoke, you might be better using a separate server on the spoke and client on the remote. Same if you just want to fool with the router.
4. Having every device be able to access every other device in all three locations seems to be over-permissioning. I remember when my office first got into road warrior stuff. We would lend out laptops, and run virus checks on them when they came back. One guy would return a laptop infected every single time, and when we asked about it he would swear he never did anything but connect to the home office. Except, it turned out, he was letting his son install games on it just to shut the brat up and getting infected every single time.
5. The spokes should be split tunnel so non-LAN internet traffic isn't going through JS (unless you want it to),

For your alternate configuration, I'd worry a bit about loops. I didn't follow it exactly.

 

[bortek]

Thanks for the suggestions and advice! KL can only have initiate outgoing connection due to 4G mobile Interne therefore server is not possible have in there. I understand that AX86 has much greater capacity than AC86 but I will let it run for a while to see evaluate the loads. This is all in the future, not now.

For now I will stick the working config as it is.