Wireguard Session Manager - Discussion (3rd) thread

[johndoe85]

Ok, great you've solved it!

If you are running latest wgm you might need to re-create your config to take full advantage of latest settings:

E:Option ==> createconfig

Any addons that may interfere?

Ok re-created config.
And uncommented disable flow cache.
Restarted wg with option 6.

And speeds are OK.

mtm 3.2.3 FW             by thelonelycoder
 RT-AX86U (aarch64) FW-386.5 @ 192.168.50.1
    The Asuswrt-Merlin Terminal Menu
 2  open     Skynet                    v7.2.8
 4  open     YazFi                     v4.4.2                                    6  open     x3mRouting                v2.4.5
 amtm 3.2.3 FW             by thelonelycoder                                     RT-AX86U (aarch64) FW-386.5 @ 192.168.50.1                                         The Asuswrt-Merlin Terminal Menu
 2  open     Skynet                    v7.2.8                                   
 4  open     YazFi                     v4.4.2                                    6  open     x3mRouting                v2.4.5                                   
 7  open     unbound Manager            v3.22                                    8  open     nsrum                    v30.4.0                                   
 j2 open     ntpMerlin                 v3.4.5
 j3 open     scMerlin                  v2.4.0

 j7 open     YazDHCP                   v1.0.4
 vn open     vnStat                    v2.0.4

 wg open     WireGuard Mgr              v4.16
 ag open     AdGuardHome               v1.5.2
             AGH binary              v0.107.6

 ep manage   Entware packages          no upd

awm Asuswrt-Merlin firmware           386.5.2

 m  menu     amtm   uu  force update   v3.2.3
_____________________________________________

 Everything's up to date (May 15 2022 22:30)
_____________________________________________

 Enter option

 

[ZebMcKayhan]

Ok re-created config.
And uncommented disable flow cache.
Restarted wg with option 6.

And speeds are OK.

mtm 3.2.3 FW             by thelonelycoder
 RT-AX86U (aarch64) FW-386.5 @ 192.168.50.1
    The Asuswrt-Merlin Terminal Menu
 2  open     Skynet                    v7.2.8
 4  open     YazFi                     v4.4.2                                    6  open     x3mRouting                v2.4.5
 amtm 3.2.3 FW             by thelonelycoder                                     RT-AX86U (aarch64) FW-386.5 @ 192.168.50.1                                         The Asuswrt-Merlin Terminal Menu
 2  open     Skynet                    v7.2.8                                   
 4  open     YazFi                     v4.4.2                                    6  open     x3mRouting                v2.4.5                                   
 7  open     unbound Manager            v3.22                                    8  open     nsrum                    v30.4.0                                   
 j2 open     ntpMerlin                 v3.4.5
 j3 open     scMerlin                  v2.4.0

 j7 open     YazDHCP                   v1.0.4
 vn open     vnStat                    v2.0.4

 wg open     WireGuard Mgr              v4.16
 ag open     AdGuardHome               v1.5.2
             AGH binary              v0.107.6

 ep manage   Entware packages          no upd

awm Asuswrt-Merlin firmware           386.5.2

 m  menu     amtm   uu  force update   v3.2.3
_____________________________________________

 Everything's up to date (May 15 2022 22:30)
_____________________________________________

 Enter option

Ok, keep an eye on it for some time, and if it happens again check the log if any addon that might affect this restarts.

 

[johndoe85]

Ok, keep an eye on it for some time, and if it happens again check the log if any addon that might affect this restarts.

I noticed after latest reboot (just did another one) the wg server didnt connected properly and I had to restart the wg server for it to function again.

I figure I might need to add a delay for wg server at bort.
I don't know why it wasn't functioning, but since a simple restart made it work again there might be something with the boot order and addons not being up yet, like adguard.
I don't understand why it would matter if wg starts esrly or late. But my guess there is something to it.

So how do i make wireguard starts at a delay?

 

[ZebMcKayhan]

I noticed after latest reboot (just did another one) the wg server didnt connected properly and I had to restart the wg server for it to function again.

I figure I might need to add a delay for wg server at bort.
I don't know why it wasn't functioning, but since a simple restart made it work again there might be something with the boot order and addons not being up yet, like adguard.
I don't understand why it would matter if wg starts esrly or late. But my guess there is something to it.

So how do i make wireguard starts at a delay?

Don't know if delay is the right solution, since any event could trigger the same thing again.

What's in the syslog from the boot after wireguard starts?

 

[chongnt]

I noticed after latest reboot (just did another one) the wg server didnt connected properly and I had to restart the wg server for it to function again.

I figure I might need to add a delay for wg server at bort.
I don't know why it wasn't functioning, but since a simple restart made it work again there might be something with the boot order and addons not being up yet, like adguard.
I don't understand why it would matter if wg starts esrly or late. But my guess there is something to it.

So how do i make wireguard starts at a delay?

Wonder if you wait long enough after bootup to connect to wgs? In your last post, I noticed you have skynet as well. In my firewall-start script, wg_firewall is after skynet firewall start. What I noticed is wg started while skynet is running. By the time wg_firewall run, all wg connection is killed and then restarted again.

sh /jffs/scripts/firewall start skynetloc=/tmp/mnt/amtm/skynet # Skynet
jffs/addons/wireguard/wg_firewall            # WireGuard

May not be your case as I don’t need to restart wgs after this. I also have issue firewall-start run twice during bootup. In order to delay wgm start before wg_firewall complete, I add a 90s delay in post-mount script.

bin/sleep 90s && /jffs/addons/wireguard/wg_manager.sh init "" & # WireGuard Manager

Not sure if this is the correct way. Now I reorder firewall-start script to let wg_firewall run first before skynet. I also add wg_firewall in post-mount before wgm init just to make sure everything is good. I have removed the delay and wgm started once during bootup. Wgm not getting killed and restart during bootup.

 

[Martineau]

Wonder if you wait long enough after bootup to connect to wgs? In your last post, I noticed you have skynet as well. In my firewall-start script, wg_firewall is after skynet firewall start. What I noticed is wg started while skynet is running. By the time wg_firewall run, all wg connection is killed and then restarted again.

sh /jffs/scripts/firewall start skynetloc=/tmp/mnt/amtm/skynet # Skynet
jffs/addons/wireguard/wg_firewall            # WireGuard

May not be your case as I don’t need to restart wgs after this. I also have issue firewall-start run twice during bootup. In order to delay wgm start before wg_firewall complete, I add a 90s delay in post-mount script.

bin/sleep 90s && /jffs/addons/wireguard/wg_manager.sh init "" & # WireGuard Manager

Not sure if this is the correct way. Now I reorder firewall-start script to let wg_firewall run first before skynet. I also add wg_firewall in post-mount before wgm init just to make sure everything is good. I have removed the delay and wgm started once during bootup. Wgm not getting killed and restart during bootup.

I'd long since forgotten the mayhem that can occur during the boot process without manually coding a suitable 'locking semaphore' at the top of each script
e.g.

LOCKFILE="/tmp/$(basename $0)-flock"
FD=nnn
eval exec "$FD>$LOCKFILE"
flock -n $FD || { logger -st "($(basename $0))" $$ "Script ALREADY running...ABORTing"; exit; }

I've uploaded wireguard_manager Beta v4.17b4

Update wg_manager.sh · MartineauUK/wireguard@0eeb79e

FIX: Prevent multiple 'INIT' requests for wireguard_manager from firing (via multiple firewall-start executions) during BOOT process when 'INITDELAY nn' directive is ACTIVE CHANG...

github.com

To upgrade use:

e  = Exit Script [?]

E:Option ==> uf dev

If the following is issued:

e  = Exit Script [?]

E:Option ==> createconfig

A default 10 second delay will be applied when the wg_manager init & request is honoured.

The 10 seconds may be customised (i.e. 90 seconds etc.) by using the vx command and editing the INITDELAY directive

The presence of an ACTIVE INITDELAY directive will also now ensure that the duplicate locked semapahore check is enabled for the wireguard_manager initialisation request.

In theory, now it shouldn't matter what script order is used and the unnecessary clunky wgm stop/wgm start sequence should no longer be executed/required.

 

[amplatfus]

I previously asked if anyone else had/has the rogue RPDB 220: from all lookup 220 rule(s)...

Wireguard - Session Manager - Discussion (2nd) thread

The point is that even when no clients are connected via Wireguard I was only able to measure about a third of regular speed.

www.snbforums.com

So something (as yet unidentified) is creating the rule(s), so assumed it was most likely only within my environment, but clearly it wasn't compatible with wireguard_manager, so I unilaterally hacked its removal if found.

P.S. If you legitimately need the RPDB rule as-as, or more importantly that you have found that is my manky script at fault then please advise.

Thank you,
It may be related to dual WAN? Do you have a dual one? I have it on with failover-failback on.

I have also:
Diversion
Skynet
Unbound
scMerlin
spdMerlin.

Kind regards,
amplatfus

 

[Martineau]

Thank you,
It may be related to dual WAN? Do you have a dual one? I have it on with failover-failback on.

I have also:
Diversion
Skynet
Unbound
scMerlin
spdMerlin.

Kind regards,
amplatfus

IIRC, in previous firmwares, tables 100 and 200 were used for Dual-WAN?

However, I haven't enabled Dual-WAN.

 

[endiz]

Are the three opening paragraphs enough to sway you towards WireGuard?

Setting Up WireGuard® on Unraid

Setting Up WireGuard® on Unraid

unraid.net

View attachment 41215

This more detailed Unraid guide should help to understand the Unraid GUI option 'Remote tunneled access'

WireGuard quickstart

Note: this community guide is offered in the hope that it is helpful, but comes with no warranty/guarantee/etc. Follow at your own risk. What can you do with WireGuard? Let's walk through each of the connection types: Remote access to server: Use your phone or computer to remotely access your Unr...

forums.unraid.net

View attachment 41219
which by default is the first topology i.e. Remote inbound access to the Unraid server/services.

View attachment 41221
so be sure to select LAN to LAN access from the GUI drop-down menu.

Using wireguard_manager, the configuration on the RT-AC86U should take 5 mins assuming that it can successfully import the 'client' Peer config generated by the Unraid WireGuard 'server' Peer - but not 100% sure as I've never tried it.

However @JGrana has had success with a WireGuard site-to-site between two ASUS routers - not sure if there was a speed/stability comparison between OpenVPN vs. WireGuard for his configuration to justify his choice of protocol.

Thanks for the pointers. I was able to get this to work utilizing the "remote tunneled access" peer type. 1 small thing to note: the config file unraid provided didn't work with wgm out of the box. Importing went fine, but when starting the tunnel I got a "***ERROR: WireGuard 'client' doesn't have a LOCAL IP Address! - try 'peer wg11 ip=xxx.xxx.xxx.xxx/32'?" error. I'm not sure what was wrong exactly, but I fixed up the formatting in the config (added spaces between '=') and removed the commented labels unraid added, and once I re-imported, everything worked great! Thanks all!

 

[Martineau]

1 small thing to note: the config file unraid provided didn't work with wgm out of the box. Importing went fine, but when starting the tunnel I got a "***ERROR: WireGuard 'client' doesn't have a LOCAL IP Address! - try 'peer wg11 ip=xxx.xxx.xxx.xxx/32'?" error. I'm not sure what was wrong exactly, but I fixed up the formatting in the config (added spaces between '=') and removed the commented labels unraid added, and once I re-imported, everything worked great! Thanks all!

View attachment 41294

Thanks for the feedback - glad you managed to get it working.

I assume you are on the stable branch i.e. wireguard_manager v4.16?

If you upgrade to wireguard_manager Beta v4.17b4 using command

e  = Exit Script [?]

E:Option ==> uf dev

then you would be informed during the peer import xxxxxx (or wgm import xxxxxx) command that there is a problem parsing the exported '.conf' file generated by Unraid.

EDIT: Beta v4.17b2 Error message ONLY issued if Address = directive is missing entriely. (Will be fixed in Beta v4.17b5)

All academic now as you manually corrected the import process, however, if you have the time, could you PM me a sample of the raw .conf file so I can review what I could do to auto-correct the failed import ( - interesting to see what wg-quick makes of it as well).

NOTE: Obfuscate the Private/Public keys..... simply replacing a few characters would suffice.

 

[ZebMcKayhan]

I really think its a pity that the xmark rule dont work for later AX- models such as the AX86U. The mark seems to only work well on AC86U and AX88U where we can keep flow cache enabled with the benefit of speed for clients not routed out Wireguard.

If I owned any of these routers I would probably set up a script to brute-force this mark to see if the mark have changed values for these routers. Thinking abit further ahead, probably the mark has not changed, but the mask might have. On older models this mark was 0x01/0x1ff but on never models it seems to be 0x01/0x7. Searching around the internet gives pretty much nothing but on some places mask of 0xff seems to pop up with BCM4908 (AX86U).

The mark was a part of merlin firmware from the beginning but was removed years ago because of the risk of conflict with trendmicro marks https://www.snbforums.com/threads/potential-bug-with-udp-nat-loopback-hairpinning.70892/post-670363.

It would be fairly straight forward for anyone with i.e ax86u router to test other marks/mask to see if any would work with flow cache enabled.
Probable mark/mask would be

0x01/0x1
0x01/0x3
0x01/0x7 #no need to test this
0x01/0xf
0x01/0x1f
0x01/0x3f
0x01/0x7f
0x01/0xff
0x01/0x1ff
a.s.o

Or is this just a super-stupid idea? Or perhaps someone has already tried this?

 

[Meshkoff]

Hi all

I'm trying to get work a road warrior setup, so my laptop being anywhere outside can reach NAS inside my home network. I read very useful guide by ZebMcKayhan here:

https://github.com/ZebMcKayhan/WireguardManager#i-cant-access-my-nassamba-share-over-vpn

But when i attempt to create new wg interface with custom ip pool it still binds the default ip (10.50.1.1/24):

E:Option ==> peer new wg21 ip=192.168.0.1/23

     Press y to Create 'server' Peer (wg21) 10.50.1.1/24:11501 or press [Enter] to SKIP.

What i'm doing wrong here?

 

[ZebMcKayhan]

Hi all

I'm trying to get work a road warrior setup, so my laptop being anywhere outside can reach NAS inside my home network. I read very useful guide by ZebMcKayhan here:

https://github.com/ZebMcKayhan/WireguardManager#i-cant-access-my-nassamba-share-over-vpn

But when i attempt to create new wg interface with custom ip pool it still binds the default ip (10.50.1.1/24):

E:Option ==> peer new wg21 ip=192.168.0.1/23

     Press y to Create 'server' Peer (wg21) 10.50.1.1/24:11501 or press [Enter] to SKIP.

What i'm doing wrong here?

Kindof looks like your trying to setup a conflict with br0, which has 192.168.0.1 ?while you are trying to setup wg21 with the same ip?
Also, why are you trying to use /23 for the wg21 server? It is usually not nessisary and I guess you are not planning to have 511 clients on this server?

Or are your br0 192.168.1.1? Then this might be some bug in wgm... try with /24 instead.

What we are trying to achieve is to change the netmask on br0 in the gui of the router to include wg server but the server should not need to include br0. Come to think about it, it may create conflict if it does as the 2 network are mutually inclusive and equal in size.

 

[Martineau]

Hi all

I'm trying to get work a road warrior setup, so my laptop being anywhere outside can reach NAS inside my home network. I read very useful guide by ZebMcKayhan here:

https://github.com/ZebMcKayhan/WireguardManager#i-cant-access-my-nassamba-share-over-vpn

But when i attempt to create new wg interface with custom ip pool it still binds the default ip (10.50.1.1/24):

E:Option ==>[CODE]peer new wg21 ip=192.168.0.1/23

Press y to Create 'server' Peer (wg21) 10.50.1.1/24:11501 or press [Enter] to SKIP.[/CODE]

What i'm doing wrong here?

Whoops

I've uploaded a patched wireguard_manager Beta v4.17b5

Update wg_manager.sh · MartineauUK/wireguard@4c7d978

FIX: Regression; Cannot specify custom IP subnet when using 'server' Peer create command 'peer new' (default 10.50.n.1/24 is ALWAYS used) - Thanks SNB Forum member @Meshkoff ...

github.com

To Upgrade use:

e  = Exit Script [?]

E:Option ==> uf dev

NOTE: You should have been able to allow creation of the 'server' Peer with the auto-determined subnet '10.50.nnn.1/24'
e.g.

e  = Exit Script [?]

E:Option ==> peer new wg21

then issue

e  = Exit Script [?]

E:Option ==> peer wg21 ip=192.168.0.1/23

 

[chongnt]

Hi all

I'm trying to get work a road warrior setup, so my laptop being anywhere outside can reach NAS inside my home network. I read very useful guide by ZebMcKayhan here:

https://github.com/ZebMcKayhan/WireguardManager#i-cant-access-my-nassamba-share-over-vpn

But when i attempt to create new wg interface with custom ip pool it still binds the default ip (10.50.1.1/24):

E:Option ==> peer new wg21 ip=192.168.0.1/23

     Press y to Create 'server' Peer (wg21) 10.50.1.1/24:11501 or press [Enter] to SKIP.

What i'm doing wrong here?

Just in case your LAN ip range is 192.168.1.1/24, it might overlap as 192.168.0.1/23 has a usable host range of 192.168.0.1 - 192.168.1.254.

 

[Meshkoff]

Kindof looks like your trying to setup a conflict with br0

Sorry, i should mentioned my setup for clarity

router br0 was 192.168.1.1/24, then i decide to create wireguard server on different subnet, to visually separate wireguard clients (0.1-0.254) from local network clients (1.1-1.254) and changed router br0 to 192.168.1.1/23 to extend the network

What we are trying to achieve is to change the netmask on br0 in the gui of the router to include wg server but the server should not need to include br0

Thanks for the point, that's was the main question: does my WG server bg0 needs to be the same as router bg0 so WG clients can reach local machines or it just needs to be on same subnet?

However then i tried to create WG server with 192.168.0.1/24 it still binds with default ip, even with 192.168.0.1/32

Just in case your LAN ip range is 192.168.1.1/24, it might overlap as 192.168.0.1/23 has a usable host range of 192.168.0.1 - 192.168.1.254.

If i got it right that is necessary to reach local devices from outside the LAN.

Router br0 includes WG server br0 but WG server bg0 not overlapping router and clients ip range, just like separating subnets as ZebMcKayhan mentioned above

I've uploaded a patched wireguard_manager Beta v4.17b5

Wow, thanks for such a quick fix here. Now i was able to create WG server on 192.168.0.1/24

Device created, automatically binds to 192.168.0.2/32

inserted iptables rule:

iptables -t nat -I POSTROUTING -s 192.168.0.0/24  -o eth0 -j MASQUERADE -m comment --comment "WireGuard_server"

Doublechecked the iptables rule exist, restart WG server, got client connected, see handshake, checked wan ip - match router, everything is good, but... Still can't access any of my local machine, even the router itself... Wait, how can it be when everything is good but i can't access even router itself?

Figured it out! I was trying to reach my local machines from another neighbor LAN which has the same br0 as my LAN - 192.168.1.1/24, that was the issue.

Create another peer, tested with phone over cellular network - everything is works, can access NAS and other things, such a magic, thank you guys

P.S. So, if i want to communicate one LAN to another LAN, it br0 must not being overlapping or i need to learn something new

 

[ZebMcKayhan]

P.S. So, if i want to communicate one LAN to another LAN, it br0 must not being overlapping or i need to learn something new

This is a special case for accessing a specific restricted share and normally we consider a server client peer to be isolated (which ofcource its not). If you plan on connect from other networks then you might think about changing br0 ip to something not commonly used. Guess why ipv6 created ula based on mac+time.

Connecting to windows machines are virtually impossible from a different network because of hard coded restrictions so you will need to MASQUERADE to make it work.

At some point you end up at site-2-site but you might have problems accessing some stuff. But its not router, firewall or routing that is not working. It is the machines itself and the ways around could vary in flavour.

inserted iptables rule:

I don't understand why you need this (or where you got it from). Eth0 usually accept all for MASQUARADE.

Edit: br0 could be overlapping (and that is kind of the point here) but as you degrade it to /23 it would drop in routing priority. I.e br0 is 192.168.1.1/23 and wg21 is 192.168.0.1/24. So if a package arrives to destination 192.168.0.2 there will be 2 destinations: br0 and wg21 according to route table. Wg21 will be decided since the route is more specific which is how we intended it. But if wg21 was also a /23 that would not be the case and routing would be based on metric or something else and wg21 will probably not work.

Edit2: br0 could overlap into adjacent networks using the netmask, but it cannot be in direct conflict. as you discovered if you are trying to access your NAS on your LAN (192.168.1.x) via Wireguard (192.168.0.x) from another 192.168.1.x network, usually the device local routing table will send packets to the local network instead of out on Wireguard. if you want to access your network on a roaming device from all various other network, you might really think of changing br0 to something like 192.168.16.x (16 = 0x10) and wg21 to 192.168.17.x (17 = 0x11) to minimize the risk for conflict.

I will try to update this section in my guide to be more clear, but it is difficult to keep it as general informative as possible without confuse things with information a specific user does not need.

 

[ZebMcKayhan]

Ok, back to this mark:

iptables -t mangle -I FORWARD -o wg11 -j MARK --set-xmark 0x01/0x7
iptables -t mangle -I PREROUTING -i wg11 -j MARK --set-xmark 0x01/0x7

Reading up on marks, what's happening is the current value is masked with the mask(0x7) which is binary AND, NOT on the current mark, basically clears the 3 least significant bits. Then XOR our mark(0x01) with the result, which basically inverts the first bit. So forcing the last 3 bits to be 0b001 while leaving rest of the mark as-is.

It would be really nice to glimpse on the code interpretating this. Guessing some module is responsible for the first 3 bits and the value 0b001 means bypass hw-acceleration.

Wonder what marks a typical package have set by various other kernel-modules if I.e trendmicro is not used. If packages typically has 0 mark then the mask don't really matter in size as the result would be the same.

No one with an ax86u that are interested enough of this to make some experiment?

 

[johndoe85]

Ok, back to this mark:

iptables -t mangle -I FORWARD -o wg11 -j MARK --set-xmark 0x01/0x7
iptables -t mangle -I PREROUTING -i wg11 -j MARK --set-xmark 0x01/0x7

Reading up on marks, what's happening is the current value is masked with the mask(0x7) which is binary AND, NOT on the current mark, basically clears the 3 least significant bits. Then XOR our mark(0x01) with the result, which basically inverts the first bit. So forcing the last 3 bits to be 0b001 while leaving rest of the mark as-is.

It would be really nice to glimpse on the code interpretating this. Guessing some module is responsible for the first 3 bits and the value 0b001 means bypass hw-acceleration.

Wonder what marks a typical package have set by various other kernel-modules if I.e trendmicro is not used. If packages typically has 0 mark then the mask don't really matter in size as the result would be the same.

No one with an ax86u that are interested enough of this to make some experiment?

I can be of your assistance. Tell me what i need to do and i do it in that order.

 

[Meshkoff]

I don't understand why you need this (or where you got it from)

Ahh, that rule was in road-warrior setup from scratch here:

https://emanuelduss.ch/2018/09/29/wireguard-vpn-road-warrior-setup/

Ended up with wg21 at 192.168.0.1/29, since I'm rarely need more than 1 client, so far works great

I will try to update this section in my guide to be more clear

Thanks for the info to think about, these guides are very handy