Wireguard Session Manager - Discussion (2nd) thread

[pkoci]

The point is that even when no clients are connected via Wireguard I was only able to measure about a third of regular speed.

 

[Martineau]

The point is that even when no clients are connected via Wireguard I was only able to measure about a third of regular speed.

With Flow Cache ENABLED; if WireGuard Peer speed/throughput performance is deemed acceptable, then there is probably no need to DISABLE Flow Cache (despite ASUS/Broadcom seemingly insisting that it should be DISABLED for specific AX routers?).

If however WireGuard Peer speed/throughput dramatically increases when Flow Cache is DISABLED in your environment, then you can decide if it warrants a script to DISABLE Flow Cache when the first WireGuard Peers is made ACTIVE then ENABLE Flow Cache when ALL WireGuard Peers are no longer ACTIVE.

 

[pkoci]

I can't use Wireguard with fc enabled at all (throughput is about 2-3 mbps and router's cores at almost 100 %). If fc disabled Wireguard performance is great. However, even if there are none active Wireguard peers, Speedtest download results won't reach the values I have with fc enabled (300 - 500 mbps down and 100mbps up instead of 940 mbps down and 100 mpbs up).

Can we expect better Wireguard performance with official Asus support? Or you should I look for more powerful router?

 

[Martineau]

I can't use Wireguard with fc enabled at all (throughput is about 2-3 mbps and router's cores at almost 100 %). If fc disabled Wireguard performance is great. However, even if there are none active Wireguard peers, Speedtest download results won't reach the values I have with fc enabled (300 - 500 mbps down and 100mbps up instead of 940 mbps down and 100 mpbs up).

There are many threads discussing the impact of Flow Cache on 1Gbps connections.

Can we expect better Wireguard performance with official Asus support? Or you should I look for more powerful router?

You can pose this question to ASUS

However, RMerlin presumably provides the same WireGuard Kernel module version 1.0.20210124 as used by ASUS in their v386 RC Beta release.

@ZebMcKayhan has kindly compiled newer WireGuard Kernel modules for a couple of AX routers

GitHub - ZebMcKayhan/Wireguard: Wireguard kernel module and userspace-tools for asus routers running Entware 3.10

Wireguard kernel module and userspace-tools for asus routers running Entware 3.10 - ZebMcKayhan/Wireguard

github.com

but I don't know if there are tangible/measurable performance increases.

I guess you'll have to see if anyone running WireGuard thru' a 1Gbps connection can provide their WireGuard throughput metrics with Flow Cache ENABLED/DISABLED for hardware recommendations.

 

[JGrana]

Already has.... December 2021

and I can personally verify that when I loaded ASUS 386 Beta RC3-3 Public Beta onto my RT-AX86U it was indeed disabled when WireGuard was configured, but cannot vouch for other AX models such as your RT-AX88U

And I can personally verify that if you don’t disable flow cash, your log file will fill with lots of “blog_emit” warnings.

 

[archiel]

And I can personally verify that if you don’t disable flow cash, your log file will fill with lots of “blog_emit” warnings.

In my case no 'blog_emit', but if I run

tcpdump -i eth0 -vv icmp6

then

protocol 0800 is buggy, dev eth0
protocol 0800 is buggy, dev eth0
protocol 0800 is buggy, dev eth0
net_ratelimit: nnnn callbacks suppressed
protocol 0800 is buggy, dev eth0
etc

with the occasional protocol 0000 and 86dd
These go away if flow cache is disabled, but I have no idea if they actually mean anything.

 

[archiel]

crap... I dont really find anything that would do this, and without having this myself, it is extremely hard to debug.

The only thing left could be to find the scope of the problem. meaning how much of you prefix could you change. while you accidently encountered the bug, you used a /48 prefix instead of your /64, so you changed 4 more numbers than your prefix allowed. this package got forwarded to eth0 successfully. so could you change more, like
2a02:c7f:: -->?
2a02:: -->?

to speed up testing, just check if the packages arrives out on eth0 (using tcpdump), but dont expect any reply. to create an ULA with 2 as prefix is risky as it could cause conflict with ips on the internet. but A.F.A.I.K only ips starting with 2 is used today, so if we could just change something:
2 = 0b0010 /4
so if the limit is /3 then perhaps 3 could be used as well as:
3 = 0b0011 /3
or can we change just one more bit:
4 = 0b0000 /2
best case, maybee even another:
6 = 0b0110 /1
finally:
a = 0b1010 (already checked this to not work)

so the full list to test would be something like:
2a02:c7f:: -->?
2a02:: -->? (/16)
2a00:: -->? (/12)
2af0:: --> ? (/8)
2000:: -->? (/4)
3000:: -->? (/3)
4000:: --> ? (/2)
6000:: --> ? (/1)
a000:: --> dont work /0

I know it is a lot of testing and will probably take like half an hour to go through. on the other hand, you could do by successive approximation, start in the middle (2000: and if it works, continue down if not continue up. Hopefully we could find a prefix that we could use that is not used on the internet.

whenever you found the smallest amount, try to use this with your masquarading rule (change the -s address accordingly) and hopefully it should work so you get a reply.

To finally resolve this, we probably need someone who has deeper knowledge on kernel routing than me.

I went in a slightly different direction, not sure if it helps

Before changing the range, I looked at adding a public IPv6 address which fell outside the allocated range 2a02:c7f:f0c3:1000::1/56

So I tried peer new ip=10.50.5.1/24 ipv6=2a02:c7f:f0c3:2010::1/64 and as with the aa, fa, & fd ranges, I had no IPv6

I then tried peer new ip=10.50.3.1/24 ipv6=2a02:c7f:f0c3:1010::1/48, which worked with IPv6 but note the the designated address is inside the allocated range, whereas peer new ip=10.50.4.1/24 ipv6=2a02:c7f:f0c3:2010::1/48 failed on IPv6, presumably because the designated address is outside.

It therefore seems possible that the routing rule is not checking to see the /CIDR range (I could test this further if you think it would help) but rather whether the selected IPv6 address falls inside the allocated range, so that IPv6 works if it does, and fails if it doesn't.

@Martineau I also looked at creating server using an ipv6 /16 address range peer new ip=10.50.6.1/24 ipv6=2a02:c7f:f0c3:1080::1/16 and ended up with IPv6 working and IPv4 broken. Noted that the device addresses were showing as
Address = 10.50.6.1/32,2a02:c7f:f0c3:1080::2/128, rather than the expected
Address = 10.50.6.2/32,2a02:c7f:f0c3:1080::2/128
and in the server conf
AllowedIPs = 10.50.6.1/32,2a02:c7f:f0c3:1080::2/128 rather than the expected
AllowedIPs = 10.50.6.2/32,2a02:c7f:f0c3:1080::2/128

After editing the server and device conf on the router and the device conf on the phone, IPv4 and IPv6 were fine.

 

[ZebMcKayhan]

I went in a slightly different direction, not sure if it helps

Before changing the range, I looked at adding a public IPv6 address which fell outside the allocated range 2a02:c7f:f0c3:1000::1/56

So I tried peer new ip=10.50.5.1/24 ipv6=2a02:c7f:f0c3:2010::1/64 and as with the aa, fa, & fd ranges, I had no IPv6

I then tried peer new ip=10.50.3.1/24 ipv6=2a02:c7f:f0c3:1010::1/48, which worked with IPv6 but note the the designated address is inside the allocated range, whereas peer new ip=10.50.4.1/24 ipv6=2a02:c7f:f0c3:2010::1/48 failed on IPv6, presumably because the designated address is outside.

It therefore seems possible that the routing rule is not checking to see the /CIDR range (I could test this further if you think it would help) but rather whether the selected IPv6 address falls inside the allocated range, so that IPv6 works if it does, and fails if it doesn't.

@Martineau I also looked at creating server using an ipv6 /16 address range peer new ip=10.50.6.1/24 ipv6=2a02:c7f:f0c3:1080::1/16 and ended up with IPv6 working and IPv4 broken. Noted that the device addresses were showing as
Address = 10.50.6.1/32,2a02:c7f:f0c3:1080::2/128, rather than the expected
Address = 10.50.6.2/32,2a02:c7f:f0c3:1080::2/128
and in the server conf
AllowedIPs = 10.50.6.1/32,2a02:c7f:f0c3:1080::2/128 rather than the expected
AllowedIPs = 10.50.6.2/32,2a02:c7f:f0c3:1080::2/128

After editing the server and device conf on the router and the device conf on the phone, IPv4 and IPv6 were fine.

I'm not sure I'm following you, but I agree that the cidr doesn't matter. For wg21 it should always be 64 or higher. My point was to see how far back you could change the ip while packages are still forwarded to eth0.

There could be any number of resons that your ipv6 is not working so right now only use tcpdump to see if the pings reaches out on eth0. Thats all that matters. You will never get a reply until we get the masquarading to work but lets focus on one problem at the time.

You prefix is 56 and I read your post as if you make changes outside this, it doesn't work. Altough in your previous test
In post #732:

admin@RT-AX88U-5050:/tmp/home/root# tcpdump -i eth0 -vv icmp6
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
19:20:51.231640 IP6 (flowlabel 0xa5c66, hlim 63, next-header ICMPv6 (58) payload length: 64) 2a02:c7f:f0c3::2 > 2600::: [icmp6 sum ok] ICMP6, echo request, seq 1
19:20:55.271737 IP6 (flowlabel 0xa5c66, hlim 63, next-header ICMPv6 (58) payload length: 64) 2a02:c7f:f0c3::2 > 2600::: [icmp6 sum ok] ICMP6, echo request, seq 1
19:20:59.331206 IP6 (flowlabel 0xa5c66, hlim 63, next-header ICMPv6 (58) payload length: 64) 2a02:c7f:f0c3::2 > 2600::: [icmp6 sum ok] ICMP6, echo request, seq 1

Here, packages were forwarded out Altough only /48 is kept the same and packages are forwarded to eth0. You never get any reply since the source address is not yours, but we can setup masquarading or snat later.
What if you try
2a02::1/64 as wg21 adress. Do packages still gets forwarded (only /16 kept)?

Whenever you change wg21+client ip you also need to update the masquarade rule if there should be any chance for you to actually get a reply. But Im still not sure the masquarade rule will work on interface with only link local address.
Another thought:
Any interface could have many ipv6 addresses, so alias doesn't exist, you just add more addresses. What if you add an address within your ULA to eth0? Like:

ip -6 address add dev eth0 fd36:7ef1:2add:aa88::100/128

Will that make any difference?

 

[Martineau]

@Martineau I also looked at creating server using an ipv6 /16 address range peer new ip=10.50.6.1/24 ipv6=2a02:c7f:f0c3:1080::1/16 and ended up with IPv6 working and IPv4 broken. Noted that the device addresses were showing as
Address = 10.50.6.1/32,2a02:c7f:f0c3:1080::2/128, rather than the expected
Address = 10.50.6.2/32,2a02:c7f:f0c3:1080::2/128
and in the server conf
AllowedIPs = 10.50.6.1/32,2a02:c7f:f0c3:1080::2/128 rather than the expected
AllowedIPs = 10.50.6.2/32,2a02:c7f:f0c3:1080::2/128

After editing the server and device conf on the router and the device conf on the phone, IPv4 and IPv6 were fine.

Whoops - extremely embarrassing ....concentrating on the IPv6 'server' Peer Base IP stuff and I borked the change for IPv4

I've uploaded wireguard_manager Beta v4.16b4

Update wg_manager.sh · MartineauUK/wireguard@3204469

FIX: Creating a new Road-Warrior 'device' Peer does not get correct IPv4 address assigned but instead uses the 'server' Peer base IPv4 address - Thanks SNB Forum member @archiel

github.com

To apply the patch

e  = Exit Script [?]

E:Option ==> uf dev

 

[Martineau]

I've uploaded wireguard_manager Beta v4.16b5

Update wg_manager.sh · MartineauUK/wireguard@d3583f7

CHANGE: Peer configuration .conf files must be in (cross-platform supported) 'wg/wg-quick' format i.e. comments are now TRUE comments i.e. '#Address = ' is now ignored, ...

github.com

Effectively wireguard_manager Peer .conf files are now fully WYSIWYG and wireguard_manager expects them now to be in wg/wg-quick format

Unfortunately, the existing wireguard_manager Peer .conf files will need to be converted either manually, or I have created a crude command to attempt an auto conversion.

e.g. In the following example

e  = Exit Script [?]

E:Option ==> formatwg-quick

    Checking Peer Config for conversion to wg-quick format:

wg11.conf
wg12.conf
wg21.conf
wg22.conf

    [✔] 'wg12.conf' converted to 'wg-quick' format
    Press y to convert 'wg22.conf' multiple '#Pre/#Post' directives or press [Enter] to SKIP:

• wg11.conf and wg21.conf are deemed to be already in wg/wg-quick compatible format.
• wg12.conf was auto converted - 'client' Peers are usually imported so it is safe to assume that only the unique '#Address =' and '#DNS =' directives were converted.
• wg22.conf contains several '#Pre*'/'#Post*' directives so whilst replying 'y' should be safe, it is flagged to remind you to check the conversion!

NOTE: You may also explicitly specify the .conf file (doesn't have to physically have .conf suffix) such as formatwg-quick Cabin.conf.Imported


****Don't forget to restart the converted Peers!!!****


Worst case scenario would be to delete ALL Peers and re-import (peer import xxxxxx) the appropriate 'client' Peer(s), and recreate (peer new) the necessary 'server' Peer(s)

Spoiler: Example showing running converted .conf

e.g. A dump of running Peer .conf

e  = Exit Script [?]

E:Option ==> raw wg22

    ================Config===============
# RT-AX58U Server 2
[Interface]
PrivateKey = oCb9xgcgKFuqWn/O0gpyTAht7jlUF7RUX/Of2fOG2UM=
Address = 10.50.2.1/24
ListenPort = 11502

e  = Exit Script [?]

E:Option ==> wg showconf wg22

    WireGuard Userspace Tool:

[Interface]
ListenPort = 11502
PrivateKey = oCb9xgcgKFuqWn/O0gpyTAht7jlUF7RUX/Of2fOG2UM=

e  = Exit Script [?]

E:Option ==> wg

    WireGuard Userspace Tool:

interface: wg21
  public key: rRZLITFAHGq6grFgg4gi8IMf4ti7lngZL2H4qDz78SA=
  private key: (hidden)
  listening port: 51820

interface: wg22
  public key: fW/5OE46hXmEHi2aGUtZx0L1X6rrTfA4Y+pSgnye6Xg=
  private key: (hidden)
  listening port: 11502

To upgrade use

e  = Exit Script [?]

E:Option ==> uf dev

 

[ZebMcKayhan]

I've uploaded wireguard_manager Beta v4.16b5

Update wg_manager.sh · MartineauUK/wireguard@d3583f7

CHANGE: Peer configuration .conf files must be in (cross-platform supported) 'wg/wg-quick' format i.e. comments are now TRUE comments i.e. '#Address = ' is now ignored, ...

github.com

Effectively wireguard_manager Peer .conf files are now fully WYSIWYG and wireguard_manager expects them now to be in wg/wg-quick format

Unfortunately, the existing wireguard_manager Peer .conf files will need to be converted either manually, or I have created a crude command to attempt an auto conversion.

e.g. In the following example

e  = Exit Script [?]

E:Option ==> formatwg-quick

    Checking Peer Config for conversion to wg-quick format:

wg11.conf
wg12.conf
wg21.conf
wg22.conf

    [✔] 'wg12.conf' converted to 'wg-quick' format
    Press y to convert 'wg22.conf' multiple '#Pre/#Post' directives or press [Enter] to SKIP:

• wg11.conf and wg21.conf are deemed to be already in wg/wg-quick compatible format.
• wg12.conf was auto converted - 'client' Peers are usually imported so it is safe to assume that only the unique '#Address =' and '#DNS =' directives were converted.
• wg22.conf contains several '#Pre*'/'#Post*' directives so whilst replying 'y' should be safe, it is flagged to remind you to check the conversion!

NOTE: You may also explicitly specify the .conf file (doesn't have to physically have .conf suffix) such as formatwg-quick Cabin.conf.Imported

Worst case scenario would be to delete ALL Peers and re-import (peer import xxxxxx) the appropriate 'client' Peer(s), and recreate (peer new) the necessary 'server' Peer(s)

Spoiler: Example showing running converted .conf

e.g. A dump of running Peer .conf

e  = Exit Script [?]

E:Option ==> raw wg22

    ================Config===============
# RT-AX58U Server 2
[Interface]
PrivateKey = oCb9xgcgKFuqWn/O0gpyTAht7jlUF7RUX/Of2fOG2UM=
Address = 10.50.2.1/24
ListenPort = 11502

e  = Exit Script [?]

E:Option ==> wg showconf wg22

    WireGuard Userspace Tool:

[Interface]
ListenPort = 11502
PrivateKey = oCb9xgcgKFuqWn/O0gpyTAht7jlUF7RUX/Of2fOG2UM=

e  = Exit Script [?]

E:Option ==> wg

    WireGuard Userspace Tool:

interface: wg21
  public key: rRZLITFAHGq6grFgg4gi8IMf4ti7lngZL2H4qDz78SA=
  private key: (hidden)
  listening port: 51820

interface: wg22
  public key: fW/5OE46hXmEHi2aGUtZx0L1X6rrTfA4Y+pSgnye6Xg=
  private key: (hidden)
  listening port: 11502

To upgrade use

e  = Exit Script [?]

E:Option ==> uf dev

Soo, moving to generate temporary files for the wg setconf? I think this a great choice for the future!

Hazzle-free update, conversion (formatwg-quick) of the conf files (wg11, wg21 and wg21) and starting them again. All works!

Thanks!

 

[Martineau]

Soo, moving to generate temporary files for the wg setconf? I think this a great choice for the future!

"A wise man changes his mind, a fool never will"

I assume that's why this exists?

wg addconf

Usage: wg addconf <interface> <configuration filename>

Hazzle-free update, conversion (formatwg-quick) of the conf files (wg11, wg21 and wg21) and starting them again. All works!

Hassle free?...surely I've left some dumb coding mistake....you've probably not looked hard enough!

Q. Can you please do me a favour and check if the following commands show a priority 220 rule to table 220

ip rule

0:        from all lookup local
220:      from all lookup 220

ip -6 rule

0:        from all lookup local
220:      from all lookup 220
32766:    from all lookup main

 

[ZebMcKayhan]

Q. Can you please do me a favour and check if the following commands show a priority 220 rule to table 220

Uuh, no:

admin@RT-AC86U-D7D8:/tmp/home/root# ip rule
0:      from all lookup local
9810:   from all fwmark 0xd2 lookup 210
9900:   from 192.168.1.1/24 fwmark 0x8000 lookup main
9910:   from all to 192.168.1.1/16 lookup main
9911:   from 192.168.1.1/24 lookup 121
9921:   from 192.168.6.0/24 lookup 122
9991:   from all fwmark 0x1000/0x1000 lookup 121
32766:  from all lookup main
32767:  from all lookup default
admin@RT-AC86U-D7D8:/tmp/home/root# ip -6 rule
0:      from all lookup local
9810:   from all fwmark 0xd2 lookup 210
9900:   from fdff:a37f:fa75:1::/64 fwmark 0x8000 lookup main
9910:   from all to fdff:a37f:fa75::/48 lookup main
9911:   from fdff:a37f:fa75:1::/64 lookup 121
9921:   from fdff:a37f:fa75:6::/64 lookup 122
9991:   from all fwmark 0x1000/0x1000 lookup 121
32766:  from all lookup main
admin@RT-AC86U-D7D8:/tmp/home/root#

Who would want to redirect ALL Traffic to another routing table? Well, not my business.

 

[RMerlin]

Heads up - Wireguard components are not present in the 386.5 release for the GT-AXE11000 as I forgot to enable it for that model when I added it to the build profiles. They should be added in the next release.

 

[Martineau]

Heads up - Wireguard components are not present in the 386.5 release for the GT-AXE11000 as I forgot to enable it for that model when I added it to the build profiles. They should be added in the next release.

OK thanks.

So just one single GT-AXE11000 user (out of 190 downloads?) eventually raised a ticket yesterday on my Github asking why my script complained about the lack of WireGuard modules.

I'm sure we'll manage to cope with this magnitude of potential new user support issues!

 

[JGrana]

I did the uf dev and formatwg-quick on both sites (Home/Cabin). Restarted both sides.
Site to site running fine!

 

[archiel]

crap... I dont really find anything that would do this, and without having this myself, it is extremely hard to debug.

The only thing left could be to find the scope of the problem. meaning how much of you prefix could you change. while you accidently encountered the bug, you used a /48 prefix instead of your /64, so you changed 4 more numbers than your prefix allowed. this package got forwarded to eth0 successfully. so could you change more, like
2a02:c7f:: -->?
2a02:: -->?

to speed up testing, just check if the packages arrives out on eth0 (using tcpdump), but dont expect any reply. to create an ULA with 2 as prefix is risky as it could cause conflict with ips on the internet. but A.F.A.I.K only ips starting with 2 is used today, so if we could just change something:
2 = 0b0010 /4
so if the limit is /3 then perhaps 3 could be used as well as:
3 = 0b0011 /3
or can we change just one more bit:
4 = 0b0000 /2
best case, maybee even another:
6 = 0b0110 /1
finally:
a = 0b1010 (already checked this to not work)

so the full list to test would be something like:
2a02:c7f:: -->?
2a02:: -->? (/16)
2a00:: -->? (/12)
2af0:: --> ? (/8)
2000:: -->? (/4)
3000:: -->? (/3)
4000:: --> ? (/2)
6000:: --> ? (/1)
a000:: --> dont work /0

I know it is a lot of testing and will probably take like half an hour to go through. on the other hand, you could do by successive approximation, start in the middle (2000: and if it works, continue down if not continue up. Hopefully we could find a prefix that we could use that is not used on the internet.

whenever you found the smallest amount, try to use this with your masquarading rule (change the -s address accordingly) and hopefully it should work so you get a reply.

To finally resolve this, we probably need someone who has deeper knowledge on kernel routing than me.

Okay so I went back to the beginning testing

peer new ip=10.50.6.1/24 ipv6=address/nn

ping from my phone using PingTools for Android and testing on the router with tcpdump

tcpdump -i eth0 -vv icmp6

and then adding routing via

ip6tables -t nat -I POSTROUTING -s address/nn -o eth0 -j SNAT --to-source 2a02:c7f:f0c3:1000::1

where 2a02:c7f:f0c3:1000::1 is may current ipv6 WAN and testing the ping again.

In the vast majority of cases I could see the ping immediately on tcpdump and after adding the routing the ping and the echo. In each of these cases I also had full IPv6 connectivity on the phone.

I started with a /48 and was able to reduce down to /8 without problem. However 2000::1/4 was a problem (though 1000::1/4, 3000::1/4 to e000::1/4 were fine) as was f000::1/4. I then tested some fc.../8 and fd.../8 addresses and as previously noted, no ping and no IPv6.

In fact with both 2600::1/8 and f000::1/4 both started okay but quickly messed up, stopping unbound, messing up the ipv6 routing and generating this log

Mar 14 20:48:30 RT-AX88U-5050 kernel: <unknown>: hw csum failure
Mar 14 20:48:30 RT-AX88U-5050 kernel: CPU: 0 PID: 4477 Comm: unbound Tainted: P           O    4.1.51 #2
Mar 14 20:48:30 RT-AX88U-5050 kernel: Hardware name: Broadcom-v8A (DT)
Mar 14 20:48:30 RT-AX88U-5050 kernel: Call trace:
Mar 14 20:48:30 RT-AX88U-5050 kernel: [<ffffffc000087658>] dump_backtrace+0x0/0x150
Mar 14 20:48:30 RT-AX88U-5050 kernel: [<ffffffc0000877bc>] show_stack+0x14/0x20
Mar 14 20:48:30 RT-AX88U-5050 kernel: [<ffffffc00052ae48>] dump_stack+0x90/0xb0
Mar 14 20:48:30 RT-AX88U-5050 kernel: [<ffffffc0003dd194>] netdev_rx_csum_fault+0x3c/0x50
Mar 14 20:48:30 RT-AX88U-5050 kernel: [<ffffffc0003d2408>] skb_copy_and_csum_datagram_msg+0xe8/0xf8
Mar 14 20:48:30 RT-AX88U-5050 kernel: [<ffffffc0004d5aa8>] udpv6_recvmsg+0x2a0/0x7d8
Mar 14 20:48:30 RT-AX88U-5050 kernel: [<ffffffc000484654>] inet_recvmsg+0xa4/0xc8
Mar 14 20:48:30 RT-AX88U-5050 kernel: [<ffffffc0003c36b0>] SyS_recvfrom+0xa8/0x128

which repeated until I stopped the WireGuard server and restarted ipv6 on the router.

With 2600::/1 I was taking up part of the Sprint range 2600::/29 and for 2000::/4 and f000::/4 the range was straddling the reserved / restricted addresses and I assume that this was causing the problem. For more detail on restricted addresses see IANA IPv6 Special-Purpose Address Registry, IPv6 Global Unicast Address Assignments (iana.org) and RFC 4193: Unique Local IPv6 Unicast Addresses (rfc-editor.org) which provides details on the routability of Unique-Local addresses. The Unique-Local prefix is drawn from the IPv6 Global Unicast Address range, but is specified as not globally routed.

If I have understood the position correctly it would seem that for users like me who have dynamic IPv6 from their ISP

1. ULAs (fc00::/7) will not work for internet IPv6
2. Link Local addresses (fe80::/10) cannot work with WireGuard
2. If you are choosing a random address, check the IANA preassigned and reserved addresses and make sure your range does not straddle any of these.
3. A better solution may be to find out your ISP's reserved range and then choose a random address within it. You 'might' have a clash with another user for the same ISP, but the odds are extremely small. For SKY (my ISP approximately I in 1.2676506002282*e^30)

And on an RT-AX88U, adding entware iptables (to allow SNAT) does not (appear) to cause any issues.

@ZebMcKayhan let me know if there any other tests that you think may be helpful.

 

[ZebMcKayhan]

Okay so I went back to the beginning testing

ping from my phone using PingTools for Android and testing on the router with tcpdump

and then adding routing via

where 2a02:c7f:f0c3:1000::1 is may current ipv6 WAN and testing the ping again.

In the vast majority of cases I could see the ping immediately on tcpdump and after adding the routing the ping and the echo. In each of these cases I also had full IPv6 connectivity on the phone.

I started with a /48 and was able to reduce down to /8 without problem. However 2000::1/4 was a problem (though 1000::1/4, 3000::1/4 to e000::1/4 were fine) as was f000::1/4. I then tested some fc.../8 and fd.../8 addresses and as previously noted, no ping and no IPv6.

In fact with both 2600::1/8 and f000::1/4 both started okay but quickly messed up, stopping unbound, messing up the ipv6 routing and generating this log

Mar 14 20:48:30 RT-AX88U-5050 kernel: <unknown>: hw csum failure
Mar 14 20:48:30 RT-AX88U-5050 kernel: CPU: 0 PID: 4477 Comm: unbound Tainted: P           O    4.1.51 #2
Mar 14 20:48:30 RT-AX88U-5050 kernel: Hardware name: Broadcom-v8A (DT)
Mar 14 20:48:30 RT-AX88U-5050 kernel: Call trace:
Mar 14 20:48:30 RT-AX88U-5050 kernel: [<ffffffc000087658>] dump_backtrace+0x0/0x150
Mar 14 20:48:30 RT-AX88U-5050 kernel: [<ffffffc0000877bc>] show_stack+0x14/0x20
Mar 14 20:48:30 RT-AX88U-5050 kernel: [<ffffffc00052ae48>] dump_stack+0x90/0xb0
Mar 14 20:48:30 RT-AX88U-5050 kernel: [<ffffffc0003dd194>] netdev_rx_csum_fault+0x3c/0x50
Mar 14 20:48:30 RT-AX88U-5050 kernel: [<ffffffc0003d2408>] skb_copy_and_csum_datagram_msg+0xe8/0xf8
Mar 14 20:48:30 RT-AX88U-5050 kernel: [<ffffffc0004d5aa8>] udpv6_recvmsg+0x2a0/0x7d8
Mar 14 20:48:30 RT-AX88U-5050 kernel: [<ffffffc000484654>] inet_recvmsg+0xa4/0xc8
Mar 14 20:48:30 RT-AX88U-5050 kernel: [<ffffffc0003c36b0>] SyS_recvfrom+0xa8/0x128

which repeated until I stopped the WireGuard server and restarted ipv6 on the router.

With 2600::/1 I was taking up part of the Sprint range 2600::/29 and for 2000::/4 and f000::/4 the range was straddling the reserved / restricted addresses and I assume that this was causing the problem. For more detail on restricted addresses see IANA IPv6 Special-Purpose Address Registry, IPv6 Global Unicast Address Assignments (iana.org) and RFC 4193: Unique Local IPv6 Unicast Addresses (rfc-editor.org) which provides details on the routability of Unique-Local addresses. The Unique-Local prefix is drawn from the IPv6 Global Unicast Address range, but is specified as not globally routed.

If I have understood the position correctly it would seem that for users like me who have dynamic IPv6 from their ISP

1. ULAs (fc00::/7) will not work for internet IPv6
2. Link Local addresses (fe80::/10) cannot work with WireGuard
2. If you are choosing a random address, check the IANA preassigned and reserved addresses and make sure your range does not straddle any of these.
3. A better solution may be to find out your ISP's reserved range and then choose a random address within it. You 'might' have a clash with another user for the same ISP, but the odds are extremely small. For SKY (my ISP approximately I in 1.2676506002282*e^30)

And on an RT-AX88U, adding entware iptables (to allow SNAT) does not (appear) to cause any issues.

@ZebMcKayhan let me know if there any other tests that you think may be helpful.

Wow, great testing! Well, one last thing, maybee youve already tested this, but do you HAVE to use SNAT for it to work? MASQUARADE dont work for you?

The problem with SNAT is that you would need to lookup the br0 ip address each time and dont know if firewall gets restarted whenever your (or any other) isp decides to change ip. Masquarade would be much more convenient if it is working.

 

[archiel]

Wow, great testing! Well, one last thing, maybee youve already tested this, but do you HAVE to use SNAT for it to work? MASQUARADE dont work for you?

The problem with SNAT is that you would need to lookup the br0 ip address each time and dont know if firewall gets restarted whenever your (or any other) isp decides to change ip. Masquarade would be much more convenient if it is working.

What would be the format to test MASQUARADE (I have lost track of which of the various commands I should be using)? Otherwise I agree you would need to have a script for up/down which checks the current WAN IP and updates it.

 

[ZebMcKayhan]

To whomever it concerns:

Just found out that NPT6 (network prefix translation ipv6) is available in our routers.

NPT6 is a stateless (no connection tracking involved) address translation that only translates the prefix but keeps the device-suffix. This is the prefferred translation method whenever possible.

To translate wg21 (fc00:192:168:100::/64) to wg11 (fdab:1337:1337:69::/64) as it goes out wg11:

ip6tables -t mangle -I POSTROUTING -s fc00:192:168:100::/64 -o wg11 -j SNPT --src-pfx fc00:192:168:100::/64 --dst-pfx fdab:1337:1337:69::/64

and to translate the replies back:

ip6tables -t mangle -I PREROUTING -i wg11 -d fdab:1337:1337:69::/64 -j DNPT --src-pfx fdab:1337:1337:69::/64 --dst-pfx fc00:192:168:100::/64

only works with /64 and would in theory make it possible to run ULA inside your network and global network on outside while still comply to rfc's as all nodes are still reachable from outside.

Have only tested on outgoing packages and it works:

admin@RT-AC86U-D7D8:/tmp/home/root# tcpdump -i wg11 icmp6
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on wg11, link-type RAW (Raw IP), capture size 262144 bytes
22:03:55.120948 IP6 fc00:192:168:100::2 > arn11s11-in-x0e.1e100.net: ICMP6, echo request, seq 1, length 64
^C
1 packet captured
1 packet received by filter
0 packets dropped by kernel

admin@RT-AC86U-D7D8:/tmp/home/root# ip6tables -t mangle -I POSTROUTING -s fc00:192:168:100::/64 -o wg11 -j SNPT --src-pfx fc00:192:168:100::/64  --dst-pfx fdab:1337:1337:69::/64
admin@RT-AC86U-D7D8:/tmp/home/root# tcpdump -i wg11 icmp6
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on wg11, link-type RAW (Raw IP), capture size 262144 bytes
22:04:13.806984 IP6 fdab:1337:1337:69:db77::2 > arn11s11-in-x0e.1e100.net: ICMP6, echo request, seq 1, length 64
^C
1 packet captured
1 packet received by filter
0 packets dropped by kernel