With Flow Cache ENABLED; if WireGuard Peer speed/throughput performance is deemed acceptable, then there is probably no need to DISABLE Flow Cache (despite ASUS/Broadcom seemingly insisting that it should be DISABLED for specific AX routers?).The point is that even when no clients are connected via Wireguard I was only able to measure about a third of regular speed.
There are many threads discussing the impact of Flow Cache on 1Gbps connections.I can't use Wireguard with fc enabled at all (throughput is about 2-3 mbps and router's cores at almost 100 %). If fc disabled Wireguard performance is great. However, even if there are none active Wireguard peers, Speedtest download results won't reach the values I have with fc enabled (300 - 500 mbps down and 100mbps up instead of 940 mbps down and 100 mpbs up).
You can pose this question to ASUSCan we expect better Wireguard performance with official Asus support? Or you should I look for more powerful router?
And I can personally verify that if you don’t disable flow cash, your log file will fill with lots of “blog_emit” warnings.Already has.... December 2021
and I can personally verify that when I loaded ASUS 386 Beta RC3-3 Public Beta onto my RT-AX86U it was indeed disabled when WireGuard was configured, but cannot vouch for other AX models such as your RT-AX88U
In my case no 'blog_emit', but if I runAnd I can personally verify that if you don’t disable flow cash, your log file will fill with lots of “blog_emit” warnings.
tcpdump -i eth0 -vv icmp6
protocol 0800 is buggy, dev eth0
protocol 0800 is buggy, dev eth0
protocol 0800 is buggy, dev eth0
net_ratelimit: nnnn callbacks suppressed
protocol 0800 is buggy, dev eth0
etc
I went in a slightly different direction, not sure if it helpscrap... I dont really find anything that would do this, and without having this myself, it is extremely hard to debug.
The only thing left could be to find the scope of the problem. meaning how much of you prefix could you change. while you accidently encountered the bug, you used a /48 prefix instead of your /64, so you changed 4 more numbers than your prefix allowed. this package got forwarded to eth0 successfully. so could you change more, like
2a02:c7f:: -->?
2a02:: -->?
to speed up testing, just check if the packages arrives out on eth0 (using tcpdump), but dont expect any reply. to create an ULA with 2 as prefix is risky as it could cause conflict with ips on the internet. but A.F.A.I.K only ips starting with 2 is used today, so if we could just change something:
2 = 0b0010 /4
so if the limit is /3 then perhaps 3 could be used as well as:
3 = 0b0011 /3
or can we change just one more bit:
4 = 0b0000 /2
best case, maybee even another:
6 = 0b0110 /1
finally:
a = 0b1010 (already checked this to not work)
so the full list to test would be something like:
2a02:c7f:: -->?
2a02:: -->? (/16)
2a00:: -->? (/12)
2af0:: --> ? (/8)
2000:: -->? (/4)
3000:: -->? (/3)
4000:: --> ? (/2)
6000:: --> ? (/1)
a000:: --> dont work /0
I know it is a lot of testing and will probably take like half an hour to go through. on the other hand, you could do by successive approximation, start in the middle (2000: and if it works, continue down if not continue up. Hopefully we could find a prefix that we could use that is not used on the internet.
whenever you found the smallest amount, try to use this with your masquarading rule (change the -s address accordingly) and hopefully it should work so you get a reply.
To finally resolve this, we probably need someone who has deeper knowledge on kernel routing than me.
I'm not sure I'm following you, but I agree that the cidr doesn't matter. For wg21 it should always be 64 or higher. My point was to see how far back you could change the ip while packages are still forwarded to eth0.I went in a slightly different direction, not sure if it helps
Before changing the range, I looked at adding a public IPv6 address which fell outside the allocated range 2a02:c7f:f0c3:1000::1/56
So I tried peer new ip=10.50.5.1/24 ipv6=2a02:c7f:f0c3:2010::1/64 and as with the aa, fa, & fd ranges, I had no IPv6
I then tried peer new ip=10.50.3.1/24 ipv6=2a02:c7f:f0c3:1010::1/48, which worked with IPv6 but note the the designated address is inside the allocated range, whereas peer new ip=10.50.4.1/24 ipv6=2a02:c7f:f0c3:2010::1/48 failed on IPv6, presumably because the designated address is outside.
It therefore seems possible that the routing rule is not checking to see the /CIDR range (I could test this further if you think it would help) but rather whether the selected IPv6 address falls inside the allocated range, so that IPv6 works if it does, and fails if it doesn't.
@Martineau I also looked at creating server using an ipv6 /16 address range peer new ip=10.50.6.1/24 ipv6=2a02:c7f:f0c3:1080::1/16 and ended up with IPv6 working and IPv4 broken. Noted that the device addresses were showing as
Address = 10.50.6.1/32,2a02:c7f:f0c3:1080::2/128, rather than the expected
Address = 10.50.6.2/32,2a02:c7f:f0c3:1080::2/128
and in the server conf
AllowedIPs = 10.50.6.1/32,2a02:c7f:f0c3:1080::2/128 rather than the expected
AllowedIPs = 10.50.6.2/32,2a02:c7f:f0c3:1080::2/128
After editing the server and device conf on the router and the device conf on the phone, IPv4 and IPv6 were fine.
Here, packages were forwarded out Altough only /48 is kept the same and packages are forwarded to eth0. You never get any reply since the source address is not yours, but we can setup masquarading or snat later.admin@RT-AX88U-5050:/tmp/home/root# tcpdump -i eth0 -vv icmp6
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
19:20:51.231640 IP6 (flowlabel 0xa5c66, hlim 63, next-header ICMPv6 (58) payload length: 64) 2a02:c7f:f0c3::2 > 2600::: [icmp6 sum ok] ICMP6, echo request, seq 1
19:20:55.271737 IP6 (flowlabel 0xa5c66, hlim 63, next-header ICMPv6 (58) payload length: 64) 2a02:c7f:f0c3::2 > 2600::: [icmp6 sum ok] ICMP6, echo request, seq 1
19:20:59.331206 IP6 (flowlabel 0xa5c66, hlim 63, next-header ICMPv6 (58) payload length: 64) 2a02:c7f:f0c3::2 > 2600::: [icmp6 sum ok] ICMP6, echo request, seq 1
ip -6 address add dev eth0 fd36:7ef1:2add:aa88::100/128
Whoops - extremely embarrassing ....concentrating on the IPv6 'server' Peer Base IP stuff and I borked the change for IPv4@Martineau I also looked at creating server using an ipv6 /16 address range peer new ip=10.50.6.1/24 ipv6=2a02:c7f:f0c3:1080::1/16 and ended up with IPv6 working and IPv4 broken. Noted that the device addresses were showing as
Address = 10.50.6.1/32,2a02:c7f:f0c3:1080::2/128, rather than the expected
Address = 10.50.6.2/32,2a02:c7f:f0c3:1080::2/128
and in the server conf
AllowedIPs = 10.50.6.1/32,2a02:c7f:f0c3:1080::2/128 rather than the expected
AllowedIPs = 10.50.6.2/32,2a02:c7f:f0c3:1080::2/128
After editing the server and device conf on the router and the device conf on the phone, IPv4 and IPv6 were fine.
wireguard_manager
Beta v4.16b4e = Exit Script [?]
E:Option ==> uf dev
wireguard_manager
Beta v4.16b5wireguard_manager
Peer .conf files are now fully WYSIWYG and wireguard_manager
expects them now to be in wg
/wg-quick
formatwireguard_manager
Peer .conf files will need to be converted either manually, or I have created a crude command to attempt an auto conversion.e = Exit Script [?]
E:Option ==> formatwg-quick
Checking Peer Config for conversion to wg-quick format:
wg11.conf
wg12.conf
wg21.conf
wg22.conf
[✔] 'wg12.conf' converted to 'wg-quick' format
Press y to convert 'wg22.conf' multiple '#Pre/#Post' directives or press [Enter] to SKIP:
wg
/wg-quick
compatible format.formatwg-quick Cabin.conf.Imported
peer import xxxxxx
) the appropriate 'client' Peer(s), and recreate (peer new
) the necessary 'server' Peer(s)e = Exit Script [?]
E:Option ==> raw wg22
================Config===============
# RT-AX58U Server 2
[Interface]
PrivateKey = oCb9xgcgKFuqWn/O0gpyTAht7jlUF7RUX/Of2fOG2UM=
Address = 10.50.2.1/24
ListenPort = 11502
e = Exit Script [?]
E:Option ==> wg showconf wg22
WireGuard Userspace Tool:
[Interface]
ListenPort = 11502
PrivateKey = oCb9xgcgKFuqWn/O0gpyTAht7jlUF7RUX/Of2fOG2UM=
e = Exit Script [?]
E:Option ==> wg
WireGuard Userspace Tool:
interface: wg21
public key: rRZLITFAHGq6grFgg4gi8IMf4ti7lngZL2H4qDz78SA=
private key: (hidden)
listening port: 51820
interface: wg22
public key: fW/5OE46hXmEHi2aGUtZx0L1X6rrTfA4Y+pSgnye6Xg=
private key: (hidden)
listening port: 11502
e = Exit Script [?]
E:Option ==> uf dev
Soo, moving to generate temporary files for the wg setconf? I think this a great choice for the future!I've uploadedwireguard_manager
Beta v4.16b5
EffectivelyUpdate wg_manager.sh · MartineauUK/wireguard@d3583f7
CHANGE: Peer configuration .conf files must be in (cross-platform supported) 'wg/wg-quick' format i.e. comments are now TRUE comments i.e. '#Address = ' is now ignored, ...github.comwireguard_manager
Peer .conf files are now fully WYSIWYG andwireguard_manager
expects them now to be inwg
/wg-quick
format
Unfortunately, the existingwireguard_manager
Peer .conf files will need to be converted either manually, or I have created a crude command to attempt an auto conversion.
e.g. In the following example
Code:e = Exit Script [?] E:Option ==> formatwg-quick Checking Peer Config for conversion to wg-quick format: wg11.conf wg12.conf wg21.conf wg22.conf [✔] 'wg12.conf' converted to 'wg-quick' format Press y to convert 'wg22.conf' multiple '#Pre/#Post' directives or press [Enter] to SKIP:
NOTE: You may also explicitly specify the .conf file (doesn't have to physically have .conf suffix) such as
- wg11.conf and wg21.conf are deemed to be already in
wg
/wg-quick
compatible format.- wg12.conf was auto converted - 'client' Peers are usually imported so it is safe to assume that only the unique '#Address =' and '#DNS =' directives were converted.
- wg22.conf contains several '#Pre*'/'#Post*' directives so whilst replying 'y' should be safe, it is flagged to remind you to check the conversion!
formatwg-quick Cabin.conf.Imported
Worst case scenario would be to delete ALL Peers and re-import (peer import xxxxxx
) the appropriate 'client' Peer(s), and recreate (peer new
) the necessary 'server' Peer(s)
To upgrade usee.g. A dump of running Peer .conf
Code:e = Exit Script [?] E:Option ==> raw wg22 ================Config=============== # RT-AX58U Server 2 [Interface] PrivateKey = oCb9xgcgKFuqWn/O0gpyTAht7jlUF7RUX/Of2fOG2UM= Address = 10.50.2.1/24 ListenPort = 11502
Code:e = Exit Script [?] E:Option ==> wg showconf wg22 WireGuard Userspace Tool: [Interface] ListenPort = 11502 PrivateKey = oCb9xgcgKFuqWn/O0gpyTAht7jlUF7RUX/Of2fOG2UM=
Code:e = Exit Script [?] E:Option ==> wg WireGuard Userspace Tool: interface: wg21 public key: rRZLITFAHGq6grFgg4gi8IMf4ti7lngZL2H4qDz78SA= private key: (hidden) listening port: 51820 interface: wg22 public key: fW/5OE46hXmEHi2aGUtZx0L1X6rrTfA4Y+pSgnye6Xg= private key: (hidden) listening port: 11502
Code:e = Exit Script [?] E:Option ==> uf dev
"A wise man changes his mind, a fool never will"Soo, moving to generate temporary files for the wg setconf? I think this a great choice for the future!
wg addconf
Usage: wg addconf <interface> <configuration filename>
Hassle free?...surely I've left some dumb coding mistake....you've probably not looked hard enough!Hazzle-free update, conversion (formatwg-quick) of the conf files (wg11, wg21 and wg21) and starting them again. All works!
ip rule
0: from all lookup local
220: from all lookup 220
ip -6 rule
0: from all lookup local
220: from all lookup 220
32766: from all lookup main
Uuh, no:Q. Can you please do me a favour and check if the following commands show a priority 220 rule to table 220
admin@RT-AC86U-D7D8:/tmp/home/root# ip rule
0: from all lookup local
9810: from all fwmark 0xd2 lookup 210
9900: from 192.168.1.1/24 fwmark 0x8000 lookup main
9910: from all to 192.168.1.1/16 lookup main
9911: from 192.168.1.1/24 lookup 121
9921: from 192.168.6.0/24 lookup 122
9991: from all fwmark 0x1000/0x1000 lookup 121
32766: from all lookup main
32767: from all lookup default
admin@RT-AC86U-D7D8:/tmp/home/root# ip -6 rule
0: from all lookup local
9810: from all fwmark 0xd2 lookup 210
9900: from fdff:a37f:fa75:1::/64 fwmark 0x8000 lookup main
9910: from all to fdff:a37f:fa75::/48 lookup main
9911: from fdff:a37f:fa75:1::/64 lookup 121
9921: from fdff:a37f:fa75:6::/64 lookup 122
9991: from all fwmark 0x1000/0x1000 lookup 121
32766: from all lookup main
admin@RT-AC86U-D7D8:/tmp/home/root#
OK thanks.Heads up - Wireguard components are not present in the 386.5 release for the GT-AXE11000 as I forgot to enable it for that model when I added it to the build profiles. They should be added in the next release.
Okay so I went back to the beginning testingcrap... I dont really find anything that would do this, and without having this myself, it is extremely hard to debug.
The only thing left could be to find the scope of the problem. meaning how much of you prefix could you change. while you accidently encountered the bug, you used a /48 prefix instead of your /64, so you changed 4 more numbers than your prefix allowed. this package got forwarded to eth0 successfully. so could you change more, like
2a02:c7f:: -->?
2a02:: -->?
to speed up testing, just check if the packages arrives out on eth0 (using tcpdump), but dont expect any reply. to create an ULA with 2 as prefix is risky as it could cause conflict with ips on the internet. but A.F.A.I.K only ips starting with 2 is used today, so if we could just change something:
2 = 0b0010 /4
so if the limit is /3 then perhaps 3 could be used as well as:
3 = 0b0011 /3
or can we change just one more bit:
4 = 0b0000 /2
best case, maybee even another:
6 = 0b0110 /1
finally:
a = 0b1010 (already checked this to not work)
so the full list to test would be something like:
2a02:c7f:: -->?
2a02:: -->? (/16)
2a00:: -->? (/12)
2af0:: --> ? (/8)
2000:: -->? (/4)
3000:: -->? (/3)
4000:: --> ? (/2)
6000:: --> ? (/1)
a000:: --> dont work /0
I know it is a lot of testing and will probably take like half an hour to go through. on the other hand, you could do by successive approximation, start in the middle (2000: and if it works, continue down if not continue up. Hopefully we could find a prefix that we could use that is not used on the internet.
whenever you found the smallest amount, try to use this with your masquarading rule (change the -s address accordingly) and hopefully it should work so you get a reply.
To finally resolve this, we probably need someone who has deeper knowledge on kernel routing than me.
ping from my phone using PingTools for Android and testing on the router with tcpdumppeer new ip=10.50.6.1/24 ipv6=address/nn
and then adding routing viatcpdump -i eth0 -vv icmp6
where 2a02:c7f:f0c3:1000::1 is may current ipv6 WAN and testing the ping again.ip6tables -t nat -I POSTROUTING -s address/nn -o eth0 -j SNAT --to-source 2a02:c7f:f0c3:1000::1
Mar 14 20:48:30 RT-AX88U-5050 kernel: <unknown>: hw csum failure
Mar 14 20:48:30 RT-AX88U-5050 kernel: CPU: 0 PID: 4477 Comm: unbound Tainted: P O 4.1.51 #2
Mar 14 20:48:30 RT-AX88U-5050 kernel: Hardware name: Broadcom-v8A (DT)
Mar 14 20:48:30 RT-AX88U-5050 kernel: Call trace:
Mar 14 20:48:30 RT-AX88U-5050 kernel: [<ffffffc000087658>] dump_backtrace+0x0/0x150
Mar 14 20:48:30 RT-AX88U-5050 kernel: [<ffffffc0000877bc>] show_stack+0x14/0x20
Mar 14 20:48:30 RT-AX88U-5050 kernel: [<ffffffc00052ae48>] dump_stack+0x90/0xb0
Mar 14 20:48:30 RT-AX88U-5050 kernel: [<ffffffc0003dd194>] netdev_rx_csum_fault+0x3c/0x50
Mar 14 20:48:30 RT-AX88U-5050 kernel: [<ffffffc0003d2408>] skb_copy_and_csum_datagram_msg+0xe8/0xf8
Mar 14 20:48:30 RT-AX88U-5050 kernel: [<ffffffc0004d5aa8>] udpv6_recvmsg+0x2a0/0x7d8
Mar 14 20:48:30 RT-AX88U-5050 kernel: [<ffffffc000484654>] inet_recvmsg+0xa4/0xc8
Mar 14 20:48:30 RT-AX88U-5050 kernel: [<ffffffc0003c36b0>] SyS_recvfrom+0xa8/0x128
Wow, great testing! Well, one last thing, maybee youve already tested this, but do you HAVE to use SNAT for it to work? MASQUARADE dont work for you?Okay so I went back to the beginning testing
ping from my phone using PingTools for Android and testing on the router with tcpdump
and then adding routing via
where 2a02:c7f:f0c3:1000::1 is may current ipv6 WAN and testing the ping again.
In the vast majority of cases I could see the ping immediately on tcpdump and after adding the routing the ping and the echo. In each of these cases I also had full IPv6 connectivity on the phone.
I started with a /48 and was able to reduce down to /8 without problem. However 2000::1/4 was a problem (though 1000::1/4, 3000::1/4 to e000::1/4 were fine) as was f000::1/4. I then tested some fc.../8 and fd.../8 addresses and as previously noted, no ping and no IPv6.
In fact with both 2600::1/8 and f000::1/4 both started okay but quickly messed up, stopping unbound, messing up the ipv6 routing and generating this log
which repeated until I stopped the WireGuard server and restarted ipv6 on the router.Code:Mar 14 20:48:30 RT-AX88U-5050 kernel: <unknown>: hw csum failure Mar 14 20:48:30 RT-AX88U-5050 kernel: CPU: 0 PID: 4477 Comm: unbound Tainted: P O 4.1.51 #2 Mar 14 20:48:30 RT-AX88U-5050 kernel: Hardware name: Broadcom-v8A (DT) Mar 14 20:48:30 RT-AX88U-5050 kernel: Call trace: Mar 14 20:48:30 RT-AX88U-5050 kernel: [<ffffffc000087658>] dump_backtrace+0x0/0x150 Mar 14 20:48:30 RT-AX88U-5050 kernel: [<ffffffc0000877bc>] show_stack+0x14/0x20 Mar 14 20:48:30 RT-AX88U-5050 kernel: [<ffffffc00052ae48>] dump_stack+0x90/0xb0 Mar 14 20:48:30 RT-AX88U-5050 kernel: [<ffffffc0003dd194>] netdev_rx_csum_fault+0x3c/0x50 Mar 14 20:48:30 RT-AX88U-5050 kernel: [<ffffffc0003d2408>] skb_copy_and_csum_datagram_msg+0xe8/0xf8 Mar 14 20:48:30 RT-AX88U-5050 kernel: [<ffffffc0004d5aa8>] udpv6_recvmsg+0x2a0/0x7d8 Mar 14 20:48:30 RT-AX88U-5050 kernel: [<ffffffc000484654>] inet_recvmsg+0xa4/0xc8 Mar 14 20:48:30 RT-AX88U-5050 kernel: [<ffffffc0003c36b0>] SyS_recvfrom+0xa8/0x128
With 2600::/1 I was taking up part of the Sprint range 2600::/29 and for 2000::/4 and f000::/4 the range was straddling the reserved / restricted addresses and I assume that this was causing the problem. For more detail on restricted addresses see IANA IPv6 Special-Purpose Address Registry, IPv6 Global Unicast Address Assignments (iana.org) and RFC 4193: Unique Local IPv6 Unicast Addresses (rfc-editor.org) which provides details on the routability of Unique-Local addresses. The Unique-Local prefix is drawn from the IPv6 Global Unicast Address range, but is specified as not globally routed.
If I have understood the position correctly it would seem that for users like me who have dynamic IPv6 from their ISP
1. ULAs (fc00::/7) will not work for internet IPv6
2. Link Local addresses (fe80::/10) cannot work with WireGuard
2. If you are choosing a random address, check the IANA preassigned and reserved addresses and make sure your range does not straddle any of these.
3. A better solution may be to find out your ISP's reserved range and then choose a random address within it. You 'might' have a clash with another user for the same ISP, but the odds are extremely small. For SKY (my ISP approximately I in 1.2676506002282*e^30)
And on an RT-AX88U, adding entware iptables (to allow SNAT) does not (appear) to cause any issues.
@ZebMcKayhan let me know if there any other tests that you think may be helpful.
What would be the format to test MASQUARADE (I have lost track of which of the various commands I should be using)? Otherwise I agree you would need to have a script for up/down which checks the current WAN IP and updates it.Wow, great testing! Well, one last thing, maybee youve already tested this, but do you HAVE to use SNAT for it to work? MASQUARADE dont work for you?
The problem with SNAT is that you would need to lookup the br0 ip address each time and dont know if firewall gets restarted whenever your (or any other) isp decides to change ip. Masquarade would be much more convenient if it is working.
ip6tables -t mangle -I POSTROUTING -s fc00:192:168:100::/64 -o wg11 -j SNPT --src-pfx fc00:192:168:100::/64 --dst-pfx fdab:1337:1337:69::/64
ip6tables -t mangle -I PREROUTING -i wg11 -d fdab:1337:1337:69::/64 -j DNPT --src-pfx fdab:1337:1337:69::/64 --dst-pfx fc00:192:168:100::/64
admin@RT-AC86U-D7D8:/tmp/home/root# tcpdump -i wg11 icmp6
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on wg11, link-type RAW (Raw IP), capture size 262144 bytes
22:03:55.120948 IP6 fc00:192:168:100::2 > arn11s11-in-x0e.1e100.net: ICMP6, echo request, seq 1, length 64
^C
1 packet captured
1 packet received by filter
0 packets dropped by kernel
admin@RT-AC86U-D7D8:/tmp/home/root# ip6tables -t mangle -I POSTROUTING -s fc00:192:168:100::/64 -o wg11 -j SNPT --src-pfx fc00:192:168:100::/64 --dst-pfx fdab:1337:1337:69::/64
admin@RT-AC86U-D7D8:/tmp/home/root# tcpdump -i wg11 icmp6
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on wg11, link-type RAW (Raw IP), capture size 262144 bytes
22:04:13.806984 IP6 fdab:1337:1337:69:db77::2 > arn11s11-in-x0e.1e100.net: ICMP6, echo request, seq 1, length 64
^C
1 packet captured
1 packet received by filter
0 packets dropped by kernel
Welcome To SNBForums
SNBForums is a community for anyone who wants to learn about or discuss the latest in wireless routers, network storage and the ins and outs of building and maintaining a small network.
If you'd like to post a question, simply register and have at it!
While you're at it, please check out SmallNetBuilder for product reviews and our famous Router Charts, Ranker and plenty more!