ASUS GT-AXE16000 CPU usage with VPN on lan machine

[Jerky_san]

Ok so I've been trying to narrow down what is going on here with slow VPN usage on local lan machines.

2.5gb fiber connection with full 10gb network throughout

First step.. speed test on the router itself..
Results: CPU usage low.. think like 5-8% utilization(totally fine) - Full line speed

Second step.. speed test on an AP that is connected via 10gb -> 10gb switch -> 10gb port on router
Results: Exact same as first step

3rd step.. speed test on lan machine 10gb -> 10gb switch -> 10gb port on router
Results: Yet again exact same

4th step.. enable wireguard VPN to high performance server(10gb backbone on VPN server) -> 10gb -> 10gb switch -> 10gb port on router
Results: Extremely high CPU usage on single core originating from bcmsw_rx.. basically can only achieve about 350 megabits a second max.

Here is the weird part.. it doesn't happen all the time. It seems to be what server I connect to(all from the same service and weirdly same location but different IP is assigned). The cipher is the same which is ChaCha20. So I can't figure out why it's doing this as it doesn't appear to be cipher related. I'm using the same client/machine for each of these tests. It seemingly doesn't route through the NAT Accelerator sometimes. Is there a way to figure out potentially why this occurs? Or if this is even a router problem? Also when it DOESN'T do this.. basically close to full line speed.. like 2gb which frankly is enough. Similar CPU utilization as other tests.

 

[Tech9]

basically can only achieve about 350 megabits a second max

Router CPU limitation. May go a bit higher, but whatever goes through WireGuard is incompatible with NAT acceleration. For the rest flow cache bypass was implemented in firmware some time ago. Your router may have fast ports, but it's similar to RPi hardware and relies heavily on NAT acceleration.

 

[Jerky_san]

Router CPU limitation. May go a bit higher, but whatever goes through WireGuard is incompatible with NAT acceleration. For the rest flow cache bypass was implemented in firmware some time ago. Your router may have fast ports, but it's similar to RPi hardware and relies heavily on NAT acceleration.

But that is the thing.. it doesn't always do it. It seemingly is only certain servers the client is round robinin through. Like I found one during my testing nearly fully line speed. Next one Router CPU spikes hard but it's not cipher related and seemingly not port related. So wondering what is the difference and why the NAT acceleration isn't working on certain ones but is on others. Also, all wireguard and same settings. I don't change things between tests besides whatever server the client randomly connects to in that location. I've started recording all the ips and which work and don't work to try to narrow it down.

Edit 2:

You'd assume as well it's server related but how can the server affect my router's CPU.. Maybe it's altering the MTU or something and breaking the NAT acceleration that way? That should be client controlled though..?

 

[Jerky_san]

Plot thickens.. I wired sharked the connection.. Even though I thought I'm not changing the port the port is indeed changing.. It would seem if it connects on port 1337 no problems but 1443 it spikes the CPU. Very weird..

 

[ZebMcKayhan]

Plot thickens.. I wired sharked the connection.. Even though I thought I'm not changing the port the port is indeed changing.. It would seem if it connects on port 1337 no problems but 1443 it spikes the CPU. Very weird..

The flowcache bypass uses 2 files to record local IP and port that should be bypassed. You can look at them by:

cat /proc/blog/skip_wireguard_port
cat /proc/blog/skip_wireguard_network

You can add/remove entries here to experiment:

echo "add 172.16.1.1/32" >> /proc/blog/skip_wireguard_network
echo "del 172.16.1.1/32" >> /proc/blog/skip_wireguard_network

Source: https://www.snbforums.com/threads/session-manager-4th-thread.81187/post-830466
https://www.snbforums.com/threads/session-manager-4th-thread.81187/post-815092

If you change Wireguard port without fw knowing you may skip the bypass. The reason for this bypass is incompability and could show as extremely poor speeds, syslog filling up with error message, router instability et.c.

 

[Jerky_san]

The flowcache bypass uses 2 files to record local IP and port that should be bypassed. You can look at them by:

cat /proc/blog/skip_wireguard_port
cat /proc/blog/skip_wireguard_network

You can add/remove entries here to experiment:

echo "add 172.16.1.1/32" >> /proc/blog/skip_wireguard_network
echo "del 172.16.1.1/32" >> /proc/blog/skip_wireguard_network

Source: https://www.snbforums.com/threads/session-manager-4th-thread.81187/post-830466
https://www.snbforums.com/threads/session-manager-4th-thread.81187/post-815092

If you change Wireguard port without fw knowing you may skip the bypass. The reason for this bypass is incompability and could show as extremely poor speeds, syslog filling up with error message, router instability et.c.

Oh snap thank you! I'll try this.

 

[Jerky_san]

The flowcache bypass uses 2 files to record local IP and port that should be bypassed. You can look at them by:

cat /proc/blog/skip_wireguard_port
cat /proc/blog/skip_wireguard_network

You can add/remove entries here to experiment:

echo "add 172.16.1.1/32" >> /proc/blog/skip_wireguard_network
echo "del 172.16.1.1/32" >> /proc/blog/skip_wireguard_network

Source: https://www.snbforums.com/threads/session-manager-4th-thread.81187/post-830466
https://www.snbforums.com/threads/session-manager-4th-thread.81187/post-815092

If you change Wireguard port without fw knowing you may skip the bypass. The reason for this bypass is incompability and could show as extremely poor speeds, syslog filling up with error message, router instability et.c.

Huh interestingly port 1443 was already in this list.. hmm

 

[ZebMcKayhan]

Huh interestingly port 1443 was already in this list.. hmm

If it's in the list it means this port will be bypassed and process by sw, which is why you see core1 spiking (sw nat only uses core 1, Wireguard encryption uses all cores).

What if you remove it?

 

[Jerky_san]

If it's in the list it means this port will be bypassed and process by sw, which is why you see core1 spiking (sw nat only uses core 1, Wireguard encryption uses all cores).

What if you remove it?

I'll try in a minute but port 1337 is also in the list and it's seemingly going through NAT acceleration just fine so that is why it is confusing me.

 

[ZebMcKayhan]

I'll try in a minute but port 1337 is also in the list and it's seemingly going through NAT acceleration just fine so that is why it is confusing me.

I don't pretend to understand everything about FlowCache or the bypass. I don't know.

But 400-600Mb/s is the limit for data transfer with sw nat. Even if nat is not used I would expect reaching maybe 1000Mb/s due to encryption processing. If you get well above 2Gb/s I have to wonder if your test is over Wireguard at all?

 

[Jerky_san]

I don't pretend to understand everything about FlowCache or the bypass. I don't know.

But 400-600Mb/s is the limit for data transfer with sw nat. Even if nat is not used I would expect reaching maybe 1000Mb/s due to encryption processing. If you get well above 2Gb/s I have to wonder if your test is over Wireguard at all?

The router isn't doing the work of the transfer wireguard encryption on the client side.. only the client machine on the lan which is a 16 core 5950x with a 10gb ethernet card. It doesn't always hit 2gb but it definitely is running it through wireguard as long as it doesn't hit this oddity which seems to not happen when it's running over port 1337. The router itself can easily pull it via it's speed test function so I assume as long as the acceleration chip is involved it can go much higher.

 

[ZebMcKayhan]

The router isn't doing the work of the transfer wireguard encryption on the client side.. only the client machine on the lan which is a 16 core 5950x with a 10gb ethernet card.

Wait, your router is not the wireguard client? It's just passing the data?

Then what are these ports doing in the blog bypass? Do you have other wireguard vpn on router using these ports?

 

[Jerky_san]

Wait, your router is not the wireguard client? It's just passing the data?

Then what are these ports doing in the blog bypass? Do you have other wireguard vpn on router using these ports?

The VPN service I use offers wireguard on ports 1337 and 1443. If you use their client it alters the port depending on which server it's connecting to at that location. I can use a "configuration" generator to generate configs for a location and then which port. After I notice in wireshark it altering back and for I used to generator to make configs for both 1337 and 1443 and it seems if it goes out 1443 it spikes the router's CPU to pass the traffic out the wan(not using NAT acceleration). If it goes out 1337 it seems to use the nat accelerator. I have no idea why it does this but I'm just changing my dockers to use that port and bam.. full speed vpns and such.