ASUS GT-AXE16000 CPU usage with VPN on lan machine

[Jerky_san]

Ok so I've been trying to narrow down what is going on here with slow VPN usage on local lan machines.

2.5gb fiber connection with full 10gb network throughout

First step.. speed test on the router itself..
Results: CPU usage low.. think like 5-8% utilization(totally fine) - Full line speed

Second step.. speed test on an AP that is connected via 10gb -> 10gb switch -> 10gb port on router
Results: Exact same as first step

3rd step.. speed test on lan machine 10gb -> 10gb switch -> 10gb port on router
Results: Yet again exact same

4th step.. enable wireguard VPN to high performance server(10gb backbone on VPN server) -> 10gb -> 10gb switch -> 10gb port on router
Results: Extremely high CPU usage on single core originating from bcmsw_rx.. basically can only achieve about 350 megabits a second max.

Here is the weird part.. it doesn't happen all the time. It seems to be what server I connect to(all from the same service and weirdly same location but different IP is assigned). The cipher is the same which is ChaCha20. So I can't figure out why it's doing this as it doesn't appear to be cipher related. I'm using the same client/machine for each of these tests. It seemingly doesn't route through the NAT Accelerator sometimes. Is there a way to figure out potentially why this occurs? Or if this is even a router problem? Also when it DOESN'T do this.. basically close to full line speed.. like 2gb which frankly is enough. Similar CPU utilization as other tests.

 

[Tech9]

basically can only achieve about 350 megabits a second max

Router CPU limitation. May go a bit higher, but whatever goes through WireGuard is incompatible with NAT acceleration. For the rest flow cache bypass was implemented in firmware some time ago. Your router may have fast ports, but it's similar to RPi hardware and relies heavily on NAT acceleration.

 

[Jerky_san]

Router CPU limitation. May go a bit higher, but whatever goes through WireGuard is incompatible with NAT acceleration. For the rest flow cache bypass was implemented in firmware some time ago. Your router may have fast ports, but it's similar to RPi hardware and relies heavily on NAT acceleration.

But that is the thing.. it doesn't always do it. It seemingly is only certain servers the client is round robinin through. Like I found one during my testing nearly fully line speed. Next one Router CPU spikes hard but it's not cipher related and seemingly not port related. So wondering what is the difference and why the NAT acceleration isn't working on certain ones but is on others. Also, all wireguard and same settings. I don't change things between tests besides whatever server the client randomly connects to in that location. I've started recording all the ips and which work and don't work to try to narrow it down.

Edit 2:

You'd assume as well it's server related but how can the server affect my router's CPU.. Maybe it's altering the MTU or something and breaking the NAT acceleration that way? That should be client controlled though..?

 

[Jerky_san]

Plot thickens.. I wired sharked the connection.. Even though I thought I'm not changing the port the port is indeed changing.. It would seem if it connects on port 1337 no problems but 1443 it spikes the CPU. Very weird..

 

[ZebMcKayhan]

Plot thickens.. I wired sharked the connection.. Even though I thought I'm not changing the port the port is indeed changing.. It would seem if it connects on port 1337 no problems but 1443 it spikes the CPU. Very weird..

The flowcache bypass uses 2 files to record local IP and port that should be bypassed. You can look at them by:

cat /proc/blog/skip_wireguard_port
cat /proc/blog/skip_wireguard_network

You can add/remove entries here to experiment:

echo "add 172.16.1.1/32" >> /proc/blog/skip_wireguard_network
echo "del 172.16.1.1/32" >> /proc/blog/skip_wireguard_network

Source: https://www.snbforums.com/threads/session-manager-4th-thread.81187/post-830466
https://www.snbforums.com/threads/session-manager-4th-thread.81187/post-815092

If you change Wireguard port without fw knowing you may skip the bypass. The reason for this bypass is incompability and could show as extremely poor speeds, syslog filling up with error message, router instability et.c.

 

[Jerky_san]

The flowcache bypass uses 2 files to record local IP and port that should be bypassed. You can look at them by:

cat /proc/blog/skip_wireguard_port
cat /proc/blog/skip_wireguard_network

You can add/remove entries here to experiment:

echo "add 172.16.1.1/32" >> /proc/blog/skip_wireguard_network
echo "del 172.16.1.1/32" >> /proc/blog/skip_wireguard_network

Source: https://www.snbforums.com/threads/session-manager-4th-thread.81187/post-830466
https://www.snbforums.com/threads/session-manager-4th-thread.81187/post-815092

If you change Wireguard port without fw knowing you may skip the bypass. The reason for this bypass is incompability and could show as extremely poor speeds, syslog filling up with error message, router instability et.c.

Oh snap thank you! I'll try this.

 

[Jerky_san]

The flowcache bypass uses 2 files to record local IP and port that should be bypassed. You can look at them by:

cat /proc/blog/skip_wireguard_port
cat /proc/blog/skip_wireguard_network

You can add/remove entries here to experiment:

echo "add 172.16.1.1/32" >> /proc/blog/skip_wireguard_network
echo "del 172.16.1.1/32" >> /proc/blog/skip_wireguard_network

Source: https://www.snbforums.com/threads/session-manager-4th-thread.81187/post-830466
https://www.snbforums.com/threads/session-manager-4th-thread.81187/post-815092

If you change Wireguard port without fw knowing you may skip the bypass. The reason for this bypass is incompability and could show as extremely poor speeds, syslog filling up with error message, router instability et.c.

Huh interestingly port 1443 was already in this list.. hmm

 

[ZebMcKayhan]

Huh interestingly port 1443 was already in this list.. hmm

If it's in the list it means this port will be bypassed and process by sw, which is why you see core1 spiking (sw nat only uses core 1, Wireguard encryption uses all cores).

What if you remove it?

 

[Jerky_san]

If it's in the list it means this port will be bypassed and process by sw, which is why you see core1 spiking (sw nat only uses core 1, Wireguard encryption uses all cores).

What if you remove it?

I'll try in a minute but port 1337 is also in the list and it's seemingly going through NAT acceleration just fine so that is why it is confusing me.

 

[ZebMcKayhan]

I'll try in a minute but port 1337 is also in the list and it's seemingly going through NAT acceleration just fine so that is why it is confusing me.

I don't pretend to understand everything about FlowCache or the bypass. I don't know.

But 400-600Mb/s is the limit for data transfer with sw nat. Even if nat is not used I would expect reaching maybe 1000Mb/s due to encryption processing. If you get well above 2Gb/s I have to wonder if your test is over Wireguard at all?

 

[Jerky_san]

I don't pretend to understand everything about FlowCache or the bypass. I don't know.

But 400-600Mb/s is the limit for data transfer with sw nat. Even if nat is not used I would expect reaching maybe 1000Mb/s due to encryption processing. If you get well above 2Gb/s I have to wonder if your test is over Wireguard at all?

The router isn't doing the work of the transfer wireguard encryption on the client side.. only the client machine on the lan which is a 16 core 5950x with a 10gb ethernet card. It doesn't always hit 2gb but it definitely is running it through wireguard as long as it doesn't hit this oddity which seems to not happen when it's running over port 1337. The router itself can easily pull it via it's speed test function so I assume as long as the acceleration chip is involved it can go much higher.

 

[ZebMcKayhan]

The router isn't doing the work of the transfer wireguard encryption on the client side.. only the client machine on the lan which is a 16 core 5950x with a 10gb ethernet card.

Wait, your router is not the wireguard client? It's just passing the data?

Then what are these ports doing in the blog bypass? Do you have other wireguard vpn on router using these ports?

 

[Jerky_san]

Wait, your router is not the wireguard client? It's just passing the data?

Then what are these ports doing in the blog bypass? Do you have other wireguard vpn on router using these ports?

The VPN service I use offers wireguard on ports 1337 and 1443. If you use their client it alters the port depending on which server it's connecting to at that location. I can use a "configuration" generator to generate configs for a location and then which port. After I notice in wireshark it altering back and for I used to generator to make configs for both 1337 and 1443 and it seems if it goes out 1443 it spikes the router's CPU to pass the traffic out the wan(not using NAT acceleration). If it goes out 1337 it seems to use the nat accelerator. I have no idea why it does this but I'm just changing my dockers to use that port and bam.. full speed vpns and such.

 

[Jerky_san]

I had a wg config that was working but for some reason after I rebooted the server that was using it. It went back to using a whole core on my router. It's so weird.

 

[ZebMcKayhan]

I had a wg config that was working but for some reason after I rebooted the server that was using it. It went back to using a whole core on my router. It's so weird.

If I understand you right, you are connecting to the same serverort on the router (as these ports are in the bypass list) for some reason. Then trying to connect to same serverort from a lan client.
I would think that your issue arise from the bypass on the router but also affects your lan client due to the same port usage. But it's a guess. But if I'm right, there nothing anyone can do about it.

If you temporarily turn off wireguard on the router does that change anything?
Do you have to run Wireguard on the router?
Do you have to use the same ports?

 

[Jerky_san]

If I understand you right, you are connecting to the same serverort on the router (as these ports are in the bypass list) for some reason. Then trying to connect to same serverort from a lan client.
I would think that your issue arise from the bypass on the router but also affects your lan client due to the same port usage. But it's a guess. But if I'm right, there nothing anyone can do about it.

If you temporarily turn off wireguard on the router does that change anything?
Do you have to run Wireguard on the router?
Do you have to use the same ports?

Turning off the torguard VPN server on the router itself does not change it. It appears using the two different ports doesn't change it either.

I guess also I didn't explain it well in the OP but

LAN CLIENT -> Wireguard server on the internet is basically what I'm attempting to do. Sometimes the router seems like it uses the NAT acceleration and other times it seems like it doesn't. I'm unsure what causes it to not and what causes to to do that. I might just try fully resetting my router and setting it up from scratch though that would be a super pain in the butt.

Sometimes though it works if I cycle through configurations on the lan client and I can get full wirespeed with no CPU usage showing on the router. Sometimes though it(seems like most of the time these days) it's 99%-100% utilization on the router.

 

[ZebMcKayhan]

Turning off the torguard VPN server on the router itself does not change it. It appears using the two different ports doesn't change it either.

Alright, but you stated earlier that your 2 port were present in the blog bypass files. They must have been put there by something you configured on your router.

LAN CLIENT -> Wireguard server on the internet is basically what I'm attempting to do. Sometimes the router seems like it uses the NAT acceleration and other times it seems like it doesn't. I'm unsure what causes it to not and what causes to to do that. I might just try fully resetting my router and setting it up from scratch though that would be a super pain in the butt.

I get the setup from your last posts.

There are some things on your router that is incompatible with nat acceleration. Wireguard is one thing but with the blog bypass files it *should* only affect clients using the router tunnel, unless destination based rules are setup in vpndirector, then it may apply for your whole network.
QoS is another router feature incompatible with nat acceleration and may completally turn it off.
There may be more but I don't have a complete list.

Firstly, check the blog files on the router so that your PC ip is not included. Also check the port you are using is not in the list. If any of these are, there is probably your issue and you should find out why these are in the lists.

If they are not in the list, you can check global status by

fc status

And compare output when high resp low speed

 

[Jerky_san]

Alright, but you stated earlier that your 2 port were present in the blog bypass files. They must have been put there by something you configured on your router.



I get the setup from your last posts.

There are some things on your router that is incompatible with nat acceleration. Wireguard is one thing but with the blog bypass files it *should* only affect clients using the router tunnel, unless destination based rules are setup in vpndirector, then it may apply for your whole network.
QoS is another router feature incompatible with nat acceleration and may completally turn it off.
There may be more but I don't have a complete list.

Firstly, check the blog files on the router so that your PC ip is not included. Also check the port you are using is not in the list. If any of these are, there is probably your issue and you should find out why these are in the lists.

If they are not in the list, you can check global status by

fc status

And compare output when high resp low speed

Alright, but you stated earlier that your 2 port were present in the blog bypass files. They must have been put there by something you configured on your router.

Unless it was me turning on wireguard but the wireguard port configured on the router is not the standard port. I changed it to an openvpn port because the standard port was being blocked on a lot of places I was visiting.

There are some things on your router that is incompatible with nat acceleration. Wireguard is one thing but with the blog bypass files it *should* only affect clients using the router tunnel, unless destination based rules are setup in vpndirector, then it may apply for your whole network.
QoS is another router feature incompatible with nat acceleration and may completally turn it off.
There may be more but I don't have a complete list.

I do use a domain based vpn routing but the thing that is strange is it works "sometimes". Once the connection is established and verified working it will work constantly till I restart the connection. You might think "the server your connecting to is different each time. I actually did a port forward on the VPN service so it would assign me an IP to use and then made sure it worked(which it did full speed for weeks). I restarted the server to do an upgrade a few days ago and it broke even though the configuration is literally the exact same so it's like sometimes the NAT acceleration is detecting it and other times it's not. I'm not using QoS or anything else and it does indeed work SOMETIMES but not all the time. My original idea was it was port based and one port wasn't working but the other was but now that got thrown out the window as well.

Firstly, check the blog files on the router so that your PC ip is not included. Also check the port you are using is not in the list. If any of these are, there is probably your issue and you should find out why these are in the lists.

The IP isn't but the ports are but makes no sense why it sometimes works. That is why I'm getting to the point of just saying factory reset and setup from scratch to see if it resolves it.

Lastly

fc status

I'm unsure what I'm supposed to be looking for in this command but it does show nat is working and such.

@Router:/tmp/home/root# fc status
        Flow Timer Interval = 10000 millisecs
        Pkt-HW Activate Deferral rate = 1
        Pkt-HW Idle Deactivate = 0
        Pkt-SW Activate Deferral count = 0
        Flow Low Pkt Rate = 10
        Acceleration Mode: <L2 & L3>
        MCast Acceleration IPv4<Enabled> IPv6<Enabled>
        IPv6 Learning <Enabled>
        L2TP Learning <Enabled>
        GRE Learning <Enabled>
        4o6 Fragmentation <Enabled>
        TCP Ack Prioritization <Enabled>
        ToS Multi Flow <Enabled>
        Notify Processing Mode <Hybrid>
        OVS Flow Learning <Disabled>
        HW Acceleration <Enabled>
        Flow Ucast Learning Enabled  : Max<16383>, Active<1456>, Cumulative [ 55                                                                                                                                                                                                                                             71290 - 5569834 ]
        Flow Mcast Learning Enabled  : Max<1152>, Active<0>, Cumulative [ 0 - 0                                                                                                                                                                                                                                              ]
@Router:/tmp/home/root#

Btw I do appreciate you attempting to help me. So thanks for responding so far. If I'm misunderstanding something sorry.

 

[ZebMcKayhan]

The IP isn't but the ports are but makes no sense why it sometimes works. That is why I'm getting to the point of just saying factory reset and setup from scratch to see if it resolves it.

Perhaps a factory reset is a good idea. But it may help others to understand what is going on.
Understanding you right is that your PC ip is not included as ip or as cidr range in blog bypass - just making sure (again)
For curiosity, how have you setup vpndirector rules? Only 1 or more single ips over wireguard?

Have you tried to remove these ports from blog bypass files, I.e:

echo "del 1337 either" >> /proc/blog/skip_wireguard_port
echo "del 1443 either" >> /proc/blog/skip_wireguard_port

Do the ports re-appear? Does it change anything?

 

[Jerky_san]

Perhaps a factory reset is a good idea. But it may help others to understand what is going on.
Understanding you right is that your PC ip is not included as ip or as cidr range in blog bypass - just making sure (again)
For curiosity, how have you setup vpndirector rules? Only 1 or more single ips over wireguard?

Have you tried to remove these ports from blog bypass files, I.e:

echo "del 1337 either" >> /proc/blog/skip_wireguard_port
echo "del 1443 either" >> /proc/blog/skip_wireguard_port

Do the ports re-appear? Does it change anything?

echo "del 1337 either" >> /proc/blog/skip_wireguard_port
echo "del 1443 either" >> /proc/blog/skip_wireguard_port

This appears to fixed the high cpu consumption like immediately. I ran these in the middle of the execution of a speed test. It immediately dropped CPU usage to near 0 and the speed test went from around 450 megabits to full line speed. Interesting..