Process [bcmsw_rx] can run at 100% CPU decreasing OpenVPN Client throughput by ~70 Mbps

[CLK]

Hello,

Almost always after a router reboot (but not 100% always), the [bcmsw_rx] process will consistently run at 100% CPU during my hardwired VPN speed tests with resulting download speeds at ~120 Mbps.

The unexpectedly high CPU usage at 100% is what I am concerned about along with the significant reduction in VPN throughput that is apparently a result of that. This issue does not resolve itself with time.

If I restart the OpenVPN client, [bcmsw_rx] will then consistently behave normally with resulting VPN download speeds between 180 to 200 Mbps.

What could be causing these observations?

Tested only on:
RT-AX88U with Asuswrt-Merlin 386.1, 386.2, 386.2_2
PIA VPN with AES-128-GCM
Cox Gigablast
Speedtest.net Mac desktop clients

In both situations above, non-VPN speed tests show [bcmsw_rx] at ~50% with no throughput degradation.
And the log file (attached) does not show any differences in the OpenVPN statements between the router reboot and the OpenVPN client restart sequences.

 

[ColinTaylor]

Someone here reported a similar problem with bcmsw_rx that they blamed on link aggregation. Are you using that?

I see from your log that your VPN client is configured to use IPv6 but the client doesn't support that. So I suggest you remove all the IPv6 commands from your VPN profile.

 

[CLK]

Link aggregation has always been disabled.
And so has IPv6 (from IPv6 > Connection type).

I don't see any IPv6 commands in my VPN profile (in VPN > VPN Client) so don't know how to do what you are suggesting.

Thank you!

 

[ColinTaylor]

I'm only going by the warning messages in your log file. But it looks like the VPN config file that you downloaded from PIA and then imported into the router has IPv6 route command(s). So look at that PIA file and remove any inappropriate commands, reset your router's VPN client and set it up again with the newly edited config.

EDIT: It looks like the IPv6 route is being pushed from the server side, in which case there's probably nothing you can do about that. But it's worth checking the config file anyway.

Where did you get the PIA profiles from? You seem to have two different ones for Los Angles but I couldn't find either of those on the PIA website.

Of course this may be completely unrelated to your problem.

 

[Tucu]

Isn't bcmsw_rx part of the driver for the switch? You might want to use tcpdump+Wireshark to see what packets are going through the interfaces.

 

[CLK]

I'm only going by the warning messages in your log file. But it looks like the VPN config file that you downloaded from PIA and then imported into the router has IPv6 route command(s). So look at that PIA file and remove any inappropriate commands, reset your router's VPN client and set it up again with the newly edited config.

EDIT: It looks like the IPv6 route is being pushed from the server side, in which case there's probably nothing you can do about that. But it's worth checking the config file anyway.

Where did you get the PIA profiles from? You seem to have two different ones for Los Angles but I couldn't find either of those on the PIA website.

Of course this may be completely unrelated to your problem.

I got it from the PIA website; though I did make some manual changes and custom configuration.

As a test, I disabled my existing OpenVPN client and created a new one from scratch using the default config file from PIA (us_california.ovpn).
The results are unfortunately the same. And there's nothing to change regarding IPv6 in the config file.

 

[ColinTaylor]

In both situations above, non-VPN speed tests show [bcmsw_rx] at ~50% with no throughput degradation.

Is this non-VPN speed test done by using policy rules to bypass the VPN or with the VPN completely turned off?

 

[CLK]

Is this non-VPN speed test done by using policy rules to bypass the VPN or with the VPN completely turned off?

Good question.
It was with policy rules; I just tested with VPN completely off and it's the same behavior with [bcmsw_rx] at ~50% during Speedtest download.

 

[jweber93]

I have the same issue with Merlin 386.2_2 (and many versions prior) and my RT-AC86U. If i have QoS enabled (I use FlexQoS), the bcmsw_rx process burns CPU for up to an hour on its own after a QoS restart or router reboot. Eventually, CPU usage goes back to normal. If i disable QoS and reboot the router, the problem doesn't occur.

Since it eventually goes away, I consider it an annoyance more than anything. But I'd love to know what that process is doing while eating CPU.

 

[CLK]

I have the same issue with Merlin 386.2_2 (and many versions prior) and my RT-AC86U. If i have QoS enabled (I use FlexQoS), the bcmsw_rx process burns CPU for up to an hour on its own after a QoS restart or router reboot. Eventually, CPU usage goes back to normal. If i disable QoS and reboot the router, the problem doesn't occur.

Since it eventually goes away, I consider it an annoyance more than anything. But I'd love to know what that process is doing while eating CPU.

I am sorry to hear about your problem. It's similar to mine, but seems to be a clearly different problem.

Mine is 100% CPU usage for bcmsw_rx with, and only with, VPN traffic, thereby causing VPN throughput degradation on Speedtest experiments. This issue does not resolve itself with time, but it does with a VPN client restart.

And that's the heart of my issue: the inconsistent behavior and throughput degradation.

Yours seems to be apparently unexplained bcmsw_rx CPU usage for some time after QoS restart or router reboot. And it resolves itself with time.

Other people more knowledgeable may be able to help you, but it's more than likely a different problem (or not a problem at all).

 

[CLK]

Just an update that I'm working around this problem by restarting the VPN client via user script on router reboot.

Still very curious to chase down a true fix or explanation, but I think there's not much else I can do for now due to my limited knowledge.

 

[Smokey613]

On my Torguard dedicated IP connection, I get better speed and less cpu utilization by using AES-256-GCM setting. I read that the AES-128-GCM does not utilize the AES-NI hardware but that AES-256-GCM will.

 

[CLK]

On my Torguard dedicated IP connection, I get better speed and less cpu utilization by using AES-256-GCM setting. I read that the AES-128-GCM does not utilize the AES-NI hardware but that AES-256-GCM will.

That's interesting and something I will experiment with.

Though it's unlikely I'm getting ~200 Mbps VPN throughput without AES-NI acceleration; this figure is also in line with RMerlin's hardware-accelerated VPN test results in a thread I came across previously.

And there's this he says about the RT-AX88U:
"OpenSSL compiled for this router's CPU will make use of AES operands, speeding up AES performance for anything that uses OpenSSL."

http://www.snbforums.com/threads/checking-for-aes-ni-use-in-openvpn-on-rt-ax88u.62894/post-563383

 

[CLK]

After rebooting my RT-AX88U, hardwired OpenVPN client Speedtest download results are typically ~120 Mbps.
If I then restart the OpenVPN client (either by toggling its 'Service state' setting or restarting it through scMerlin), the download results increase dramatically to ~190 Mbps.

Digging deeper, the culprit for the initial slowdown appears to be the [bcmsw_rx] process.

Almost always after a router reboot, the [bcmsw_rx] process will consistently run at 100% CPU during (and only during) my hardwired VPN speed tests, and seems to be the limiting factor.

After I restart the OpenVPN client, [bcmsw_rx] will consistently behave what I assume is normally with resulting VPN download speeds consistently between 180 to 200 Mbps.

I do not have the knowledge to troubleshoot further why [bcmsw_rx] exhibits this behavior.
I am hoping someone more knowledgeable can assist.

It would also be really good to see if your OpenVPN download Speedtest runs are similarly affected to see how widespread this issue is.
If you are affected, please post your router model, firmware, VPN provider, AES type, ISP service plan, and your Speedtest results.


OTHER INFORMATION

My log file does not show any difference in the OpenVPN statements between the router reboot and the OpenVPN client restart sequences.
And in both cases, non-VPN hardwired Speedtest runs show [bcmsw_rx] at ~50% with no throughput degradation.

Note that if you are using htop to look at processes, you have to hit F2 (for Setup) and under 'Display options' turn off 'Hide kernel threads' in order to see [bcmsw_rx]

Other tests I have done that don't change the observations:
- Disabled my existing OpenVPN client and created a new one from scratch using the default config file from PIA (us_california.ovpn)
- Testing using OpenVPN Client 1 and 2
- Uninstalled NextDNS (my only non-firmware, non-Merlin service) and reran Speedtests
- Uninstalled scMerlin (my only Merlin addon) and reran Speedtests
- Did non-VPN Speedtest runs bypassing VPN using both policy rules and turning VPN completely off
- Used Speedtest servers other than Giggle Fiber (which is what I use normally for VPN testing and the fastest I have found for PIA VPN servers in California)

Tested only on:
RT-AX88U with Asuswrt-Merlin 386.1, 386.2, 386.2_2
PIA VPN with AES-128-GCM
Cox Gigablast (940 Mbps down, 35 Mbps up)
Speedtest.net Mac desktop clients
QoS is OFF, AiProtection is ON

I am currently working around this problem by restarting the VPN client via user script on router reboot.

(I had originally posted in the VPN forum but did not get many hits so posting more broadly; excuse my cross-posting)

 

[MvW]

I am currently working around this problem by restarting the VPN client via user script on router reboot.

Your VPN client is automatically restarted on router reboot.

You might consider installing spdMerlin by @Jack Yaz and see how you're VPN performs throughout the day. I just tried to reproduce it, but I can't. I don't even have the same proces running, so it might be specific to your model. However, I did find an old issue on Github:

CPU between 80% - 95% of usage on RT-AC86U · Issue #248 · RMerl/asuswrt-merlin.ng

After upgrading from v384.7 to 384.8_2 the two cores of the CPU are always on high usage. I downgraded to v384.8 and the issue still happens. So I downgraded again (to v384.7_2) and the issue does ...

github.com

which was closed after 6 months, as there were no further reports.

 

[CLK]

Your VPN client is automatically restarted on router reboot.

You might consider installing spdMerlin by @Jack Yaz and see how you're VPN performs throughout the day. I just tried to reproduce it, but I can't. I don't even have the same proces running, so it might be specific to your model. However, I did find an old issue on Github:

CPU between 80% - 95% of usage on RT-AC86U · Issue #248 · RMerl/asuswrt-merlin.ng

After upgrading from v384.7 to 384.8_2 the two cores of the CPU are always on high usage. I downgraded to v384.8 and the issue still happens. So I downgraded again (to v384.7_2) and the issue does ...

github.com

which was closed after 6 months, as there were no further reports.

That's exactly one outcome of the issue: I have to restart my VPN client (either manually or via script) after router reboot to workaround the issue.

As for spdMerlin, I've found that running Speedtest on the router does not max out my ISP service plan. I've read it's got to do with the fact the client is Python based.
Running it manually on my desktop machines, which I do constantly, allows me to see what the maximum throughput can be.

As for the old issue, it does not seem related to VPN client usage:
"I detected this issue without doing anything on the router, I mean, I just logged in and checked the CPU usage."

Thank you for trying to reproduce the issue!
Note that if you are using htop to look at processes, you have to hit F2 (for Setup) and under 'Display options' turn off 'Hide kernel threads' in order to see [bcmsw_rx]