Huge bandwith limit with skynet and others

[dominik]

Hello,
Recently I moved from AC86U to AX86U and upgraded my ISP to 1G Internet connection. One of first things I've done was installing latest merlin, adding amtm and spdMerlin. Speedtest result was ~920Mbit download and ~600Mbit up.
I left spdMerlin for few days and tweaked some settings. Some time later i found out that download speed dropped to ~720Mbit. Also I noticed that there is some unblocked traffic - time for firewall - I added skynet and that caused yet another download limit - now it's 420/500. I left that for few days and still speedtests are low. Here is speedtest graph from spdMerlin:

test are done every 6h, result on working days at about 10-14 are bit lower and probably caused by isp limit (same thing on weekend or free day has higher result like on 29 may - sunday).
Skynet was installed on 15 Jun, on 22 jun I used option to temporary turn it off and that is spike at 700Mbit. On that day I turned off logging of blocked traffic and it improve speeds by about 200Mbit +

I tried to find something here on forums about such problems and it's claimed that skynet should not decrease speed altou some people experienced that, the only thing to check was usb speed, I added usb3-nvme drive to my older usb3-ssd but that don't changed anything. Then tried several things like formatting new drive to few FS with and without journaling, different swap sizes (on this router with it's ram swap is so far not used, but required for some extensions) .

Today I upgraded to newest melin 386.7 - no change

I think speed decrease may be caused by something else than skynet itself, I tried to investigate issue with htop, no luck. Any ideas what else to check? For sure when skynet is off 700Mbit is back.
I plan to connect directly to PPP via my computer and check if first speed 920Mbit can be still achieved. For now with skynet and no logging I have about half of first speed.

 

[ColinTaylor]

Test from a client connected to the router by Ethernet. Do you see the same speed drop?

 

[dominik]

Test from a client connected to the router by Ethernet. Do you see the same speed drop?

Yes, about same thing - now (on isp peak hours) with skynet 450/345Mbit vs 580/440Mbit without.
I tried that several time from few devices and also done some iperf3 test to make sure that ethernet get full 1GBit on network (it's about 940Mbit, sound ok)

 

[ColinTaylor]

I'd guess that you have AiProtection enabled, or some other TrendMicro component.

Monitor the router's CPU usage while performing your speed test from wired LAN client.

 

[dominik]

I'd guess that you have AiProtection enabled, or some other TrendMicro component.

Monitor the router's CPU usage while performing your speed test from wired LAN client.

I did not agree to anything trendmicro-related so all their features should be disabled, double checked that now - aiProtection is off and there is only ddns/asus agreement on privacy tab. What "other TrendMicro component" You mean? How can I check status of TrendMicro services (even if I did not agree or turned on them)?

Of course I tried to find that bottleneck with htop and atop, here are two speedtests performed on computer connected via ethernet - first time it used four cores, secondly two. On atop bcmsw_rx has highest cpu usage but still around 90%, I made one more test with skynet off and from router itself - can't see any difference in cpu usage except ookla proces and of course +150Mbit

 

[ColinTaylor]

If you're seeing >1% load on bcmsw_rx when testing from a LAN client* that means that hardware acceleration is at least partially disabled. As bcmsw_rx is singled threaded this creates a CPU bottleneck and has a noticeable effect on high speed internet connections like you have. This has been commented on before in these forums.

I can't guess what is creating the problem in your case. It's usually things like AiProtection, QoS, etc. I don't know whether any third party scripts deliberately disable hardware acceleration.

* Ignore speed tests done on the router as they are unreliable because of the CPU issue.

 

[SAL9K]

How does one check the status of hw acceleration on ax86u, to see if it's either disabled or not being used?

 

[ColinTaylor]

How does one check the status of hw acceleration on ax86u, to see if it's either disabled or not being used?

Try: fc status. For example,

# fc status
        Flow Timer Interval = 10000 millisecs
        Pkt-HW Activate Deferral rate = 1
        Pkt-HW Idle Deactivate = 0
        Pkt-SW Activate Deferral count = 0
        Flow Low Pkt Rate = 10
        Acceleration Mode: <L2 & L3>
        MCast Learning <Disabled>
        MCast Acceleration IPv4<Enabled> IPv6<Enabled>
        IPv6 Learning <Enabled>
        GRE Learning <Enabled>
        4o6 Fragmentation <Enabled>
        TCP Ack Prioritization <Enabled>
        HW Acceleration <Enabled>
        Notify Processing Mode <Hybrid>
        OVS Flow Learning <Disabled>
        Flow Learning Enabled : Max<16383>, Active<182>, Cummulative [ 10950 - 10768 ]

 

[SAL9K]

Ah, excellent, mine shows the same. I wouldn't expect a feature like AiProtection to disable acceleration, but perhaps in inspecting all inbound/outbound it's effectively doing so by crippling throughput.

Am I correct in saying that fc status tells you if acceleration is enabled, but the only way to see if it's not being used (like when AiProtection is enabled) is to monitor CPU load?

 

[ColinTaylor]

AiProtection has to do packet inspection so there's a CPU overhead there. How that works in conjunction with hardware acceleration I don't know. You'll have to experiment for yourself as I don't use any of those optional features.

 

[dominik]

The only thing that is different on my fc status output is last line - flow learning disabled, same max, active<0>, cumulative [ 295 -295 ].

Both qos and aiprotection should be disabled. While I expect that those features can have some impact on bandwidth it was claimed that skynet should not limit it. Interestingly I gained about 100Mbit more with less verbose logs.

What else should I check? I could not find option to move atmt files to nvme-usb drive. (I still think that this may be somehow related to that).

Back to bcmsw_rx - any way to debug this process? All I got is outpu from atop, but its not refreshed frequently so bcm.. process have something about 90% and soon after test is finished. I don't see big difference in result when its on router or ethernet connected pc.
Do anybody reached ~920Mbit on such router with skynet?

 

[ColinTaylor]

The only thing that is different on my fc status output is last line - flow learning disabled,

So it looks like something has disabled Flow Cache. Go to Tools > System Information > HW acceleration > Network > HW acceleration and check the status of Runner and Flow Cache.

Also check the output of this:

# nvram show | grep ^fc
fc_disable=0
fc_pt_udp_war=0
fc_pt_war=0

 

[dominik]

Runner is enabled, flow disabled, fc shows fc_disable=1

So, something disables this, I assume that I can try to enable it and repeat test, then try to find what and why it's disabled.
Is there any log regarding this vars?

 

[ColinTaylor]

Are you using a VPN? I believe Wireguard disables fc.

 

[dominik]

Are you using a VPN? I believe Wireguard disables fc.

Yes, one openvpn server but sure I also was playing with wg. I'll check that for sure, thanks for this clue.

 

[dominik]

Are you using a VPN? I believe Wireguard disables fc.

I just completely removed wireguard and flow accelleration is enabled again.
quick test from spdMerlin - bandwidth jumped imidiately to 940Mbit/560Mbit
Thank Colin for You suggestions - wg caused all those issues, now I'll try to find out more on that - is that known issue and are there any chances to reconfigure it. OpenVPN works ok.

Also I made some tests and:
+ as You said - when doing test on device bcmsw_rx is not using cpu as earlier, but result is same as from eth device
+ speed test from eth device gives same speed and no cpu load on router
+ below two graphs from test

 

[Martineau]

I just completely removed wireguard and flow accelleration is enabled again.
quick test from spdMerlin - bandwidth jumped imidiately to 940Mbit/560Mbit
Thank Colin for You suggestions - wg caused all those issues, now I'll try to find out more on that - is that known issue and are there any chances to reconfigure it. OpenVPN works

Help replacing OpenVPN with Wireguard on Asus RT-AX88U

Hello I just realized that with OpenVPN I was getting limited speeds. I only had 160 mbps service until today I upgraded to 500 mbps. If I use my VPN I only get 200 mbps and if I disconnect I am getting 500 mbps. I am using Torguard as my VPN provided and wanted to install wireguard but wasn't...

www.snbforums.com