Wireguard benchmark test on RT-BE96U

[sfx2000]

Here's another test done from the router itself. I had to double check with "wg show wgc2" during the test to confirm that this traffic was really going through the VPN and not directly to my ISP. The traffic count was indeed going up throughout the test.

Numbers look better - what happens if flow cache is disabled?

On a 1GBbit connection, I would think that the CPU could handle this without HW offload...

 

[sfx2000]

Isn't it as expected? This router has quad 2.6GHz B53 CPU. B for Broadcom version of this ARMv8 core. I've got around 470Mbps on my quad 1.5GHz A53 CPU gateway (A in Qualcomm version). I believe with further optimizations this faster CPU router can go higher.

As a side note - Broadcom's B53 core is very different than the licensed ARM Cortex-A53.

B stands for Brahma - and the first iteration was B15, a "large" out-of-order core, similar to Qualcomm's Krait - both of which benefited from the ARM Architecture Licenses.

Broadcom B53 was B15 with ARMV8-A...

It's a pretty stout core, performance wise it's between a Cortex-A57 and A72 running 64-bit ARM instructions.

The SoC running on RT-BE96u outclasses the Mediatek Filogic 830 by a fair degree - Filogic being a quad A53@2.0GHz...

Not everything on a Router SoC is just core and/or offload engines - that being said - the BE96U should be close to wired speed on wireguard if running as a client node to an upstream server...

 

[sfx2000]

Software NAT acceleration is perhaps going away.

No - linux netfilter fast path is here to stay - and that's ok...

Thing is - if one wants to do some magic with QoS, then the blackboxes of NSS/PPE/FlowCache (QC/MTK/BRCM) have some benefit, but it's not transparent on how this would be done.

 

[RMerlin]

Numbers look better - what happens if flow cache is disabled?

My WAN itself caps at around 450 Mbps without FC.

 

[sfx2000]

My WAN itself caps at around 450 Mbps without FC.

Something's not right there...

 

[Tech9]

Thing is - if one wants to do some magic with QoS, then the blackboxes of NSS

Perhaps form of Qualcomm NSS on my gateway. It doesn't use NAT acceleration and still does Gigabit speeds with queue management and IDS/IPS. Suricata is for sure multicore, but the routing after uses what you call... some magic. Good to see things are moving forward with ARM. I have nothing to complain about 6W device. The previous x86 appliance was using 3x more power when doing the same thing.

 

[RMerlin]

Something's not right there...

No, that's perfectly in line with every previous Broadcom routers I've had. From 150 Mbps on the RT-N16, to around 200-220 Mbps on the RT-N66U, 300-350 Mbps on the RT-AC88U, 350-400 Mbps on the GT-AXE16000 - it's fairly linear, and in line with everyone else's report.

 

[sfx2000]

No, that's perfectly in line with every previous Broadcom routers I've had. From 150 Mbps on the RT-N16, to around 200-220 Mbps on the RT-N66U, 300-350 Mbps on the RT-AC88U, 350-400 Mbps on the GT-AXE16000 - it's fairly linear, and in line with everyone else's report.

I get it, but at the same time, it just does not seem right in my honest opinion...

When I have time, I'll have to unbox my old RT-AC68U and put it back on the bench to test basic NAT performance with and without CTF - old platform all told, but still...

 

[Tech9]

RT-AC68U

This one with original BCM4708A0 (800MHz) CPU can do about 250Mbps, the fastest variant with BCM4709C0 (1.4GHz) can reach 350Mbps.

 

[ZebMcKayhan]

I have a 1 Gbps symetric fiber connection

Does anyone know why the speed is so asymmetrical when over Wireguard? Upload is always 20-50% slower than download.

Is it due to flowcache bypass somehow not used for incooming packets?

 

[RMerlin]

Does anyone know why the speed is so asymmetrical when over Wireguard? Upload is always 20-50% slower than download.

Is it due to flowcache bypass somehow not used for incooming packets?

Could be the CPU having to do more heavy work encrypting than decrypting, becoming the bottleneck. Could also be the VPN provider being slower at receiving than sending. I don't know.

 

[Tech9]

Is it due to flowcache bypass somehow not used for incooming packets?

Based on limited information and test results online Broadcom Flow Cache bypass indeed works one way only and on specific CPUs only, most likely limited to ARMv8 cores. Folks with BCM6750/6756 ARMv7 based devices will be limited both ways to whatever the CPU can process. I have no way to test this on any of the available hardware though due to my asymmetric cable ISP.

 

[sfx2000]

have no way to test this on any of the available hardware though due to my asymmetric cable ISP.

Yeah, but it's pretty easy to set up a test bench for performance eval... including NAT

As I've mentioned earlier, the numbers do seen low...

Simply put - if a $130USD router can do near wirespeed on Wireguard without NAT acceleration on OpenWRT, then something is wrong with the core AsusWRT drops...

I don't think this is a Broadcom hardware issue, or even on their delivered SDK's...

 

[RMerlin]

Simply put - if a $130USD router can do near wirespeed on Wireguard without NAT acceleration on OpenWRT, then something is wrong with the core AsusWRT drops...

I don't think this is a Broadcom hardware issue, or even on their delivered SDK's...

That $130 router isn't using a 12 years old CPU.

Linksys performance of their Broadcom devices (like the WRT54G) was on par with Asus'. And Tomato-based devices also have similar NAT performance, regardless of whether they run on Linksys, Asus or Netgear devices. This isn't anything to do with Asuswrt, this is a very well known metric.

 

[Darkje]

I get my full speed on my x86u pro 1g fiber. Should this be wrong...

 

[RMerlin]

I get my full speed on my x86u pro 1g fiber. Should this be wrong...

No, your router is using NAT acceleration, which can achieve multiple Gigabits. We were discussing performance when that NAT acceleration was disabled.

 

[ZebMcKayhan]

Based on limited information and test results online Broadcom Flow Cache bypass indeed works one way only

Yeah, that was my previous assumption as well, but same thing on @RMerlin tests on the router itself when nat is not used as packets are generated locally with correct source address. But possibly combined loading with Wireguard encryption with test packet generation causes this in this specific use case.

 

[Tech9]

The speed is pretty good for a home AIO device tuned for power efficiency. I believe everyone visiting SNB Forums is aware that RPi-like hardware inside home routers has limitations. What Asus can do is publish some numbers users may be interested in. Commercial VPN business is getting popular lately and many other manufacturers have OpenVPN/IPSec/WireGuard up to speeds published so it's more clear what to expect. What Asus does is strictly marketing numbers. No one is getting 30Gbps wired/wireless "performance" from any BE-class product, guaranteed. They want customers to take them seriously and treat the same customers like gamer kids.

 

[sfx2000]

That $130 router isn't using a 12 years old CPU.

No it is not - but it's a Quad Core A53, and that core has been around for some time...

ARM’s Cortex A53: Tiny But Important

Tech enthusiasts probably know ARM as a company that develops reasonably performant CPU architectures with a focus on power efficiency.

chipsandcheese.com

Thing is - over in WiFi6 land, most SoC's have landed on that core...

Comms, compared to applications, is rather predictable - so up to 1Gbe, without CTF/Runner/NSS/PPE, software NAT has always been about clock speeds...

So it's reasonable that a 2GHz A53 can handle a 1Gbit flow in software...

QCA's consumer solutions are similar for WiFi6 based devices - both Cypress and Hawkeye can handle things on a 1Gbe connection at wirespeed - Dakota (IPQ40xx), it can get close...

Going back to that $130 router - it's a mediatek device, and yes, it has flow acceleration, but the PPE's there don't really make things that much faster, rather they just offload traffic once characterized...

 

[sfx2000]

The speed is pretty good for a home AIO device tuned for power efficiency. I believe everyone visiting SNB Forums is aware that RPi-like hardware inside home routers has limitations. What Asus can do is publish some numbers users may be interested in. Commercial VPN business is getting popular lately and many other manufacturers have OpenVPN/IPSec/WireGuard up to speeds published so it's more clear what to expect.

Well there are folks that want to tinker/exploit the HW performance, perhaps I'm one - and those tend to gravitate towards solutions that allow that.

I'm not sure that is representative of folks looking in to SNB for solutions...

There is an echo-chamber around the Asus HW, third party firmware support around Eric's efforts with the RMerlin builds across Broadcom hardware... Folks are going to believe absolutely in what Eric says, and that's ok...

kool-aid is damn tasty...