OpenVPN performance of the RT-AC86U

[RMerlin]

Openvpn 2.5 will support Chacha20-Poly1305 in data channel.

Requires OpenSSL 1.1.x, which isn't backward compatible with 1.0.2, and therefore cannot be upgraded by me. API has changed, and therefore Asus will have to compile their closed source bits against 1.1.x.

OpenVPN could always be compiled to use a different static openssl build, but the bloat just cannot be justified.

 

[Germain]

Hi.. I’m new here! I have r7000 and love your job rmerlin! I want to upgrade my router because my speed is 20/10 on a 60/10 with liquidvpn.. I’m confused with rt-ac86u..rt-ac3200(256mb ram) and rt-3100 (512 ram) ...Which is best for top speed on OpenVPN on router..which will have the more long support with merlin firmware.. thanks!!

 

[RMerlin]

Hi.. I’m new here! I have r7000 and love your job rmerlin! I want to upgrade my router because my speed is 20/10 on a 60/10 with liquidvpn.. I’m confused with rt-ac86u..rt-ac3200(256mb ram) and rt-3100 (512 ram) ...Which is best for top speed on OpenVPN on router..which will have the more long support with merlin firmware.. thanks!!

RAM is completely irrelevant there, what matters is the CPU. The RT-AC3200 and RT-AC3100 all use the same CPU as the Netgear R7000 (BCM4709) at various clock rates between 1 GHz and 1.4 GHz. The RT-AC86U uses a completely different CPU, the BCM4906), which supports hardware-accelerated AES, and a hardware crypto engine (the latter only of use with IPSEC). That AES acceleration is what allows OpenVPN to run at over twice the speed of the previous CPUs.

Hard to predict longevity of support since I'm highly dependent on Asus's own support (due to the closed source components). The RT-AC3200 is certainly the one closest to reaching End of Life status of these three.

 

[Germain]

Thanks!! And for openvpn which overclocked or not will give me the more speed on vpn server.. rt-ac86u? Thanks I appreciate your work!!

 

[RMerlin]

Thanks!! And for openvpn which overclocked or not will give me the more speed on vpn server.. rt-ac86u? Thanks I appreciate your work!!

No other Asus router will come even close to the RT-AC86U/GT-AC5300/RT-AX88U CPU.

 

[ajlimajr]

Hello guys, good morning.

First of all, thank you Merlin for such a good job... you're amazing. I installed NordVPN on my AC86U and everything is great, except my Nvidia Shield that is not working under VPN, so i took it off. All my other devices are working fine, even Netflix, just perfect. Do you guys have any idea? The Nvidia Shield under vpn not even load the netflix's thumbnails, but the netflix works great on my samsung tv under vpn (for example). I'm talking about Netflix, but everything on Nvidia Shield is off under VPN. Maybe the Android TV app could work, i'm not home right now to test it.

Thank you guys, greetings from Brazil.
Antonio

 

[RMerlin]

Android Netflix uses hardcoded DNS servers, which might be blocked by the VPN. Could be other things in Android hardcoding nameservers.

 

[ajlimajr]

Android Netflix uses hardcoded DNS servers, which might be blocked by the VPN. Could be other things in Android hardcoding nameservers.

Thank you Merlin, i'll give a try at the weekend with the native app to Android TV (by NordVPN) and report here.

 

[ludnell]

Thank you Merlin, i'll give a try at the weekeng with the native app to Android TV (by NordVPN) and report here.

I'm also using a Nvidia Shield for Netflix and other streaming services. Please let us know if you sort it out somehow.

 

[ajlimajr]

I'm also using a Nvidia Shield for Netflix and other streaming services. Please let us know if you sort it out somehow.

I'll do my best to try it today, my idea is to see how the native app will perform on Shield. Maybe, i'll let all my devices running under AC86U/VPN, and the Shield by itself.

Edited: I just tested it and worked fine. I installed the Nord VPN native app to Android TV and I was able to watch Netflix (US Netflix - even when connected to brazilian servers), YouTube, etc.

 

[RMerlin]

Can't remember if I've posted these before, but for the sake of completeness, here are test results when using IPSEC, with the router's hardware crypto module in use (test results also show CPU usage during the tests):

Downstream (bcmspu):
P:\Tools>iperf -c 192.168.1.51 -M 1400 -N -t 30
------------------------------------------------------------
Client connecting to 192.168.1.51, TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[296] local 10.10.10.1 port 8334 connected with 192.168.1.51 port 5001
[ ID] Interval       Transfer     Bandwidth
[296]  0.0-30.0 sec  1.08 GBytes    309 Mbits/sec

CPU:  0.6% usr 64.6% sys  0.0% nic  8.2% idle  0.0% io  0.0% irq 26.4% sirq
Load average: 3.48 2.55 1.39 3/150 8377
  PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
  215    2 admin    RW       0  0.0   1 47.6 [pdc_rx]
  206    2 admin    RW       0  0.0   0 41.6 [bcmsw_rx]
  813    1 admin    S     8336  1.8   1  0.6 watchdog
  943    1 admin    S     4924  1.1   0  0.3 networkmap --bootwait



Upstream (bcmspu):
C:\Users\Eric\Documents>iperf -c 10.10.10.1 -M 1400 -N -t 30
------------------------------------------------------------
Client connecting to 10.10.10.1, TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[296] local 192.168.1.51 port 2644 connected with 10.10.10.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[296]  0.0-30.0 sec    886 MBytes    248 Mbits/sec

CPU:  0.3% usr 67.1% sys  0.0% nic 12.2% idle  0.0% io  0.0% irq 20.3% sirq
Load average: 3.46 3.11 2.11 2/150 8645
  PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
  206    2 admin    RW       0  0.0   1 45.6 [bcmsw_rx]
  215    2 admin    RW       0  0.0   0 40.5 [pdc_rx]
  805    1 admin    S     8672  1.9   0  0.2 httpd -i br0

 

[kvic]

Can't remember if I've posted these before, but for the sake of completeness, here are test results when using IPSEC, with the router's hardware crypto module in use (test results also show CPU usage during the tests)

The numbers nothing outstanding honestly..

EdgeRouter-X can do 377Mbit/s from my tests..details HERE. Microtik's hEX can do close to 500Mbit/s. As always Microtik excels at system optimization over Ubiquiti. Both routers use a cheap SoC and much less powerful than the SoC in RT-AC86U.

I'm not promoting either brands. Just provide another two yardsticks when comparing VPN performance..

Interestingly, what the processes "pdc_rx" and "bcmsw_rx" are doing in Asus/Merlin FWs?

 

[RMerlin]

The numbers nothing outstanding honestly..

Compared to other Asus routers, they are quite good. They also illustrate the performance difference between OpenVPN and IPSEC on the same device.

EdgeRouter-X can do 377Mbit/s from my tests..details HERE. Microtik's hEX can do close to 500Mbit/s. As always Microtik excels at system optimization over Ubiquiti. Both routers use a cheap SoC and much less powerful than the SoC in RT-AC86U.

Apple and oranges. Configuring an Asus router takes a few minutes and can be done by anyone. Configuring a Microtik can take hours, and requires expect training to achieve. Only a week ago I saw a customer get billed 2 hours by his (other) consultant to configure a VPN tunnel between two Microtiks... And it's the consultant who actually sold him the devices (and configured the first one at their office last spring), so they're not first-time users. If you want to talk costs, then you need to introduce TCO into the equation, not just compare MSRPs.

Beside, the point of this thread is to provide VPN performance datapoints specific to the RT-AC86U, not to start advertising the RT-AC86U as the end-it-all IPSEC solution for businesses.

Interestingly, what the processes "pdc_rx" and "bcmsw_rx" are doing in Asus/Merlin FWs?

From what I can deduce, one is the switch, the other would be the hardware crypto engine. Here is the same test, done with a purely software implementation (with the hardware engine disabled):

Downstream (software only)

P:\Tools>iperf -c 192.168.1.51 -M 1400 -N -t 30
------------------------------------------------------------
Client connecting to 192.168.1.51, TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[292] local 10.10.10.1 port 1406 connected with 192.168.1.51 port 5001
[ ID] Interval       Transfer     Bandwidth
[292]  0.0-30.0 sec    475 MBytes    133 Mbits/sec

CPU:  0.1% usr 32.8% sys  0.0% nic 58.0% idle  0.0% io  0.0% irq  8.9% sirq
Load average: 3.16 2.99 2.44 3/150 8986
  PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
  206    2 admin    RW       0  0.0   0 40.7 [bcmsw_rx]
  805    1 admin    S     8672  1.9   1  0.2 httpd -i br0
  943    1 admin    R     4924  1.1   1  0.2 networkmap --bootwait


Upstream (software only)

C:\Users\Eric\Documents>iperf -c 10.10.10.1 -M 1400 -N -t 30
------------------------------------------------------------
Client connecting to 10.10.10.1, TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[292] local 192.168.1.51 port 2851 connected with 10.10.10.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[292]  0.0-30.0 sec    381 MBytes    106 Mbits/sec

CPU:  0.2% usr 43.4% sys  0.0% nic 49.0% idle  0.0% io  0.0% irq  7.3% sirq
Load average: 3.12 2.97 2.38 3/152 8928
  PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
  206    2 admin    RW       0  0.0   0 49.7 [bcmsw_rx]
  805    1 admin    S     8672  1.9   1  0.2 httpd -i br0
  943    1 admin    R     4924  1.1   1  0.2 networkmap --bootwait

This illustrates well the impact a crypto engine can have on actual throughput (and also indicates the kind of performance difference you can expect between an RT-AC88U and an RT-AC86U).

 

[Odkrys]

206 2 admin RW 0 0.0 0 49.7 [bcmsw_rx]

I doubt that broadcom crypto engine really impact on ipsec throughput.
It looks pdc_rx does irq balancing.
Kernel uses only the first core for processing ipsec + nat when pdc_rx isn't loaded.
The benchmark results seem to be due to multicore. (2nd core was free)
[1 core = 133mbps, 2 cores = 309mpbs]

In openvpn, I would expect to get about 100mbps when there is no crypto engine on 1.8Ghz one core.
And ipsec runs on kernel space, so it should be achieved more than openvpn throughput.
/sys/kernel/debug/bcmspu/stats shows the numbers but compared to openvpn(100->250), ipsec does not seem to have much affected by crypto engine.

Edit: If you have some free time, could you test RT-AC88U's ipsec throughput ?

 

[RMerlin]

I doubt that broadcom crypto engine really impact on ipsec throughput.

It does, check my second post showing test results without the engine. Throughput drops to only 133 Mb/s when the crypto engine is not used. Crypto cannot use multiple cores because it's not multithreaded.

In openvpn, I would expect to get about 100mbps when there is no crypto engine on 1.8Ghz one core.

The RT-AC86U reaches over 200 Mbps, due to the support for AES acceleration at the CPU level. The crypto engine is not used then. When I tested with the crypto engine (through cryptodev), performance actually dropped due to the context switch required by OpenVPN (OpenVPN runs in userspace, crypto engine runs in kernel space).

 

[RMerlin]

Edit: If you have some free time, could you test RT-AC88U's ipsec throughput ?

Would be hard to do since my RT-AC88U is fronting my LAN, and my Internet connection would be a bottleneck.

 

[kvic]

Apple and oranges. Configuring an Asus router takes a few minutes and can be done by anyone. Configuring a Microtik can take hours, and requires expect training to achieve. Only a week ago I saw a customer get billed 2 hours by his (other) consultant to configure a VPN tunnel between two Microtiks... And it's the consultant who actually sold him the devices (and configured the first one at their office last spring), so they're not first-time users. If you want to talk costs, then you need to introduce TCO into the equation, not just compare MSRPs.

Beside, the point of this thread is to provide VPN performance datapoints specific to the RT-AC86U, not to start advertising the RT-AC86U as the end-it-all IPSEC solution for businesses.

I'm amused to see you spin this way.. but not surprised.

Regarding OpenVPN, it should be retired a while back. See the speed difference in real-world usage in the above link: OpenVPN vs IPsec vs Shadowsocks.

I could understand your personal bonding to OpenVPN since you first brought it to Asus FW a few years ago. You defend OpenVPN until recently that you have IPsec to "sell" to users on newer Asus routers. lol

Crypto cannot use multiple cores because it's not multithreaded.

Crypto could run in multi-threads in Linux kernel. It even doesn't require any effort from SoC vendors. I used to run IPSec in RT-AC56U utilising both cores in my mod.

 

[Odkrys]

It does, check my second post showing test results without the engine. Throughput drops to only 133 Mb/s when the crypto engine is not used. Crypto cannot use multiple cores because it's not multithreaded.

I mean that crypto engine [bcmspu (crypto driver) + bcmpdc (parallel encryption)] seems just only did parallel encryption.
When compared with significant improvement of openssl, I felt aes encryption didn't accelerated in ipsec.

 

[RMerlin]

I could understand your personal bonding to OpenVPN since you first brought it to Asus FW a few years ago. You defend OpenVPN until recently that you have IPsec to "sell" to users on newer Asus routers. lol

OpenVPN's greatest strength is its flexibility. It can handle a large variety of scenario, can use user-configurable ciphers based on one's needs, and can be run on any tcp/udp port chosen by the user, and its less prone to firewall-related issues than VPN technologies relying on other protocols than TCP or UDP. Its code passed two independent audits 1-2 years ago, and it's very actively developed.

I'm not "selling" IPSEC, in fact I still prefer OpenVPN over IPSEC when using an Asus router. All I did was provide benchmark results since someone actually asked me for those a few days ago.

Nobody's saying OpenVPN is the perfect solution. But just because a newer fad introduces faster protocols does not mean it's suddenly obsolete and everyone should rush toward new, unproven and poorly supported technologies. It might not be the fastest, but it has other strong points in its favor, making it still very much relevant today. And above everything else: it works just fine. Why change just for the sake of changing?

 

[RMerlin]

Crypto could run in multi-threads in Linux kernel. It even doesn't require any effort from SoC vendors. I used to run IPSec in RT-AC56U utilising both cores in my mod.

I'd have to do some tests on the RT-AX88U, where having four cores might make it easier to witness whether the crypto part can truly be multithreaded or not.