OpenVPN performance of the RT-AC86U

[RMerlin]

I've run a few iperf tests through an OpenVPN tunnel with the OpenVPN 2.4 server running on the RT-AC86U (on an early alpha build of Asuswrt-Merlin). The iperf client ran on my i7 7700K desktop, and the iperf server was on an i5 5200 laptop, connected on the other side of the RT-AC86U. Both computers are connected over Gigabit Ethernet.

Tests were run with various ciphers. I used the same iperf parameters as I've used in the past, so these can be compared with my previous test results posted here on SNBForums. From memory, the RT-N66U was getting around 22 Mbps, and the RT-AC68U was around 50-60 Mbps. Both were using AES-128-CBC, with OpenVPN 2.3.

iperf command parameters:

P:\Tools>iperf -c 192.168.1.10 -M 1400 -N -l 64K -t 30

Here are the results.

AES-128-CBC + LZO (Adaptive):

[ ID] Interval       Transfer     Bandwidth
[296]  0.0-30.0 sec    698 MBytes    195 Mbits/sec

AES-128-GCM + LZO:

[ ID] Interval       Transfer     Bandwidth
[292]  0.0-30.0 sec    735 MBytes    205 Mbits/sec

AES-128-GCM + LZ4:

[ ID] Interval       Transfer     Bandwidth
[296]  0.0-30.0 sec    768 MBytes    215 Mbits/sec

Interestingly, the performance hit of AES-256-GCM over AES-128-GCM is negligeable, implying that the bottleneck does not lies in the cipher, but in the rest of the OpenVPN code.
AES-256-GCM + LZ4:

[ ID] Interval       Transfer     Bandwidth
[292]  0.0-30.0 sec    755 MBytes    211 Mbits/sec

As I expected, this router is a beast for OpenVPN. Performance should be close to that with the GT-AC5300 (Asus's OpenVPN isn't entirely as optimized as my implementation, but it shouldn't be far behind).

So basically, expect around 200 Mbps of throughput (I doubt the GT-AC5300's extra cores will make any real difference, OpenVPN being single threaded). That's about 4x faster than the RT-AC68U, and probably around 3x faster than the RT-AC88U/RT-AC3100/RT-AC5300.

 

[Temchenko]

Great! Now will have to wait for your firmware to roll out. Really exited about that. Hope that AC86U will be new bestseller for value/money standpoint.

Отправлено с моего XT1635-02 через Tapatalk

 

[pete y testing]

just got a hold of one of these to have a look at

to my big surprise is its form factor , its pretty much the exact same size as the rt-ac68u , so im glad its not a big honking router to start with

just did a quick samba read throughput and it was sitting around 110Mb/s when tested with my asus pce-ac88 and test comp with my ds415+ connected to the router

so this thing has some big kahunas

 

[RMerlin]

just did a quick samba read throughput and it was sitting around 110Mb/s when tested with my asus pce-ac88 and test comp with my ds415+ connected to the router

Asus' stock firmware is also able to push 70 MB/s in write speed (I made it past 100 MB/s in my firmware by adjusting CPU affinity). And that's with SMB2 enabled. Quite impressive compared to past generations.

Try an OpenSSL speed test for more fun...

admin@RT-AC86U-DFD8:/tmp/home/root# openssl speed -evp aes-128-cbc
Doing aes-128-cbc for 3s on 16 size blocks: 34942605 aes-128-cbc's in 2.98s
Doing aes-128-cbc for 3s on 64 size blocks: 24912812 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 256 size blocks: 11306808 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 1024 size blocks: 3619044 aes-128-cbc's in 2.99s
Doing aes-128-cbc for 3s on 8192 size blocks: 490938 aes-128-cbc's in 2.97s
OpenSSL 1.0.2j  26 Sep 2016
built on: reproducible build, date unspecified
options:bn(64,32) rc4(ptr,char) des(idx,cisc,16,long) aes(partial) idea(int) blowfish(ptr)
compiler: /opt/toolchains/crosstools-arm-gcc-5.3-linux-4.1-glibc-2.22-binutils-2.25/usr/bin/arm-buildroot-linux-gnueabi-gcc -I. -I.. -I../include  -fPIC -DOPENSSL_PIC -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -DOPENSSL_NO_HEARTBEATS -DL_ENDIAN -Os -march=armv8-a -fomit-frame-pointer -mabi=aapcs-linux -marm -ffixed-r8 -msoft-float -D__ARM_ARCH_8__ -ffunction-sections -fdata-sections -O3 -Wall -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DAES_ASM -DBSAES_ASM -DGHASH_ASM
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-128-cbc     187611.30k   531473.32k   964847.62k  1239431.79k  1354129.33k

(Make sure to use -evp to get the CPU-enhanced code path).

 

[LouisvilleUK]

And... Order has been placed.

 

[maxbraketorque]

Impressive. Looking forward to hearing about more tests. This router clearly has the potential to completely kill off RT-AC88U sales, so this is yet another reason that I'm also puzzled as to what ASUS plans to do with the RT-AC88U.

 

[RMerlin]

Impressive. Looking forward to hearing about more tests. This router clearly has the potential to completely kill off RT-AC88U sales, so this is yet another reason that I'm also puzzled as to what ASUS plans to do with the RT-AC88U.

The RT-AC86U is indeed in an odd spot. Price is between the RT-AC68U and RT-AC88U, but performance is better than the RT-AC88U, at the cost of one single stream lost on the 2.4 GHz band, and only LAN 4 ports. If anything, it's the RT-AC3100 that feels redundant now at this point. It might have slightly better coverage than the RT-AC86U due to having four detachable external antennas.

I wonder if going with a 4x4 design on the 5 GHz band wasn't a last minute change by Asus, as the fourth antenna needed by it is internal.

The platform still has growing pains, as it's a MAJOR change over past Broadcom products, but it's getting there.

 

[pete y testing]

Asus' stock firmware is also able to push 70 MB/s in write speed (I made it past 100 MB/s in my firmware by adjusting CPU affinity). And that's with SMB2 enabled. Quite impressive compared to past generations.

btw are you using the pce-ac88 for testing ? i have a strange issue with write speeds over the pce-ac88 where it seems to be limiting it to around 22MB/s , irs a brand new test rig with new install of windows 10 and using the asus drivers

will have a play with OpenSSL once i get this dam write issue sorted

If anything, it's the RT-AC3100 that feels redundant now at this point. It might have slightly better coverage than the RT-AC86U due to having four detachable external antennas.

and thats how i see it too , the 88u is aimed at a different crowd , the 86u is the direct replacement for the 68u ( we dont even get the ac3100 here in OZ anyway ) , and as the direct replacement for the rt-ac68u this is indeed worth the upgrade , this is prob the international version that everyone will get instead of the limited rel;ease of the rt-ac3100

I wonder if going with a 4x4 design on the 5 GHz band wasn't a last minute change by Asus, as the fourth antenna needed by it is internal.

was ether that or form factor choice , i quite like the fact its not as big as the titanic , i will say im still getting 1,6M sync at 15 meters through a semi solid wall on 5 gig and that compares well to the other 2156M 5 gig clients if have tested

 

[RMerlin]

btw are you using the pce-ac88 for testing ?

No, all my development is done over Ethernet and serial, from my laptop. I did get some odd SMB results however when I was doing my initial test, I was getting less than 1 MB/s. Was caused by my USB Ethernet adapter being in an USB2 instead of an USB3 port. Which is still odd, as I was getting nowhere close to the 20-30 MB/s I'd expect out of USB2. Finetuning the router's socket options in smb.conf did make a difference. I assume the issue is when the network link isn't fast enough, the router isn't properly filling up the pipe. I don't remember which specific socket option was causing it tho, I'll have to revisit that at a later time.

So maybe you are experiencing something similar with your wireless adapter.

One thing you can try is manually running smbd on the second CPU core, using the taskset program (should also be present in stock FW).

was ether that or form factor choice , i quite like the fact its not as big as the titanic

The form factor is indeed really good. Won't appeal to people wanting to wall mount it, but still. It should also be doing fairly good for cooling, even tho they used one single large heatsink.

BTW if you ever want to dismantle it, the screws are hidden behind the back label. I opened mine so I could hook up some wires to the serial header, ran them out through the back grill. The connector headers barely fit once the router was closed (I had to leave the pcboard screws a bit looser to accomodate for it).

I need to find wires with shorter connection headers than what ships with all my TTL adapters. So far I was only able to run them through my RT-AC5300 (drilling holes on the side to run them out) and on the back of this RT-AC86U. My RT-AC88U and RT-AC87U have their top permanently left unscrewed, so I can quickly hook up wires to the header.

 

[Voxel]

(Make sure to use -evp to get the CPU-enhanced code path).

Interesting. Very impressive...

Do you have the same results with "-elapsed" option?

And what does "openssl engine" say?

Voxel.

 

[RMerlin]

Interesting. Very impressive...

Do you have the same results with "-elapsed" option?

Not much difference with elapsed:

The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-128-cbc     192624.81k   540520.01k   969324.08k  1235202.71k  1348357.09k

And what does "openssl engine" say?

Voxel.

Asus disabled OpenSSL engines on the HND platform. I had to re-enable them otherwise Tor would fail to build, however there's no Broadcom-specific engine in the firmware (and Asus doesn't even include the default ones either, as they ain't used anyway). At this time, all these improvements come from the B53 architecture and HW AES instructions.

I haven't had time to dig through the kernel options yet to see if they have a BCM HW acceleration option that could be enabled or disabled, I'm still at the implementation phase for the whole platform. Optimization will come later.

 

[RMerlin]

I spent some time playing with it last night. I suspect that OpenSSL -evp is already using the Broadcom crypto engine (it's the bcmspu module, which is loaded in memory), since those results I obtained were beating the pants out of all other home routers I've seen so far. Would be nice having a way to be 100% sure that this performance is coming from bcmspu and not just ISA improvements (AES operand).

I experimented with cryptodev-linux. I added it to the firmware, and loaded it with OpenSSL, and the performance results were all over the place:

type         16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
EVP noeng   193108.05k    543690.04k   974732.03k  1244698.97k  1351245.82k
Plain        67742.24k     73860.50k    77246.59k    78053.03k    78311.68k
Engine       67613.86k     74011.39k    77270.08k    78063.62k    78289.77k
Evp          30849.28k    112378.76k   533779.91k  2081088.00k 21977224.53k
Evp+Eng      24085.54k    138212.27k   398918.40k  2079193.60k  9392713.14k

Really odd results. Those numbers with the engine would vary greatly between test runs, while the EVP noengine was always consistant.

 

[RMerlin]

Some further tests, as I found how to obtain usage stats of the crypto engine. Turns out I was wrong - the engine is not used at all when getting this 200 Mbps of OpenVPN throughput. So I gave it a shot, using cryptodev-linux, and enabling it in OpenVPN.

This time, the engine is indeed used:

admin@RT-AC86U-DFD8:/tmp/home/root# cat /debug/bcmspu/stats
Number of SPUs.........1
Number of channels.....1
Current sessions.......2
Session count..........137
Cipher setkey..........137
HMAC setkey............0
AEAD setkey............0
Cipher Ops.............2599887
Hash Ops...............0
HMAC Ops...............0
AEAD Ops...............0
AEAD fallback Ops......0
Bytes of req data......2382296400
Bytes of resp data.....2382296400
Message send failures..0
Check ICV errors.......0

Test results are kinda disappointing however:

[292] local 10.8.0.2 port 6168 connected with 192.168.1.10 port 5001
[ ID] Interval       Transfer     Bandwidth
[292]  0.0-30.0 sec    370 MBytes    104 Mbits/sec

Using the hw crypto engine ends up giving only half of the throughput achieved by using the CPU-only (with AES operand) performance. Makes me wonder what's the point in that crypto engine if for small packets such as those used by OpenVPN we get worse performance than with a CPU-based method. That would only make it useful to encrypt a file (for instance).

I also noticed that the engine was not used when using AES-128-GCM, I had to revert back to AES-128-CBC. Might be because the OpenSSL cryptoengine code is outdated tho, I heard there was a newer patch available from the cryptodev-linux devs.

Might be interesting to see someday if the same results could be seen with IPSec instead of OpenVPN.

EDIT: Some further tests, this time done solely at OpenVPN's level rather than using iperf through a tunnel:

admin@RT-AC86U-DFD8:/tmp/home/root# time openvpn --test-crypto --secret /tmp/test --verb 0 --tun-mtu 20000 --cipher aes-128-cbc --engine cryptodev
Thu Sep 28 01:53:32 2017 disabling NCP mode (--ncp-disable) because not in P2MP client or server mode
real    0m 16.27s
user    0m 13.87s
sys    0m 0.53s
admin@RT-AC86U-DFD8:/tmp/home/root# time openvpn --test-crypto --secret /tmp/test --verb 0 --tun-mtu 20000 --cipher aes-128-cbc
Thu Sep 28 01:53:58 2017 disabling NCP mode (--ncp-disable) because not in P2MP client or server mode
real    0m 14.09s
user    0m 13.96s
sys    0m 0.00s

Performance does go down when using the hardware engine.

 

[LouisvilleUK]

Using the hw crypto engine ends up giving only half of the throughput achieved by using the CPU-only (with AES operand) performance. Makes me wonder what's the point in that crypto engine if for small packets such as those used by OpenVPN we get worse performance than with a CPU-based method. That would only make it useful to encrypt a file (for instance).

I also noticed that the engine was not used when using AES-128-GCM, I had to revert back to AES-128-CBC.

This is indeed a head scratcher and very disappointing.

 

[RMerlin]

This is indeed a head scratcher and very disappointing.

It's possible that IPSEC might do better. One big problem with OpenVPN is that the crypto code is all in userspace, requiring context switches if using a kernel-based crypto engine (as it uses OpenSSL, a userland library).

With IPSEC, you can use kernel modules for the crypto code, so it's more optimal if also using a kernel mode crypto engine.

The engine itself is certainly more than capable, if you look at the numbers on the 1 KB and 8 KB blocks, where it blows away CPU-only crypto.

 

[SeaConn]

Best of all, it doesn't look like a toy alien spaceship... well, less so than the RT-AC5300, anyway. I thought that that was necessary for "blazing speeds", kind of like a flame job on a 1940's roadster.

I'm sold. I'm putting one on order now. I'm gonna miss the policy based routing feature in Asuswrt-Merlin while awaiting the beta release for the RT-AC86U. I guess I'll put the whole house on the VPN for now using the stock firmware. I'll be the Oprah Winfrey of VPN client providers - a VPN for you, and a VPN for you, and now look under your seats. Everybody gets a VPN client!

 

[LouisvilleUK]

I'm gonna miss the policy based routing feature in Asuswrt-Merlin while awaiting the beta release for the RT-AC86U.

You can give alpha3 a spin. It has policy based routing…

https://www.snbforums.com/threads/alpha-asuswrt-merlin-382-1-alpha.41380/

I’ve been running it with OpenVPN client for days with no issues after correcting my own user errors. Very pleased with performance.

 

[SeaConn]

You can give alpha3 a spin. It has policy based routing…

https://www.snbforums.com/threads/alpha-asuswrt-merlin-382-1-alpha.41380/

I’ve been running it with OpenVPN client for days with no issues after correcting my own user errors. Very pleased with performance.

Really? Sweet! I'll load up the alpha when it arrives this Friday. Thanks! I was concerned about updating the stock firmware anyway, as I've heard that on some other models, upgrading the stock firmware can make the installation of Merlin difficult if not impossible later down the line. I have no idea if that's true -- it's just something I read in the Amazon comments.

I'm also wondering if/when LACP 802.3ad can/will be enabled on the RT-AC86U. Not that it much matters. I have it running on my RT-AC66U and it does seem to make the fast forward function in Plex more responsive for some reason.

There will also be a small funeral procession for IPsec V4 this Friday as I cart off ye ole RT-AC66U to its final resting place (queue funeral music). It's going to the other side -- repeater land. And by "other side", I obviously mean the other side of the house.

 

[RMerlin]

OpenVPN still is quirky with the latest alpha build that's currently available. Best wait for the next test build, quite a few fixes have been applied for OpenVPN.

 

[SeaConn]

OpenVPN still is quirky with the latest alpha build that's currently available. Best wait for the next test build, quite a few fixes have been applied for OpenVPN.

Firstly, thanks for all the hard work, RMerlin, past and present. I burned my WRT54g like Jimi Hendrix many years ago after discovering Asuswrt-Merlin and have never looked back.

I'll take your expert advice and wait for the next release. I see there were a lot of fixes from the linked thread above. I run a Transmission+OpenVPN docker on my NAS anyway; i.e., if the router drops the VPN client, it doesn't really matter as it'll only expose the second VPN client's IPv4 address on the docker. Of course, it's all for naught if I just plain can't get it to work.

I'll be reading up on AP mode for the RT-AC66U and patiently awaiting the next release