Converted a PC to a pfSense Router to test OpenVPN performance

[Xentrk]

VPN in a must have for my use case. But as we all know, the CPUs inside most consumer routers struggle in this regard.

I have been keeping eyes open for a PC to become available that had a CPU with AES-NI support so I could flash it with pfSense to see how OpenVPN performance compared with the AC88U. One became available yesterday. So I ran out to the store and purchased an extra Network Adapter. After installing the NIC, I installed pfSense using a USB. I used the config backup from my current pfSense appliance so setup was a non-event except I had to reinstall the pfBlockerNG package.

The specs of the CPU are:

Intel(R) Core(TM) i5-3450 CPU @ 3.10GHz
4 CPUs: 1 package(s) x 4 core(s)
AES-NI CPU Crypto: Yes (active)

It has been up and running under 24 hours. I am in SE Asia and connected to a VPN Server on the west coast of USA.

I am surprised by the increase in performance, especially when using an ethernet (ETH) connection. The other surprise is the difference between ethernet and wireless performance. I use a D-Link 880L flashed with DD-WRT as the Access Point.

Numbers are Mbps

I think I just found a new router for my home network!

 

[Xentrk]

This contains more information on the VPN performance of the AC86U and the pfSense router.

I also compared the WAN Ethernet and WiFI performance of the pfSense PC paired with a D-Link 880L as the AP and the AC88U.

openssl speed of AC-86U per @RMerlin post

admin@RT-AC86U-DFD8:/tmp/home/root# openssl speed -evp aes-128-cbc
Doing aes-128-cbc for 3s on 16 size blocks: 34942605 aes-128-cbc's in 2.98s
Doing aes-128-cbc for 3s on 64 size blocks: 24912812 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 256 size blocks: 11306808 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 1024 size blocks: 3619044 aes-128-cbc's in 2.99s
Doing aes-128-cbc for 3s on 8192 size blocks: 490938 aes-128-cbc's in 2.97s
OpenSSL 1.0.2j  26 Sep 2016
built on: reproducible build, date unspecified
options:bn(64,32) rc4(ptr,char) des(idx,cisc,16,long) aes(partial) idea(int) blowfish(ptr)
compiler: /opt/toolchains/crosstools-arm-gcc-5.3-linux-4.1-glibc-2.22-binutils-2.25/usr/bin/arm-buildroot-linux-gnueabi-gcc -I. -I.. -I../include  -fPIC -DOPENSSL_PIC -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -DOPENSSL_NO_HEARTBEATS -DL_ENDIAN -Os -march=armv8-a -fomit-frame-pointer -mabi=aapcs-linux -marm -ffixed-r8 -msoft-float -D__ARM_ARCH_8__ -ffunction-sections -fdata-sections -O3 -Wall -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DAES_ASM -DBSAES_ASM -DGHASH_ASM
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-128-cbc     187611.30k   531473.32k   964847.62k  1239431.79k  1354129.33k

pfSense metrics

openssl speed -evp aes-128-cbc
Doing aes-128-cbc for 3s on 16 size blocks: 115298666 aes-128-cbc's in 3.09s
Doing aes-128-cbc for 3s on 64 size blocks: 30287051 aes-128-cbc's in 3.05s
Doing aes-128-cbc for 3s on 256 size blocks: 7651089 aes-128-cbc's in 3.02s
Doing aes-128-cbc for 3s on 1024 size blocks: 1901163 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 8192 size blocks: 243243 aes-128-cbc's in 3.07s
OpenSSL 1.0.2m-freebsd  2 Nov 2017
built on: date not available
options:bn(64,64) rc4(16x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
compiler: clang
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-128-cbc     597801.69k   636183.39k   647831.74k   648930.30k   649004.51k

Side by side comparison. Second row is pfSense router. Third row is AC-86U

type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-128-cbc     597801.69k   636183.39k   647831.74k   648930.30k   649004.51k
aes-128-cbc     187611.30k   531473.32k   964847.62k  1239431.79k  1354129.33k

Ethernet speeds are nearly the same across the two routers. From this data, I need to look at tuning the Wifi in the D-Link AP. I plan to also test using the AC88U as an AP to see if that improves WiFi performance when it is paired with the pfSense router.

 

[MichaelCG]

So looking at the openSSL numbers only...the AC-86U should be able to easily out perform the pfSense i5 box in VPN performance for 256B or larger block sizes? I am purposely ignoring network bandwidth at the moment and just looking at raw VPN performance.

Holy crap...I need a new pfSense box...the numbers from both of the systems you posted just make my current system look pretty bad.

Intel Core2 DUO E4600 @ 2.4GHz

type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-128-cbc      90379.12k   103593.85k   108091.19k   108606.46k   109062.80k

 

[john9527]

Thought I'd pull all the previous data into a single table and add a few entries for comparison

type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes

Merlin (AC86U @ 1.8GHz)
aes-128-cbc     187611.30k   531473.32k   964847.62k  1239431.79k  1354129.33k

pfsense (Intel i5-3450 CPU @ 3.10GHz)
aes-128-cbc     597801.69k   636183.39k   647831.74k   648930.30k   649004.51k

opnsense (Intel N3700 @ 1.6GHz)
aes-128-cbc     200326.87k   322785.00k   384672.70k   405475.07k   411444.68k

pfsense (Intel Core2 DUO E4600 @ 2.4GHz)
aes-128-cbc      90379.12k   103593.85k   108091.19k   108606.46k   109062.80k

LTS Fork (AC68P @ 1200MHz - overclock)
aes-128-cbc      36018.40k    41274.95k    43366.49k    43291.23k    44250.54k

 

[Xentrk]

So looking at the openSSL numbers only...the AC-86U should be able to easily out perform the pfSense i5 box in VPN performance for 256B or larger block sizes? I am purposely ignoring network bandwidth at the moment and just looking at raw VPN performance.

Holy crap...I need a new pfSense box...the numbers from both of the systems you posted just make my current system look pretty bad.

Intel Core2 DUO E4600 @ 2.4GHz

type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-128-cbc      90379.12k   103593.85k   108091.19k   108606.46k   109062.80k

Thanks for posting your metrics. I came to the same conclusion about the pfSense i5 vs the AC86U as you. The PC is a custom build and is approximately six years old. I was able to obtain another box that used to be a Windows Server 2008. I may flash with pfSense to test with. It has an Intel i7 CPU. It stopped booting up the other day so I have to fix that issue first.

 

[sfx2000]

Thought I'd pull all the previous data into a single table and add a few entries for comparison

pfSense 2.4.2_1 - Netgate SG-2440 (Intel Rangeley C2358 @ 1.74GHz)

[2.4.2-RELEASE][admin@router.local.lan]/root: openssl speed -evp aes-128-cbc
Doing aes-128-cbc for 3s on 16 size blocks: 665810 aes-128-cbc's in 0.38s
Doing aes-128-cbc for 3s on 64 size blocks: 640868 aes-128-cbc's in 0.23s
Doing aes-128-cbc for 3s on 256 size blocks: 527398 aes-128-cbc's in 0.19s
Doing aes-128-cbc for 3s on 1024 size blocks: 325775 aes-128-cbc's in 0.20s
Doing aes-128-cbc for 3s on 8192 size blocks: 69190 aes-128-cbc's in 0.05s
OpenSSL 1.0.2m-freebsd  2 Nov 2017
built on: date not available
options:bn(64,64) rc4(16x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
compiler: clang
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-128-cbc      34722.97k   155002.76k   702903.75k  3514848.60k 10336263.02k

Putting things thru the Turbo Entabulator... means these numbers are basically useless for what is really OVPN performance in general - one has to look beyond things..

Merlin (AC86U @ 1.8GHz)
aes-128-cbc     187611.30k   531473.32k   964847.62k  1239431.79k  1354129.33k

pfSense SG-2440 (Intel C2358 @ 1.74GHz)
aes-128-cbc      34722.97k   155002.76k   702903.75k  3514848.60k 10336263.02k

 

[sfx2000]

So these days with OpenSSL speed testing

Add the -elapsed flag... provides for much more accurate numbers. It removes some of the silliness I pointed out above - in any event, it's not just OpenSSL that make a difference with OpenVPN, it's the hardware and the platform.

intel C2358 @ 1.7GHz w/pfSense 2.4.2

aes-128-cbc       3528.67k    13660.13k    45295.19k   108952.23k   188904.79k
aes-128-gcm      83243.74k   163860.12k   231950.25k   258325.55k   264888.32k

intel N3700 @ 1.6GHz... Intel NUC on Ubuntu 16.04LTS

aes-128-cbc     212792.71k   321387.71k   385178.03k   403900.42k   411178.33k
aes-128-gcm     125464.67k   235635.99k   322191.87k   358735.53k   368831.15k

This one is fun - Asus Tinkerboard - Rockchip RK3288 (Quad A17@1.8GHz) - same command line, but Tinkerboard has OpenSSL 1.1.0f

aes-128-cbc      63643.25k    81057.79k    88047.70k    89871.36k    90565.29k    90587.14k
aes-128-gcm      47259.19k    58618.75k    66093.14k    70863.87k    71832.92k    71636.31k

And MacOS 10.13.4 on Core I7@2.3GHz - IvyBridge - MacMini 2012 - running OpenSSL 1.02.n there via HomeBrew...

aes-128-cbc     586287.55k   624049.90k   635853.23k   637082.97k   637580.63k
aes-128-gcm     294048.12k   749649.66k  1104695.21k  1216231.34k  1258162.86k

 

[st3v3n]

sfx2000, Dazzle vs Baffle: my guitarist always said that prior to going onstage in the late 60's. He was brilliant and dazzling, but also threw in a pinch of the BS**. I'm always dazzled by the senior member's brilliance, so excuse my polite babble lest I embarrass myself

Xentrx, After reading your post re the i5-3450 and data from Michael and John, and sfx, I wondered if your box/project relates to mine, and if the performance would be similar for OPNSense, if anyone wants to share. I built the PfSense box in late Nov, preliminary testing with a Xeon 3450/ 4-core w/8 GB ram, the MB NIC and a 4-pt NIC for LAN. The Xeon 3450 installed and ran OK but obviously wasn't going to provide VPN AES going forward, so I dropped an i5-660 in and retested for a couple of days until work interrupted. The i5-660 was surprisingly faster than the 3450.

There aren't many valid comparisons out there for PfSense vs OPNSense and I didn't find much in the way of technical data or videos relating to OPNSense. I discounted flames about who did/didn't do what before, much less to whom. I had read enough about OPNSense to decide to install on our box before finally committing to PfSense. In late January I installed OPNSense and immediately was impressed with the speed, interface, menus, etc https://opnsense.org/users/get-started/#hardware-requirements

Seldom has a firewall/router felt 'right' so quickly. The dev seems clear about the fork, the way forward and how they're going about it: https://opnsense.org/about/about-opnsense/#so-why-did-we-fork-pfsense It hasn't existed as long as PfSense, but regardless of the spat and parting of ways, if OPNSense relates in any way to your project or if you gents care to comment on it if you know, please do. Didn't mean to interrupt, thanks.

 

[sfx2000]

sfx2000, Dazzle vs Baffle: my guitarist always said that prior to going onstage in the late 60's. He was brilliant and dazzling, but also threw in a pinch of the BS**. I'm always dazzled by the senior member's brilliance, so excuse my polite babble lest I embarrass myself

If you can't dazzle them with brilliance - baffle them with B* - they'll never know the difference, lol...

Seriously though - there's a lot of fun numbers flung about - but if one is serious about VPN, one does know what HW to look at, and any consumer router might flout some numbers, proof remains...

And OpenVPN is a poor choice, there are better...

There aren't many valid comparisons out there for PfSense vs OPNSense

There are reasons for this - none of them technical - most folks just pick one or the other - they're more alike than different. But that's all accepted in the BSD community...

 

[john9527]

I thought I remembered another. better test for predicting OpenVPN throughput.....and it was from @sfx2000
https://www.snbforums.com/threads/openvpn-estimate-performance-via-openvpn.33416/

Using that technique.....

(AC68P @ 1200MHz - overclock)
aes-128-cbc  3200/43.22 = 74.04 Mbps
aes 256-cbc  3200/46.94 = 68.17 Mbps

opnsense (Intel N3700 @ 1.6GHz)
aes-128-cbc  3200/22.52 = 142.10 Mbps
aes 256-cbc  3200/23.06 = 138.79 Mbps

Would like to see an AC86 number on this test....

 

[sfx2000]

Rockchip 3288 @ 1.8GHz (cortex-a17) - Asus Tinkerboard

aes-256-cbc 3200/20.54 = 155,79
aes-128-cbc 3200/18.94 = 168.95
aes-128-gcm 3200/18.73 = 170.85

Good reason perhaps to consider the Tinkerboard...

 

[st3v3n]

Indeed. For the most part that used to be the case as far as Dazzle/Baffle, or fool most of the fools much of the time, but in this era, now so many are in the know such as you gents, that's the part that matters. It comes down to what one prefers whether it's Fender vs Gibson, AMD v Intel, Chevy vs Ford, or a Tesla Roadster shot into orbit rather than a Ferrari. Mr. Musk apparently has the pull, the money and the car to waste, not to mention the rocket and broke no national security laws. Since no one complained he touched them, the car looks cool out there without much chance of a head-on collision.

As for the PfSense/OPNSense box and my serious query, the box was on hand; no spending required. The Intel box was pre-built, so free. Our Asus router, hard working and capable as it is, runs only two concurrent OpenVPN tunnels; I wanted to try for four. Commercial VPNs don't care to allow users more than two OpenVPN tunnels, running L2TP or OpenVPN from the user's IP over a private IP address contracted for by the user, regardless pf how many devices are routed over the two encrypted tunnels (I think I said that correctly; long week). If OpenVPN isn't the better of the two choices provided, enlighten away.

It's not as extreme an achievement as launching cars, but anyone who's bent on achieving personal privacy, say streaming favorite episodes of (whatever) from Netflix in addition to other more/less encrypted data on other tunnels, so the local ISP doesn't know exactly which episodes are being watched, well, it's only one example off the top of my head. Granted, that sort of thing only goes so far so if the NSA or others wants to know what the user is watching, they'd look at IPs, the Netflix customer data, or just hack the connection and watch along with said user. I managed two concurrent tunnels using both forks and haven't gotten further, the goal remains getting four concurrent tunnels up and running well both with and without streaming video. Not a burning question, just curious and interested, and I appreciate being able to ask intelligent folks without getting flamed, always a danger in site and places I don't frequent. Such a great forum, thanks for your comments.

 

[MichaelCG]

I fully understand the OpenSSL test is not a true test of overall VPN performance since there are so many other portions to OpenVPN....but still getting a baseline comparison across different platforms can give you an "idea" of what to expect or not to expect out of a specific CPU. Knowing that a specific CPU tops out at 80Mbps on OpenSSL pretty tells you to never expect 100Mbps out of OpenVPN from it. However what it doesn't tell you is that if you have a CPU that can do 240Mbps on OpenSSL does NOT mean it can do anywhere near that on OpenVPN. What a specific platform can do just at the CPU doesn't always take into account what happens when you try shoving the same data through the rest of the buses on the platform.

I am going to go back and read the other thread now to see what testing methods were used there and will repeat on my well aged Core2 DUO.

 

[MichaelCG]

Based on this other test...my previous statements don't hold much water either about being able to make the assumption that OpenVPN speeds in theory will not be able to exceed OpenSSL speeds....very confused here.

So based on the other thread, this gives me 187.9Mbps for OpenVPN??

/root: time openvpn --test-crypto --secret /tmp/secret --verb 0 --tun-mtu 20000 --cipher aes-128-cbc
16.995u 0.015s 0:17.03 99.8%    722+172k 0+0io 0pf+0w

So how does this work if basic OpenSSL can only do around 100Mbps on my system, but the OpenVPN tests are reporting 180Mbps? What am I missing?

 

[sfx2000]

Based on this other test...my previous statements don't hold much water either about being able to make the assumption that OpenVPN speeds in theory will not be able to exceed OpenVPN speeds....very confused here.

So based on the other thread, this gives me 187.9Mbps for OpenVPN??

/root: time openvpn --test-crypto --secret /tmp/secret --verb 0 --tun-mtu 20000 --cipher aes-128-cbc
16.995u 0.015s 0:17.03 99.8%    722+172k 0+0io 0pf+0w

So how does this work if basic OpenSSL can only do around 100Mbps on my system, but the OpenVPN tests are reporting 180Mbps? What am I missing?

There's other factors at play - the OpenSSL and OpenVPN performance tests can give some idea of relative performance, but also goes to machine architecture efficiency and bandwidth across different buses... even the Ethernet card and drivers can play a small factor...

 

[System Error Message]

i have a thinkerboard and a x86 udoo too, so perhaps i could test them soon.

for the asus routers, the broadcom chips they use in them does have some flaws which leads to huge performance drop. When doing VPN you also have to do NAT and routing at the same time so not being able to use hardware NAT with VPN could be what kills performance on a lot of these routers. VPN is CPU intensive even with hardware acceleration, so depending on what your router already does it could cause a significant performance loss. With x86 and even TILE they have many cores and the CPUs are complex enough that the extra processing power needed isnt an issue thanks to the CPUs being super-scalar or vector CPUs with predictions to make use of every cycle.

When i was looking at NIC driver tutorials, intel server NICs are well supported because the NIC chip isnt a blackbox so you can skip kernel stack and even directly interface with the chip yourself. as far as NIC features go, even broadcom, atheros and other decent brands pack quite a lot of features and processing power in their NICs but dont have that driver transparency as good as intel does. This is why mellanox 10G NICs are cheap because they are a blackbox and no one wants them anymore which is why even i have issues with some of them with compatibility.

 

[st3v3n]

System, most informative. The Xeon 4-core ran the firewall fine, the but the dual-core i5-660 was faster, and necessary if we want to use AES. The 4-pt NIC is a dual-CPU HP-NC354T, which is Intel with 4GB aggregate. So far it's never broken a sweat so to speak and real world is more than we'll use day to day, experiments notwithstanding. The NIC was selling for between $40-50 on Amazon. It runs a bit warm, but a fan blowing past the heat-sink sends that out the back. One of these days a tinkerboard would be fun to play with. We have an Android tablet that has an 8-core Rockhip in it; I don't remember which Rockchip but considering the lack of performance even though it supposedly is a 1.8 Ghz 8-core, it can't be the same Rockchip referred to. Thanks for the great post. Cheers.

 

[Xentrk]

Here are my metrics:

aes-256-cbc 3200 9.32 343.35 Mbps
aes-128-cbc 3200 9.15 349.73 Mbps
aes-128-gcm 3200 8.5  376.47 Mbps

 

[sfx2000]

I thought I remembered another. better test for predicting OpenVPN throughput.....and it was from @sfx2000
https://www.snbforums.com/threads/openvpn-estimate-performance-via-openvpn.33416/

Using that technique.....

opnsense (Intel N3700 @ 1.6GHz)
aes-128-cbc 3200/22.52 = 142.10 Mbps
aes 256-cbc 3200/23.06 = 138.79 Mbps

Would like to see an AC86 number on this test....

Interesting - OpenVPN 2.3 (Ubuntu 16.04LTS) on N3700... Intel Nuc box...

aes-128-cbc 3200/12.16 = 263.16 Mbps
aes-256-cbc 3200/12.47 = 256.61 Mbps

Something's up with the OpnSense numbers - maybe config?

Agree - it would be interesting to see where the 86U lands - it's OpenSSL numbers were pretty interesting, but OVPN is more than just that...

 

[Deepcuts]

i3 7100T @ 3.4 Ghz 35W on IPFire.

type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
aes-128-cbc 1151736.60k 1241351.91k 1275924.65k 1285137.75k 1295377.12k
aes-256-cbc 861152.57k 909512.00k 930465.71k 936499.54k 938377.22k

[root@ipfire ~]# time openvpn --test-crypto --secret /tmp/secret --verb 0 --tun-mtu 20000 --cipher aes-256-cbc

real 0m3.903s
user 0m3.893s
sys 0m0.007s

819 Mbps