Extremely slow VPN Server transfer speeds even on AES-NI supported RT-AC86U. Out of ideas, please help.

[domic]

Hello. After researching online, I bought the RT-AC86U today after reading online for a few weeks that it would be great for faster VPN speeds since it had hardware acceleration built in for encrypting traffic that other models didn't have. (I went from a RT-AC66U B1 that lacks such HW acceleration capabilities).
However, after trying all day to setup a OpenVPN Server and (PPTP Server) on both Asus latest and AsusWRT-Merlin latest firmwares, I still have terrible VPN speeds, at about 4-10 Mbps (200-800 kBytes/s) while doing test file transfers over SMB, FTP and internet speedtests.
I need SMB to work fast over a VPN connection to my home network when I'm traveling so I can access both my USB-connected HDD in my Asus router, plus my Windows computers in the local network.

FTP is out of the question and people online seem to be against port forwarding SMB connections to the internet, that's why I (so far) chose the VPN Server approach.

So, how can I achieve fast VPN speeds for file transfer when the VPN server (whichever protocol works best) is hosted on the AC86U router itself? I bought it specifically to get accelerated VPN speeds.
I mean, This router has the hardware (Dual Core 1,8 GHz CPU, extra "AES-NI" encryption chip for HW acceleration, etc), it SHOULD be able to run well for at least one of the built in VPN servers on the router, as it already has been proven in other threads that even 90-200 MBit connections when it comes to using the RT-AC86U as a OpenVPN client to connect to VPN providers online (PIA, NordVPN for example).

@RMerlin, you seemed quite informed about the HW acceleration thing in some newer Asus routers, is there any way I could use it for the OpenVPN server on the Merlin software, so my phone/laptop can connect to my home network and still transfer files much faster than the meager 4-10 Mpbs both my AC66U B1 and now my more powerful AC86U (so far with my tried configs) offer?

Non-VPN connections reach max speeds that I pay my ISP for btw. 300 Mbps no problem.
I really appreciate any help I can get.
Surely there must be someone else out there who wants their own private "cloud storage" with fast VPN connections to their home router like me.
At this point I don't think buying a RT-AX88U would make any difference either since this one I have not didnt even improve the speed even a little.

 

[ColinTaylor]

What is the upload speed of your internet connection? (you said your download is 300Mbps)

How are you testing for the speeds you are seeing?

 

[CaptainSTX]

What is your upload speed? When you connect to your VPN server from outside your home the speed won't be any faster than the upload speed from your ISP and the VPN may reduce that even further.

 

[eibgrad]

As suggested above, you're always limited by your weakest link, which in many cases is the crappy asymmetric bandwidth provided by your ISP. If this is indeed the problem, and you have the option, consider fiber.

 

[domic]

What is the upload speed of your internet connection? (you said your download is 300Mbps)

How are you testing for the speeds you are seeing?

My upload speed is 10-ish Mbps (coaxial cable), but it doesn't make sense (to me) that over VPN it would be the same download speed as the download speed.
I am testing speeds using my phone, in a file manager app, sending and receiving test files one way, to the hard drive connected to my router via the ftp server onboard, then check the file transfer speed, first on unencrypted FTP over the internet, which gives full 300 Mbit/s downlink, 10-15 uplink, then over both PPTP and OpenVPN server on the router, and the speeds are both less than 10 Mbit most of the time.

 

[ColinTaylor]

It sounds like your phone might not be capable of encrypting the data any faster. Can you try with a more powerful client, like a laptop?

 

[Xentrk]

If it makes you feel any better, I too never experienced improved OpenVPN speeds on RT-AC86U model when compared to RT-AC88U that others reported. I connect to servers half way across the globe so distance may come into play. I did see significant OpenVPN improvement on recent testing with RT-AX88U model when I was staging it last weekend. I don't have the data to post on it right now though.

RT-AC86U does have support for Wireguard though. Here is a OpenVPN vs Wireguard speed comparison on RT-AC86U.

OpenVPN

Just for comparison sakes, on my pfSense v2.5.0 PC, i5 AES-NI enabled, I am getting 163 Mpbs download.

Wireguard

 

[domic]

If it makes you feel any better, I too never experienced improved OpenVPN speeds on RT-AC86U model when compared to RT-AC88U that others reported. I connect to servers half way across the globe so distance may come into play. I did see significant OpenVPN improvement on recent testing with RT-AX88U model when I was staging it last weekend. I don't have the data to post on it right now though.

RT-AC86U does have support for Wireguard though. Here is a OpenVPN vs Wireguard speed comparison on RT-AC86U.

OpenVPN
View attachment 31355

Just for comparison sakes, on my pfSense v2.5.0 PC, i5 AES-NI enabled, I am getting 163 Mpbs download.

Wireguard
View attachment 31354

Dude, please read the thread before you reply. I am not using any VPN providers, I'm trying to setup my own personal 'cloud' at home, using my Asus router's own VPN server software that's available in the Web GUI. I want to connect to my router from anywhere in the world and still be able to (securely) access all my files on my USB connected har hard-drive. Like a NAS, but everything is setup in he router.

 

[domic]

It sounds like your phone might not be capable of encrypting the data any faster. Can you try with a more powerful client, like a laptop?

Well, I have a Snapdragon 845, but I'll give it a quick try tomorrow morning on my laptop to follow the process of elimination, however I doubt the problem is there.

 

[Martineau]

Well, I have a Snapdragon 845, but I'll give it a quick try tomorrow morning on my laptop to follow the process of elimination, however I doubt the problem is there.

If you have tried both PPTP and OpenVPN as servers, have you considered setting up an IPSec VPN or a WireGuard server?
If all four protocols perform equally badly then that may help in the process of elimination.

 

[dosborne]

You may want to check the stats on your router while your tests are running to see if a process is maxing out the cpu or memory?

 

[Xentrk]

Dude, please read the thread before you reply. I am not using any VPN providers, I'm trying to setup my own personal 'cloud' at home, using my Asus router's own VPN server software that's available in the Web GUI. I want to connect to my router from anywhere in the world and still be able to (securely) access all my files on my USB connected har hard-drive. Like a NAS, but everything is setup in

My name is not Dude. It is DOUGLAS aka Xentrk. Just trying my best to help. It was the “..out of ideas. Please help.” plea that made me think my post might be helpful. Just trying to point out that the CPU in AC86U might not result in improved OpenVPN speeds as others have reported on the forum and perhaps implementing Wireguard on server and client may improve speeds.

 

[domic]

If you have tried both PPTP and OpenVPN as servers, have you considered setting up an IPSec VPN or a WireGuard server?
If all four protocols perform equally badly then that may help in the process of elimination.

Hi. I tried a quick IPSec server too last night but same story sadly, no better luck there. I don't know what that is or how to setup Wireguard server.

 

[domic]

My name is not Dude. It is DOUGLAS aka Xentrk. Just trying my best to help. It was the “..out of ideas. Please help.” plea that made me think my post might be helpful. Just trying to point out that the CPU in AC86U might not result in improved OpenVPN speeds as others have reported on the forum and perhaps implementing Wireguard on server and client may improve speeds.

Sorry Xentrk. I forgot my manners.

Well, everything is possible, but I thought I'd ask for help at this point as I've reached the extent of my own knowledge on the matter.

Is Wireguard Web GUI 'setupable' or is that a part of CLI territory?

 

[eibgrad]

Well, I have a Snapdragon 845, but I'll give it a quick try tomorrow morning on my laptop to follow the process of elimination, however I doubt the problem is there.

I for one would NOT be surprised if a laptop proved much better.

When users upgrade their routers for better OpenVPN (or PPTP) performance, it's usually because they're using the OpenVPN client. But when you switch to needing OpenVPN server support on your own router, you're sort of back to the same problem, where you're potentially limited by the capabilities of the OpenVPN client again, this time on the smartphone. And that's why you need more than one data point when it comes to determining the actual source of the problem, whether its multiple client devices, or even protocols (what about scp?).

It would be interesting to try Wireguard and see if things improved dramatically (since its the only VPN I know of that runs in the kernel, which accounts for much of the improvement over other VPNs that must run in user-space), but given the state of Wireguard today, it's probably not reasonable to expect this from the OP.

 

[domic]

I for one would NOT be surprised if a laptop proved much better.

When users upgrade their routers for better OpenVPN (or PPTP) performance, it's usually because they're using the OpenVPN client. But when you switch to needing OpenVPN server support on your own router, you're sort of back to the same problem, where you're potentially limited by the capabilities of the OpenVPN client again, this time on the smartphone. And that's why you need more than one data point when it comes to determining the actual source of the problem, whether its multiple client devices, or even protocols (what about scp?).

It would be interesting to try Wireguard and see if things improved dramatically (since its the only VPN I know of that runs in the kernel, which accounts for much of the improvement over other VPNs that must run in user-space), but given the state of Wireguard today, it's probably not reasonable to expect this from the OP.

I don't know how comfortable I would be with trying Wireguard if it's all CLI based. Too many things that could go wrong with my knowledge.

I could look into trying to host a VPN server on a PC or (maybe a NAS?) to let my devices into the network. Are there any easy ways to get that up and running on a Windows machine (with the windows firewall and all that stuff to account for in terms of obstacles/security so I don't give full access to my PC unknowingly that I'm hosting VPN server on?
Is there an 'easy' way to do it?

 

[Xentrk]

Sorry Xentrk. I forgot my manners.

Well, everything is possible, but I thought I'd ask for help at this point as I've reached the extent of my own knowledge on the matter.

Is Wireguard Web GUI 'setupable' or is that a part of CLI territory?

It is all CLI for now. Very minimal though. Instructions are in the post. I have some existing code I can patch together to perform the download of the appropriate apk and install of entware packages as a first step though.

I thought Wireguard was worth a mention since OpenVPN performance was an issue you mentioned. I have been testing WG client in parallel this past week on a RT-AC86U and pfSense appliance. Very happy with the performance bump. I really noticed an improvement on a FireTV that connects to the router over WiFi. The sound and picture quality are improved and no more buffering.

I plan on testing the client/server config next with my Android phone as the client.

Just watched this VDO today. Even though it is pfSense, it will help explain some concepts that cross platforms. One thing I learned is that there is not really a client and a server in WG. Each one are called peers and the connection is a tunnel.

Other reference

Quick Start - WireGuard

www.wireguard.com

 

[domic]

It is all CLI for now. Very minimal though. Instructions are in the post. I have some existing code I can patch together to perform the download of the appropriate apk and install of entware packages as a first step though.

I thought Wireguard was worth a mention since OpenVPN performance was an issue you mentioned. I have been testing WG client in parallel this past week on a RT-AC86U and pfSense appliance. Very happy with the performance bump. I really noticed an improvement on a FireTV that connects to the router over WiFi. The sound and picture quality are improved and no more buffering.

I plan on testing the client/server config next with my Android phone as the client.

Just watched this VDO today. Even though it is pfSense, it will help explain some concepts that cross platforms. One thing I learned is that there is not really a client and a server in WG. Each one are called peers and the connection is a tunnel.

Other reference

Quick Start - WireGuard

www.wireguard.com

I read about wireguard on their website just now and the technology sounds amazing, with minimal attack vectors and it's goal to be simple to use and all that stuff.

But it's not yet consumer-ready from what I can see, which is probably why neither Asus stock firmware or Merlin firmware has it in the router Web GUI yet. Shame, maybe in a future firmware.
I need to figure out something that works at the moment though. Thanks for the suggestion though.

 

[RMerlin]

If your upstream is only 10 Mbps, that will be your hardcap when moving files from the home network to your remote client. 4-10 Mbps seems pretty much in line with that 10 Mbps limit.

Test both TCP and UDP, also pay close attention to your MTU, especially in UDP mode.

 

[eibgrad]

If your upstream is only 10 Mbps, that will be your hardcap when moving files from the home network to your remote client. 4-10 Mbps seems pretty much in line with that 10 Mbps limit.

I agree, if the issue is *download* speeds from the perspective of the smartphone as a remote OpenVPN client. But it's my understanding (and I could be wrong) that the OP is claiming the *upload* speeds from his remote OpenVPN client and to his FTP server back home are only 10Mbps, which if he has 300Mbps download from his ISP, doesn't make sense. That's why I and others are questioning if his smartphone as an OpenVPN client is the problem.