OpenVPN performance

[kvic]

can you please send it as PM "start a conversation" ?

Just be patient folk.. US will be in soon. I revised the rule further after input from a buddy. I believe that will work very well in stock Merlin firmware.

 

[bilboSNB]

I did a search for dpi and openvpn and ended up here, basically when the bandwidth /app analysis functions are activated
my openvpn client speed drops alot.

Is the jist of this thread that the newer firmware will stop this??

 

[EnF70]

@EnF70, I think the mark set e.g. 0x1 in your case isn't consumed anywhere. I believe the idea is that we want the packets to bypass CTF with the actions of marking. What actually marked doesn't matter. Binary OR gives us a sweet deal.

I believe the difference is there but I can't measure it at hundreds of megabit speed.

I believe the mark set 0x1 it is used (consumed) by the CTF module.
You can see the difference in CPU usage and speed in my original post.
Another point to make is stability, without the mark set and CTF enabled I see sometimes "Broken pipe" messages in syslog and VPN disconnects like this:

Mar  2 11:47:11 kernel: [truncated] L/npch ==NULL/npch ==NULL/npch ==NULL/npch ==NULL/npch ==NULL/npch ==NULL/npch ==NULL/npch ==NULL/npch ==NULL/npch ==NULL/npch ==NUL
Mar  2 11:47:14 openvpn[782]: client1/0.0.0.0:50691 write TCPv4_SERVER: Connection timed out (code=110)                                                         
Mar  2 11:47:14 openvpn[782]: client1/0.0.0.0:50691 write TCPv4_SERVER: Broken pipe (code=32)                                                                   
Mar  2 11:47:14 openvpn[782]: client1/0.0.0.0:50691 write TCPv4_SERVER: Broken pipe (code=32)                                                                   
Mar  2 11:47:14 openvpn[782]: client1/0.0.0.0:50691 write TCPv4_SERVER: Broken pipe (code=32)
[...]
Mar  2 11:47:14 openvpn[782]: client1/0.0.0.0:50691 Connection reset, restarting [0]            
Mar  2 11:47:14 openvpn[782]: client1/0.0.0.0:50691 SIGUSR1[soft,connection-reset] received, client-instance restarting                                         
Mar  2 11:47:56 openvpn[782]: client1/0.0.0.0:50824 Connection reset, restarting [0]                                                                            
Mar  2 11:47:56 openvpn[782]: client1/0.0.0.0:50824 SIGUSR1[soft,connection-reset] received, client-instance restarting

My VPN connection saturates at around 50mbit/sec due to CPU usage reaching 100% on my RT-AC87U with AES-128. When you say you can't measure it at hundreds of megabit speed are you referring to VPN traffic?

Also, I was thinking to continue the investigation with marking packets as the rules I posted are only half of the solution. We are only marking packets coming in from VPN interfaces but we might need to look at marking packets going out to VPN interfaces. For this I was thinking to save the mark in the POSTROUTING mangle chain and then restore the mark in the PREROUTING chain.
This will ensure packets related to VPN connections can be marked before reaching the CTF module (which might not have any effect on the CTF but it's worth checking anyway)
Testing-wise, we should test with CTF disabled and find out if there is any use case where VPN is faster and/or CPU usage is lower than with CTF enabled. This should tell us if we need to do more testing marking packets with CTF enabled.

For me, the rules I posted (and Eric already included in the latest alpha3) have a big impact with CTF enabled.

 

[kvic]

When you say you can't measure it at hundreds of megabit speed are you referring to VPN traffic?

I was referring to efficiency of "--set-mark 0x1" vs "--or-mark 0".


Regarding the effect (and to how much extent it helps overcome CTF pitfalls) of marking packets, personally I've reached my conclusion at the moment as I posted in #178...

 

[RMerlin]

I would assert the speed boost seen in latest Alpha 3 the magic in CTF improvement itself.

https://github.com/RMerl/asuswrt-merlin/commit/cf77301761ed9a94950d34a415d3a69cf2912e61

 

[kvic]

https://github.com/RMerl/asuswrt-merlin/commit/cf77301761ed9a94950d34a415d3a69cf2912e61

From #173...

I know latest Alpha 3 has marking rules added. So flush everything in iptables before my quick tests.

 

[kvic]

I did a search for dpi and openvpn and ended up here, basically when the bandwidth /app analysis functions are activated
my openvpn client speed drops alot.

Is the jist of this thread that the newer firmware will stop this??

I didn't see any slow down with bwdpi/app analysis + openvpn server. But I've only been using TrendMicro stuff for a few days...so do not have much observation on its nitty gritty.

Can you try reproducing the issue in latest 58 Alpha 3?

 

[cosmoxl]

I just wanted to chime in here speaking with regard to CTF and openvpn client usage on my AC68. I disabled CTF long ago because when enabled I noticed that I was getting lots of "leaks" around the VPN tunnel. These "leaks" were connections that were established prior to VPN tunnel creation that were not cut off as they should have been. All traffic was meant to pass through the VPN tunnel, yet netstat-nat tables showed the WAN interface was being used for the still-established connections. My testing showed that with CTF off the "leaks" went away. So, any chance this behavior has changed recently?

 

[EnF70]

I was referring to efficiency of "--set-mark 0x1" vs "--or-mark 0".

Regarding the effect (and to how much extent it helps overcome CTF pitfalls) of marking packets, personally I've reached my conclusion at the moment as I posted in #178...

Now I understand why it works with "--or-mark 0".
It doesn't matter what value we mark the packets with. It seems there is a difference between unmarked and marked packets.
I thought a default mark of 0 (zero) is present on all packets but it seems that's not the case.
I changed my rule to use "--set-mark 0" and it works the same (I even tried with "--or-mark 0", same thing)
From this point of view I like more your "--or-mark 0" as it doesn't change a pre-existing mark on the packets.

The other side of the rule is "-j CONNMARK". This sets the mark on the entire connection, so all (future) packets related to the connection will be marked - if not by an iptable rule then by the "conntrack" module under the hood.
This is a viable solution. I wonder if we need to add a rule to the POSTROUTING chain to catch new connections there as well.
(For instance connections coming on other interfaces going to VPN interfaces).

 

[peraburek]

here is short test of
ASUS RT-AC68U running VPN Client (Core2), downloading 1GB.zip test file from Core i7 OpenVPN server on the other side (Server is using 100/100Mbps)
OpenVPN client is using 32Mbps link
AES-256 encryption (compression disabled)
Asus Merlin FW 380.57, stock CPU speed (not overclocked)
summary: (naked ~ 4,0MB/sec ; with VPN ~ 3,6MB/sec) with ~50% cpu use on Core2 only

if OpenVPN could do multi-thread (multi-core) I guess we could have ~100Mbps VPN with AES-128 on AC68U

 

[kvic]

The other side of the rule is "-j CONNMARK".

connmark was a mistake to suggest. If you want to keep a CTF bypass rule, consider the revised version from my earlier post. It potentially could recoup up to an extra 5 Mbit/s.

But again, the effect of such a rule is nullified for OpenVPN in the Alpha 3 code base... 'cos CTF itself is doing very well. And in the older firmware, such a bypass rule cannot beat the performance with CTF disabled...


For enterprising enthusiasts, I would suggest a little project: port the kernel from SDK 7 (i.e. AC88U) to AC56/AC68/AC87. I expect we will gain lots of performance including CTF.

 

[kvic]

if OpenVPN could do multi-thread (multi-core) I guess we could have ~100Mbps VPN with AES-128 on AC68U

Unfortunatley OpenVPN won't go multi-thread in the foreseeable future. As a reference, AES-128 IPsec (multi-threaded) could hit near 90Mbit/s on 1.4GHz Cortex-A9. So with AES-128 OpenVPN hitting 70Mbit/s isn't that bad...

 

[peraburek]

it isn't bad at all, for advanced home router, 50Mbps VPN speed is pretty decent, and everything above that is just a bonus

 

[EnF70]

connmark was a mistake to suggest. If you want to keep a CTF bypass rule, consider the revised version from my earlier post. It potentially could recoup up to an extra 5 Mbit/s.

But again, the effect of such a rule is nullified for OpenVPN in the Alpha 3 code base... 'cos CTF itself is doing very well. And in the older firmware, such a bypass rule cannot beat the performance with CTF disabled...


For enterprising enthusiasts, I would suggest a little project: port the kernel from SDK 7 (i.e. AC88U) to AC56/AC68/AC87. I expect we will gain lots of performance including CTF.

I am on Alpha3 and I think there are still issues with CTF enabled if we don't mark packets.
Can you post again the latest rule, I want to make sure I get the correct one. (I can still see a connmark in the latest one)

EDIT: I think I know which one you are referring to now:
iptables -t mangle -A PREROUTING -i tun21 -m conntrack --ctstate NEW -j MARK --or-mark 0

 

[kvic]

I just wanted to chime in here speaking with regard to CTF and openvpn client usage on my AC68. I disabled CTF long ago because when enabled I noticed that I was getting lots of "leaks" around the VPN tunnel. These "leaks" were connections that were established prior to VPN tunnel creation that were not cut off as they should have been. All traffic was meant to pass through the VPN tunnel, yet netstat-nat tables showed the WAN interface was being used for the still-established connections. My testing showed that with CTF off the "leaks" went away. So, any chance this behavior has changed recently?

Interesting observations...personally I don't run VPN client. I wish I could offer you some ideas..but really not much here. Perhaps give Alpha 3 a try and let us know...

 

[kvic]

I am on Alpha3 and I think there are still issues with CTF enabled if we don't mark packets.

Then I can only attribute the difference to me using a different Toolchains to compile the firmware (as I documented in #173) but I doubt a slightly different uClibc library could make such a huge difference...

 

[RMerlin]

I am on Alpha3 and I think there are still issues with CTF enabled if we don't mark packets

alpha 3 marks packets already.

 

[EnF70]

alpha 3 marks packets already.

I know, I see the rules in iptables - thanks for adding them
I'm flushing the chains before testing different rules and I tested Alpha3 with no rules at all.
That's why I said that there are still issues with CTF enabled if we don't mark packets.

 

[Goobi]

So, I just want to confirm the rule and see if I could add it dynamically to the running configuration. So the rule is:

iptables -t mangle -A PREROUTING -i tun21 -m conntrack --ctstate NEW -j MARK --or-mark 0

I suspect that this goes into the firewall-start script. I already got stuff in there, does this needs to be placed in a certain location or can it be put at the bottom of the file? Also, can I run the above command dynamically? Last question, if I restart the VPN client do I need to reissue this command?

Thanks in advance.

 

[peraburek]

@kvic or @EnF70
is this current (latest, best?) command to mark VPN packets for OpenVPN Server 1:

iptables -t mangle -A PREROUTING -i tun21 -m conntrack --ctstate NEW -j MARK --or-mark 0

@Goobi - you are right, put that line in firewall-start and you can check if packets are marked with command

iptables -L -t mangle -vn