OpenVPN issues on AC88u using latest 384.19

[Phk]

Hello all,

I wonder if anyone's noticing a problem with the openvpn connections using merlinWRT as the openvpn server, and I'd like to know how to fix this.

I'm using AC88u with latest 384.19 and the issue is that VPN becomes extremely slow (down to like 50k/s instead of 8MB/s) after using it for a while to transfer large files. I mean, it does work of for half an hour or so, but then it becomes miserable until I reboot the router.

I don't remember this happening with the previous versions, although I cannot be sure of when this started.

The only thing I can see in the logs is thousands of lines with this info:
Sep 6 18:10:34 ovpn-server1[1318]: username/A.B.C.D:6916 AEAD Decrypt error: bad packet ID (may be a replay): [ #1112770 ] -- see the man page entry for --no-replay and --replay-window for more info or silence this warning with --mute-replay-warnings Sep 6 18:10:34 ovpn-server1[1318]: username/A.B.C.D:6916 AEAD Decrypt error: bad packet ID (may be a replay): [ #1112771 ] -- see the man page entry for --no-replay and --replay-window for more info or silence this warning with --mute-replay-warnings Sep 6 18:10:34 ovpn-server1[1318]: username/A.B.C.D:6916 AEAD Decrypt error: bad packet ID (may be a replay): [ #1112772 ] -- see the man page entry for --no-replay and --replay-window for more info or silence this warning with --mute-replay-warnings Sep 6 18:10:34 ovpn-server1[1318]: username/A.B.C.D:6916 AEAD Decrypt error: bad packet ID (may be a replay): [ #1112773 ] -- see the man page entry for --no-replay and --replay-window for more info or silence this warning with --mute-replay-warnings Sep 6 18:10:34 ovpn-server1[1318]: username/A.B.C.D:6916 AEAD Decrypt error: bad packet ID (may be a replay): [ #1112774 ] -- see the man page entry for --no-replay and --replay-window for more info or silence this warning with --mute-replay-warnings Sep 6 18:10:34 ovpn-server1[1318]: username/A.B.C.D:6916 AEAD Decrypt error: bad packet ID (may be a replay): [ #1112775 ] -- see the man page entry for --no-replay and --replay-window for more info or silence this warning with --mute-replay-warnings Sep 6 18:10:34 ovpn-server1[1318]: username/A.B.C.D:6916 AEAD Decrypt error: bad packet ID (may be a replay): [ #1112776 ] -- see the man page entry for --no-replay and --replay-window for more info or silence this warning with --mute-replay-warnings Sep 6 18:10:34 ovpn-server1[1318]: username/A.B.C.D:6916 AEAD Decrypt error: bad packet ID (may be a replay): [ #1112777 ] -- see the man page entry for --no-replay and --replay-window for more info or silence this warning with --mute-replay-warnings

And here's the relevant part in the setup UI:

Any help with this would be really appreciated.

Thanks

 

[Martineau]

Hello all,

I wonder if anyone's noticing a problem with the openvpn connections using merlinWRT as the openvpn server, and I'd like to know how to fix this.

I'm using AC88u with latest 384.19 and the issue is that VPN becomes extremely slow (down to like 50k/s instead of 8MB/s) after using it for a while to transfer large files. I mean, it does work of for half an hour or so, but then it becomes miserable until I reboot the router.

I don't remember this happening with the previous versions, although I cannot be sure of when this started.

The only thing I can see in the logs is thousands of lines with this info:
Sep 6 18:10:34 ovpn-server1[1318]: username/A.B.C.D:6916 AEAD Decrypt error: bad packet ID (may be a replay): [ #1112770 ] -- see the man page entry for --no-replay and --replay-window for more info or silence this warning with --mute-replay-warnings Sep 6 18:10:34 ovpn-server1[1318]: username/A.B.C.D:6916 AEAD Decrypt error: bad packet ID (may be a replay): [ #1112771 ] -- see the man page entry for --no-replay and --replay-window for more info or silence this warning with --mute-replay-warnings Sep 6 18:10:34 ovpn-server1[1318]: username/A.B.C.D:6916 AEAD Decrypt error: bad packet ID (may be a replay): [ #1112772 ] -- see the man page entry for --no-replay and --replay-window for more info or silence this warning with --mute-replay-warnings Sep 6 18:10:34 ovpn-server1[1318]: username/A.B.C.D:6916 AEAD Decrypt error: bad packet ID (may be a replay): [ #1112773 ] -- see the man page entry for --no-replay and --replay-window for more info or silence this warning with --mute-replay-warnings Sep 6 18:10:34 ovpn-server1[1318]: username/A.B.C.D:6916 AEAD Decrypt error: bad packet ID (may be a replay): [ #1112774 ] -- see the man page entry for --no-replay and --replay-window for more info or silence this warning with --mute-replay-warnings Sep 6 18:10:34 ovpn-server1[1318]: username/A.B.C.D:6916 AEAD Decrypt error: bad packet ID (may be a replay): [ #1112775 ] -- see the man page entry for --no-replay and --replay-window for more info or silence this warning with --mute-replay-warnings Sep 6 18:10:34 ovpn-server1[1318]: username/A.B.C.D:6916 AEAD Decrypt error: bad packet ID (may be a replay): [ #1112776 ] -- see the man page entry for --no-replay and --replay-window for more info or silence this warning with --mute-replay-warnings Sep 6 18:10:34 ovpn-server1[1318]: username/A.B.C.D:6916 AEAD Decrypt error: bad packet ID (may be a replay): [ #1112777 ] -- see the man page entry for --no-replay and --replay-window for more info or silence this warning with --mute-replay-warnings

And here's the relevant part in the setup UI:

View attachment 26076

Any help with this would be really appreciated.

Thanks

'Username/Password Auth. Only=YES' is not advised as you greatly reduce the security of the OpenVPN Server by allowing hackers to simply guess the password without the need for the OpenVPN PKI certificate.

I suggest you change it immediately to 'Username/Password Auth. Only=No'

 

[MaziahBebop]

Hello all,

I wonder if anyone's noticing a problem with the openvpn connections using merlinWRT as the openvpn server, and I'd like to know how to fix this.




Any help with this would be really appreciated.

Thanks

Regarding replay warnings, I tried to resolve that issue myself and found only 2 options that would resolve it (given more time for research maybe more options would have been uncovered) :

First was config - - mute-replay-warnings, second was use TCP tunnel. I went with option 2 after testing no issues with performance (not usually recommended but it worked for me so went with it).

Regarding security, using a non standard port is good (e.g. 11944), prevents a bunch of malicious bot activity on the port 1194. And enable TLS control channel security "encrypted channel", which for me, stopped log messages appearing from random apparently malicious IPs.

Regarding speed issues and rebooting, temperature hasn't been a common issue with these routers, but it could be worth looking at if your workloads are high enough.

 

[elorimer]

I would try a few things. First, I would disable compression on both sides. Second, is there a need for client<>client connections and to manage client-specific options?

Then, switch to TCP and see if that makes a difference. If not, it may be a problem with the MTU setting in UDP you are using.

But first, do what @Martineau said. This error message could indicate an attack.

 

[Phk]

Hello guys,

First of all thank you for your very fast attention on this. This is sign of a healthy community supporting this project

Let me iterate over the several points mentioned:

1. Default port -- the screenshot was "sanitized", I use a (very) high port
2. Use user\pass only -- thanks for the tip, I didn't realize that this removed the TLS auth phase, which I now corrected and applied for both inbound and outbound. Anyway I will still need using user\pass because of a legacy openvpn client in a remote machine
3. TCP for packet queuing -- I've chosen UDP for the minor speed upgrade and lighter in-mem ip stack footprint. Switched now to TCP and don't find any difference, except for that bloody warning disappearance great.
4. Client isolation -- Correct. Thanks for the tip, disabled.
5. Compression -- this is probably **the issue**. Let me explain, a while ago I saw some benchmarks comparing performance in LZO\LZOAdap and Lz4. I found LZ4 to be more performant, and since Ac88u has enough CPU and the old legacy client has LZ4 support, I've changed from LZO Adaptative to that. I've now fallen back to LZO Adaptative, and will see if the issue remains.

I noticed another log entry (or thousand of entries like this one):
AEAD Decrypt error: cipher final failed

This doesn't clearly state that the problem is in the compression algorithm, and can be a sign of corruption in the transmission (even more when being UDP). But i'd like to know if someone has used LZ4 and transferred like 30GB using a similar hardware\firmware version. Anyone?

I'll reproduce the steps as soon as I can and post here some result on VPN stability.

Cheers
Phk

 

[elorimer]

Progress, yay!

2. It seems to be okay to use user/Pass authentication as long as the cert is also in the mix.
3. This suggests the MSS fix might work for you to get back to UDP. More importantly, did switching to TCP fix your throughput issue?
5. Compression is a security hole, and anecdotally it may not do much because much traffic (video, RDP) is already compressed.

I'm surprised with the cipher final failed message a connection is being made. But it suggests that a client doesn't have your ciphers. and no encryption is being negotiated. Perhaps that legacy client is not 2.4 or higher? Maybe change the NCP from "Enabled" to "enabled with fallback"

 

[MaziahBebop]

VPN becomes extremely slow (down to like 50k/s instead of 8MB/s) after using it for a while to transfer large files. I mean, it does work of for half an hour or so, but then it becomes miserable until I reboot the router.

Looks like a potential scenario where overheating the CPU could come into play. I mean, 8MB/s sustained for an hour, of VPN traffic, with compression on, as well as the router doing other router stuff. Depending on the age of the hardware, the environmental conditions, the sustained workload... Can you rule out overheating by checking the temp, next time the 50k/s thing happens?

 

[Phk]

Hello everyone, sorry it took a while to be able to reproduce de exact same scenario, but now I have everything ready and I can test it whenever I want, so, now I can try to find the problem.

3. This suggests the MSS fix might work for you to get back to UDP. More importantly, did switching to TCP fix your throughput issue?

So, I've switched to TCP and compression to LZO-Adap. Result:
1. 1,5mb/s cap, never really got into more than 10% available bandwidth
2. VPN crashed totally after 20 or 30 minutes transfering and I got a top of logs, every 10 seconds, with this error:
write TCPv6_SERVER: Broken pipe (code=32)

I now turned off compression totally and retrying the transfer.

 

[Phk]

Ok guys, so, everything is stable now, unfortunately the bandwidth is still about 20%.

I've checked the temperature and it's always around 87ºC, in the top command, the openvpn server takes around 25% of one core.

The speed is always capped around 700kb/s when without VPN using a direct IP connection is around 4mb/s.

I don't know what else to "optimize" here, since I can't find an issue in the logs now :|

I'm using as a client a standard win10 machine with openVPN 2.4.9 installed.

Any further tips here would be really appreciated.

Thanks

 

[eibgrad]

write TCPv6_SERVER: Broken pipe (code=32)

Why is it reporting an IPv6 error and not IPv4 error?!

 

[Phk]

Why is it reporting an IPv6 error and not IPv4 error?!

Good question, I noticed that also, TBH the only thing I found about IPv6 was the IPv6 Firewall, which I now disabled.

 

[RMerlin]

TCP6 and UDP6 are dual stack, it supports both IPv4 and IPv6.

BTW, don't use compression. It's vulnerable to security exploits, and is considered deprecated by the OpenVPN devs, and has a high change of being removed in 2.6. You should also be using AES-*-GCM for best performance.

 

[Phk]

TCP6 and UDP6 are dual stack, it supports both IPv4 and IPv6.

BTW, don't use compression. It's vulnerable to security exploits, and is considered deprecated by the OpenVPN devs, and has a high change of being removed in 2.6. You should also be using AES-*-GCM for best performance.

Hi Merlin, thanks for the tip. Not considering it, i have set LZO to "no" in the last test above.
Do you think it's a normal overhead? To drop from 4.5mb/s to 0.7mb/s? Something is not making sense to me.
The router is not overheating, I have no issues in the logs.. :| Will try some minor changes but I'm not sure what's the issue here...

 

[MaziahBebop]

The router is not overheating

Sorry to be harping on about temperature every time I post in this thread, just for my own satisfaction I hope someone here can correct my thoughts on this:
My assumptions were that the Broadcom chip in the ac88u was Cortext A9 based, with a rated range up to 85°C.
87°C in the screen shot is near the max this device will allow before it mitigates by throttling back processing.
The RasPi I'm more familiar with, starts throttling at 80°C, and it's a different Cortex (a53) but if not 90°C then how hot can you get the ac88u before it throttles?

I'd wager $1, that a good fan under your router would solve this. Get your temp down to low 70's with a fan and if the slowdown persists, you're $1 richer at the end of the day.

 

[RMerlin]

Hi Merlin, thanks for the tip. Not considering it, i have set LZO to "no" in the last test above.
Do you think it's a normal overhead? To drop from 4.5mb/s to 0.7mb/s? Something is not making sense to me.
The router is not overheating, I have no issues in the logs.. :| Will try some minor changes but I'm not sure what's the issue here...

No idea, I never tested the performance of using an encrypted control channel in addition to the regular crypto.

My assumptions were that the Broadcom chip in the ac88u was Cortext A9 based, with a rated range up to 85°C.

The CPU is rated for a maximum of 120C if I recall correctly. Asus generally throttles at 100C.

 

[elorimer]

Try it with compression "disabled", not "none" (none means traffic is framed for compression but not used). Enable cipher negotiation and have the GCM ciphers first. And don't encrypt the control channel. See if that helps.

EDIT: Looking back, that looks close to where you started, except for TCP instead of UDP.

 

[eibgrad]

Any thought here given to the possibility it's the ISP that's the problem? That maybe the ISP is throttling the connection, esp. when most ISPs are asymmetric, and your download performance via the OpenVPN server is always limited by the upload side of your connection to the ISP. And unless you have a business account w/ the ISP, they don't take kindly to anyone (as they perceive it) abusing the system and trying to use a consumer account for business purposes.

Of course, it's always difficult to make these kinds of distinctions. But the red flag for me is this idea it slows down over time. According to the OP, as long as a *half hour* before things go south!

 

[Phk]

Alright, still have issues, but here's the debugging per suggestion:

@elorimer - I ruled out the compression to "none", still happens a cap <20%

@eibgrad - I tried 2 ISPs, and even tried using my 4G network. Same values (even in mobile, it's stuck at 650kb/s approximately).

@MaziahBebop - Np for keeping your hintch. I went down to my basement to get a fan, my temperature dropped from 87 to 65ºC.
I cron'd:
logger "Temperature: "`cat /proc/dmu/temperature | grep -Eo '[0-9]{1,4}'`"C"
to see temperature rising\dropping.
Starting the VPN test raised the temperature from 65 to 66 approximately, no more. So I guess we can clear that out too.

@RMerlin - I'm not encrypting the control channel now. Only authing both sides. However it's funny you mentioned that. about 5 minutes after the file transfer started, I'm having this log entry every 5 seconds:
client/81.xxx.142.31:50003 TLS Error: local/remote TLS keys are out of sync: [AF_INET6]::ffff:81.xxx.142.31:50003 [3]
The transfer was still capped at 650kb/s, and I noticed that it's not constant. It comes in "windows" of 1MB length. Even in UDP.


Here's my latest conf screen:

Next test: will try without any TLS control just to rule that out too.

Thank guys

Phk

 

[elorimer]

You are back to UDP. Are you still getting those bad decrypt errors?

Me, I'd print out the pages of settings I need to remember, factory reset nuclear wise and start all over.

 

[Phk]

You are back to UDP. Are you still getting those bad decrypt errors?

Nop, no errors this time with this configuration.

I've changed to:

• no control channel encryption,
• AES-128-GCM forced encryption.

Results are:

• Little improvement from 650kb/s to 900kb/s (but still bad since its 20% versus a 4.5mb/s in direct transfer using the same ISP\machine scenario)
• No warnings anywhere
• Temperature all-time-high at 89C now
• WAN gets really really slow even for cabled clients....! and it's only at 10% use!

I wonder if someone else can test OpenVPN with this hardware\firmware\configuration. It may need a factory reset, it's true, but I'd do that if someone tells me it's working OK for them with this setup.
Now that I see the other clients hanging to access WAN, even though it's a symmetric 100mbit line, I'm once more thinking this is a firmware bug. Just tell me how to debug that.

**EDIT** I'm posting here the configuration screenshot so that anyone can reproduce the same scenario: