OVPN connection loss after TLS re-key

[cowst]

Crazy thought: would it be a lot of work to just release a version with openvpn 2.3 to prove that's the reason?
And for me to stay on that and have vpn working

@john9527?

I understand about the 2.3 client.

In my case I have the satellite receiver behind router vpn, and it goes always down, within 1 to 5 days.

The openvpn for Android is on my fire tv stick, again always on, and I never found it down.

 

[john9527]

Crazy thought: would it be a lot of work to just release a version with openvpn 2.3 to prove that's the reason?

I've thought about doing that.....unfortunately it isn't straight forward. In addition to the OpenVPN code, it would require backing out a bunch of the webui changes and changes to the router openvpn code that processes options that have been changed/deprecated.

But I do have another change for you (or others to try)....Make a
/jffs/scripts/openvpnclient1.postconf
script with the following contents.

#!/bin/sh
CONFIG=$1
source /usr/sbin/helper.sh

pc_delete "persist-key" $CONFIG

exit

 

[cowst]

trying this.

perhaps I can try an old version of the FW with 2.3 afterwards, just to put a stamp on the culprit
I am afraid the reason I updated at the time was the missing watchdog fix that restored connectivity after my stupid modem reboots.

I've thought about doing that.....unfortunately it isn't straight forward. In addition to the OpenVPN code, it would require backing out a bunch of the webui changes and changes to the router openvpn code that processes options that have been changed/deprecated.

But I do have another change for you (or others to try)....Make a
/jffs/scripts/openvpnclient1.postconf
script with the following contents.

#!/bin/sh
CONFIG=$1
source /usr/sbin/helper.sh

pc_delete "persist-key" $CONFIG

exit

 

[Room 7609]

I've been following along with this thread and crossing fingers that someone finds a solution.

Just thought I'd chime in now, because while I have the same symptoms with NordVPN random TLS reauthorization failures, I have different hardware and software, so I doubt this is an issue with your devices. I am running pfSense 2.3.4 on a mini-PC.

I am running OpenVPN 2.3.17, so I don't think your use of 2.4 is the issue.

 

[Room 7609]

I don't know whether this gives us a clue, but according to my own observation with NordVPN connection randomness using pfSense 2.3.4, here is the play-by-play.

-Each hour a TLS renegotiation occurs, and the log indicates that renegotiation completes. Or at least, it doesn't look different than previous renegotiations.

Sep 14 10:22:19 pf openvpn[21924]: TLS: tls_process: killed expiring key
Sep 14 10:22:21 pf openvpn[21924]: VERIFY OK: depth=1, C=PA, ST=PA, L=Panama, O=NordVPN, OU=NordVPN, CN=us723.nordvpn.com, name=NordVPN,emailAddress=cert@nordvpn.com
Sep 14 10:22:21 pf openvpn[21924]: Validating certificate key usage
Sep 14 10:22:21 pf openvpn[21924]: ++ Certificate has key usage  00a0, expects 00a0
Sep 14 10:22:21 pf openvpn[21924]: VERIFY KU OK
Sep 14 10:22:21 pf openvpn[21924]: Validating certificate extended key usage
Sep 14 10:22:21 pf openvpn[21924]: ++ Certificate has EKU (str) TLS Web Server Authentication, expects TLS Web Server Authentication
Sep 14 10:22:21 pf openvpn[21924]: VERIFY EKU OK
Sep 14 10:22:21 pf openvpn[21924]: VERIFY OK: depth=0, C=PA, ST=PA, L=Panama, O=NordVPN, OU=NordVPN, CN=us723.nordvpn.com, name=NordVPN, emailAddress=cert@nordvpn.com
Sep 14 10:22:23 pf openvpn[21924]: Data Channel Encrypt: Cipher 'AES-256-CBC' initialized with 256 bit key
Sep 14 10:22:23 pf openvpn[21924]: Data Channel Encrypt: Using 512 bit message hash 'SHA512' for HMAC authentication
Sep 14 10:22:23 pf openvpn[21924]: Data Channel Decrypt: Cipher 'AES-256-CBC' initialized with 256 bit key
Sep 14 10:22:23 pf openvpn[21924]: Data Channel Decrypt: Using 512 bit message hash 'SHA512' for HMAC authentication
Sep 14 10:22:23 pf openvpn[21924]: Control Channel: TLSv1.2, cipher TLSv1/SSLv3ECDHE-RSA-AES256-GCM-SHA384, 2048 bit RSA

-After renegotiation, the VPN client (my pfSense router) cannot ping the VPN server (NordVPN). Here I'm pinging us723.nordvpn.com:

$ ping 168.242.211.137
PING 168.242.211.137 (168.242.211.137): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1

-Client cannot ping the NordVPN gateway

$ ping 10.8.8.1
PING 10.8.8.1 (10.8.8.1): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1

-DNS lookups fail

$ nslookup www.google.com
;; connection timed out; no servers could be reached

-Despite all this, data continues to be sent from the server and received by the client. However, data sent by the client (which continually tries) doesn't seem to reach the server.

-Eventually, usually within 5-10 minutes, server stops sending traffic to client. I assume this is because the server has sent all it has been asked to send and, in the absence of receiving any new requests from the client since renegotiation, it has nothing to do. Client continues to attempt to send to server.

-Once enough time passes without receiving data from the server, a ping timeout occurs, causing OpenVPN to reset the connection

Sep 14 10:31:45 pf openvpn[21924]: [us723.nordvpn.com] Inactivity timeout (--ping-restart), restarting

-After reset, the connection once again operates normally.

 

[Jack Yaz]

I don't know whether this gives us a clue, but according to my own observation with NordVPN connection randomness using pfSense 2.3.4, here is the play-by-play.

-Each hour a TLS renegotiation occurs, and the log indicates that renegotiation completes. Or at least, it doesn't look different than previous renegotiations.

Sep 14 10:22:19 pf openvpn[21924]: TLS: tls_process: killed expiring key
Sep 14 10:22:21 pf openvpn[21924]: VERIFY OK: depth=1, C=PA, ST=PA, L=Panama, O=NordVPN, OU=NordVPN, CN=us723.nordvpn.com, name=NordVPN,emailAddress=cert@nordvpn.com
Sep 14 10:22:21 pf openvpn[21924]: Validating certificate key usage
Sep 14 10:22:21 pf openvpn[21924]: ++ Certificate has key usage  00a0, expects 00a0
Sep 14 10:22:21 pf openvpn[21924]: VERIFY KU OK
Sep 14 10:22:21 pf openvpn[21924]: Validating certificate extended key usage
Sep 14 10:22:21 pf openvpn[21924]: ++ Certificate has EKU (str) TLS Web Server Authentication, expects TLS Web Server Authentication
Sep 14 10:22:21 pf openvpn[21924]: VERIFY EKU OK
Sep 14 10:22:21 pf openvpn[21924]: VERIFY OK: depth=0, C=PA, ST=PA, L=Panama, O=NordVPN, OU=NordVPN, CN=us723.nordvpn.com, name=NordVPN, emailAddress=cert@nordvpn.com
Sep 14 10:22:23 pf openvpn[21924]: Data Channel Encrypt: Cipher 'AES-256-CBC' initialized with 256 bit key
Sep 14 10:22:23 pf openvpn[21924]: Data Channel Encrypt: Using 512 bit message hash 'SHA512' for HMAC authentication
Sep 14 10:22:23 pf openvpn[21924]: Data Channel Decrypt: Cipher 'AES-256-CBC' initialized with 256 bit key
Sep 14 10:22:23 pf openvpn[21924]: Data Channel Decrypt: Using 512 bit message hash 'SHA512' for HMAC authentication
Sep 14 10:22:23 pf openvpn[21924]: Control Channel: TLSv1.2, cipher TLSv1/SSLv3ECDHE-RSA-AES256-GCM-SHA384, 2048 bit RSA

-After renegotiation, the VPN client (my pfSense router) cannot ping the VPN server (NordVPN). Here I'm pinging us723.nordvpn.com:

$ ping 168.242.211.137
PING 168.242.211.137 (168.242.211.137): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1

-Client cannot ping the NordVPN gateway

$ ping 10.8.8.1
PING 10.8.8.1 (10.8.8.1): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1

-DNS lookups fail

$ nslookup www.google.com
;; connection timed out; no servers could be reached

-Despite all this, data continues to be sent from the server and received by the client. However, data sent by the client (which continually tries) doesn't seem to reach the server.

-Eventually, usually within 5-10 minutes, server stops sending traffic to client. I assume this is because the server has sent all it has been asked to send and, in the absence of receiving any new requests from the client since renegotiation, it has nothing to do. Client continues to attempt to send to server.

-Once enough time passes without receiving data from the server, a ping timeout occurs, causing OpenVPN to reset the connection

Sep 14 10:31:45 pf openvpn[21924]: [us723.nordvpn.com] Inactivity timeout (--ping-restart), restarting

-After reset, the connection once again operates normally.

May be worth passing this to NordVPN's support as well? Since this is affecting 2 different implementations of OpenVPN (Merlin and pfSense), it suggests to me that they haven't got their server configuration quite right.

 

[Galaxysurfer]

May be worth passing this to NordVPN's support as well? Since this is affecting 2 different implementations of OpenVPN (Merlin and pfSense), it suggests to me that they haven't got their server configuration quite right.

Has anyone found a solution to our connection issues with Nordvpn other than changing vpn provider? This is annoying having to reboot router to get connection back. I have my network set to disconnect if i lose vpn. Havent heard anything back from nordvpn tech support. The silence is deafening.

 

[Room 7609]

Earlier I observed that right after a TLS renegotiation, sometimes traffic flows only one-way, from NordVPN to client. My suspicion is that in these instances, NordVPN is dropping all packets sent by the client.

The reason I suspect this is that netstat shows all Send-Q's to the VPN are clear and that connections from client to server are in SYN_SENT status. So it looks like my router is successfully sending out packets to establish connections to internet servers (websites etc) but is not receiving any responses back. In the following table, 10.8.8.43 is the IP address assigned to me by NordVPN. For example:

[2.3.4-RELEASE][admin@pf.localdomain]/root: netstat -n4
Active Internet connections
Proto Recv-Q Send-Q Local Address          Foreign Address        (state)
tcp4       0      0 10.8.8.43.7192         84.53.139.129.53       SYN_SENT
tcp4       0      0 10.8.8.43.1279         23.61.199.131.53       SYN_SENT
tcp4       0      0 10.8.8.43.64480        95.100.168.130.53      SYN_SENT
tcp4       0      0 10.8.8.43.5591         96.7.49.129.53         SYN_SENT
tcp4       0      0 10.8.8.43.52403        23.211.61.130.53       SYN_SENT
tcp4       0      0 10.8.8.43.12155        23.211.132.131.53      SYN_SENT

Likewise, the firewall state table shows many connections from VPN client to server are in a state of one initial packet sent, but none received.

 

[cowst]

Maybe you got it.
Several days without disconnection.
Or at least without the permanent vpn client down I used to get at some point.

I think it would be nice if someone else currently having the issue would try the suggestion at the bottom.

I've thought about doing that.....unfortunately it isn't straight forward. In addition to the OpenVPN code, it would require backing out a bunch of the webui changes and changes to the router openvpn code that processes options that have been changed/deprecated.

But I do have another change for you (or others to try)....Make a
/jffs/scripts/openvpnclient1.postconf
script with the following contents.

#!/bin/sh
CONFIG=$1
source /usr/sbin/helper.sh

pc_delete "persist-key" $CONFIG

exit

 

[Room 7609]

I have had not made any changes, but likewise have not seen any connection issues in the past 48 hours. Either they've made some changes to their servers, or I'm just getting lucky. I'm hoping it is the former.

Maybe you got it.
Several days without disconnection.
Or at least without the permanent vpn client down I used to get at some point.

I think it would be nice if someone else currently having the issue would try the suggestion at the bottom.

 

[Room 7609]

It's now been a week since I've seen any issues. Nothing has changed on my end. I'm connecting to their us723 server. How's everyone else looking?

 

[Swazz]

Cowst and I had the same issue. I referenced his ticket # in my communication with NordVPN. Said they were aware of and still looking into the issue a few weeks ago. Like you, I am not experiencing any issues. Connecting with a different server as well which points to them having this issue resolved. Fingers crossed.

It's now been a week since I've seen any issues. Nothing has changed on my end. I'm connecting to their us723 server. How's everyone else looking?

 

[Room 7609]

It has now been two weeks, and in that period there has only been one outage, on the 17th at precisely 2am Pacific. It was a different sort of outage though, as restarts failed with "TLS key negotiation failed to occur within 60 seconds" for about 5 minutes. Then all was normal again. I'm going to assume that hiccup was unrelated.

In any case, I've never seen a two week period go by with out the periodic ping-restart style failures described earlier in this thread. So I'm going to assume that this issue has been addressed.

 

[snourbs]

I had been experiencing the same issues, and it has been noticeably more stable here too lately (without making any changes on my end).

 

[Andaman]

Looks like NordVPN fixed this. (?)

I've not posted on this issue for months because I have not had access to the Merlin router... It's at a remote vacation home we only visit occasionally. But for the last 48+ hours I have been on-site here and I can report that I'm not seeing any disconnects due to this issue. Haven't changed any settings in my configuration. I think this issue may have been fixed in the NordVPN network. Knock on wood!

 

[cowst]

Looks like NordVPN fixed this. (?)

I've not posted on this issue for months because I have not had access to the Merlin router... It's at a remote vacation home we only visit occasionally. But for the last 48+ hours I have been on-site here and I can report that I'm not seeing any disconnects due to this issue. Haven't changed any settings in my configuration. I think this issue may have been fixed in the NordVPN network. Knock on wood!

I believe so.
Sad that we don't know what