OpenVPN TLS handshake failed

[KrisS]

I really like the custom firmware, however I can't make VPN function, although it worked fine with the OEM firmware.

The log:
Nov 19 17:57:00 openvpn[23552]: 192.168.0.225:58746 TLS: Initial packet from [AF_INET]192.168.0.225:58746, sid=5f54a7a3 58683605
Nov 19 17:57:09 openvpn[23552]: 192.168.0.225:56339 TLS: Initial packet from [AF_INET]192.168.0.225:56339, sid=f017a422 9cdd808e
Nov 19 17:57:19 openvpn[23552]: 192.168.0.225:50077 TLS: Initial packet from [AF_INET]192.168.0.225:50077, sid=0ae1971e c0e07cf1
Nov 19 17:57:29 openvpn[23552]: 192.168.0.225:64500 TLS: Initial packet from [AF_INET]192.168.0.225:64500, sid=8c70558b 59819df4
Nov 19 17:57:39 openvpn[23552]: 192.168.0.225:62171 TLS: Initial packet from [AF_INET]192.168.0.225:62171, sid=45c0e499 37ce9b71
Nov 19 17:57:49 openvpn[23552]: 192.168.0.225:61655 TLS: Initial packet from [AF_INET]192.168.0.225:61655, sid=11a909b0 9c146aeb
Nov 19 17:58:00 openvpn[23552]: 192.168.0.225:58746 TLS Error: TLS key negotiation failed to occur within 60 seconds (check your network connectivity)
Nov 19 17:58:00 openvpn[23552]: 192.168.0.225:58746 TLS Error: TLS handshake failed
Nov 19 17:58:00 openvpn[23552]: 192.168.0.225:58746 SIGUSR1[soft,tls-error] received, client-instance restarting
Nov 19 17:58:09 openvpn[23552]: 192.168.0.225:56339 TLS Error: TLS key negotiation failed to occur within 60 seconds (check your network connectivity)
Nov 19 17:58:09 openvpn[23552]: 192.168.0.225:56339 TLS Error: TLS handshake failed
Nov 19 17:58:09 openvpn[23552]: 192.168.0.225:56339 SIGUSR1[soft,tls-error] received, client-instance restarting
Nov 19 17:58:19 openvpn[23552]: 192.168.0.225:50077 TLS Error: TLS key negotiation failed to occur within 60 seconds (check your network connectivity)
Nov 19 17:58:19 openvpn[23552]: 192.168.0.225:50077 TLS Error: TLS handshake failed
Nov 19 17:58:19 openvpn[23552]: 192.168.0.225:50077 SIGUSR1[soft,tls-error] received, client-instance restarting

Some Detail:

Nov 19 17:40:28 openvpn[20552]: MULTI: multi_create_instance called
Nov 19 17:40:28 openvpn[20552]: 192.168.0.225:52280 Re-using SSL/TLS context
Nov 19 17:40:28 openvpn[20552]: 192.168.0.225:52280 LZO compression initialized
Nov 19 17:40:28 openvpn[20552]: 192.168.0.225:52280 Control Channel MTU parms [ L:1558 D:1212 EF:38 EB:0 ET:0 EL:3 ]
Nov 19 17:40:28 openvpn[20552]: 192.168.0.225:52280 Data Channel MTU parms [ L:1558 D:1450 EF:58 EB:143 ET:0 EL:3 AF:3/1 ]
Nov 19 17:40:28 openvpn[20552]: 192.168.0.225:52280 Local Options String: 'V4,dev-type tun,link-mtu 1558,tun-mtu 1500,proto UDPv4,comp-lzo,cipher AES-256-CBC,auth SHA1,keysize 256,key-method 2,tls-server'
Nov 19 17:40:28 openvpn[20552]: 192.168.0.225:52280 Expected Remote Options String: 'V4,dev-type tun,link-mtu 1558,tun-mtu 1500,proto UDPv4,comp-lzo,cipher AES-256-CBC,auth SHA1,keysize 256,key-method 2,tls-client'
Nov 19 17:40:28 openvpn[20552]: 192.168.0.225:52280 Local Options hash (VER=V4): 'a8f55717'
Nov 19 17:40:28 openvpn[20552]: 192.168.0.225:52280 Expected Remote Options hash (VER=V4): '22188c5b'
Nov 19 17:40:28 openvpn[20552]: 192.168.0.225:52280 TLS: Initial packet from [AF_INET]192.168.0.225:52280, sid=eb0df4ce d55605bc
Nov 19 17:40:37 openvpn[20552]: MULTI: multi_create_instance called
Nov 19 17:40:37 openvpn[20552]: 192.168.0.225:55491 Re-using SSL/TLS context
Nov 19 17:40:37 openvpn[20552]: 192.168.0.225:55491 LZO compression initialized
Nov 19 17:40:37 openvpn[20552]: 192.168.0.225:55491 Control Channel MTU parms [ L:1558 D:1212 EF:38 EB:0 ET:0 EL:3 ]
Nov 19 17:40:37 openvpn[20552]: 192.168.0.225:55491 Data Channel MTU parms [ L:1558 D:1450 EF:58 EB:143 ET:0 EL:3 AF:3/1 ]
Nov 19 17:40:37 openvpn[20552]: 192.168.0.225:55491 Local Options String: 'V4,dev-type tun,link-mtu 1558,tun-mtu 1500,proto UDPv4,comp-lzo,cipher AES-256-CBC,auth SHA1,keysize 256,key-method 2,tls-server'
Nov 19 17:40:37 openvpn[20552]: 192.168.0.225:55491 Expected Remote Options String: 'V4,dev-type tun,link-mtu 1558,tun-mtu 1500,proto UDPv4,comp-lzo,cipher AES-256-CBC,auth SHA1,keysize 256,key-method 2,tls-client'

My settings:

 

[octopus]

Set your "Extra HMAC Authorization" to "0"
Set your "Auth Digest" to SHA1
and you may have to change "LZO compression" to "Enabled"

Good luck!

 

[KrisS]

Thanks for the input! I was excited for an end to this, but now I get:

Nov 20 09:58:13 openvpn[29378]: 192.168.0.225:52171 TLS: Initial packet from [AF_INET]192.168.0.225:52171, sid=d82a9e7b 02453bce
Nov 20 09:58:15 openvpn[29378]: 192.168.0.225:52171 Authenticate/Decrypt packet error: bad packet ID (may be a replay): [ #1 / time = (1479657493) Sun Nov 20 09:58:13 2016 ] -- see the man page entry for --no-replay and --replay-window for more info or silence this warning with --mute-replay-warnings
Nov 20 09:58:15 openvpn[29378]: 192.168.0.225:52171 TLS Error: incoming packet authentication failed from [AF_INET]192.168.0.225:52171
Nov 20 09:58:17 openvpn[29378]: 192.168.0.225:52171 Authenticate/Decrypt packet error: bad packet ID (may be a replay): [ #1 / time = (1479657493) Sun Nov 20 09:58:13 2016 ] -- see the man page entry for --no-replay and --replay-window for more info or silence this warning with --mute-replay-warnings
Nov 20 09:58:17 openvpn[29378]: 192.168.0.225:52171 TLS Error: incoming packet authentication failed from [AF_INET]192.168.0.225:52171
Nov 20 09:58:19 openvpn[29378]: 192.168.0.225:52171 Authenticate/Decrypt packet error: bad packet ID (may be a replay): [ #1 / time = (1479657493) Sun Nov 20 09:58:13 2016 ] -- see the man page entry for --no-replay and --replay-window for more info or silence this warning with --mute-replay-warnings

Settings:

 

[octopus]

What do you have in Custom Configuration?
What do you have your router connected to?

 

[KrisS]

What do you have in Custom Configuration?
What do you have your router connected to?

Nothing, as I had nothing in the OEM custom config.

Router is connected to FIOS, just as OEM was.

I tried HMAC bidirectional as well: connect fail also

 

[KrisS]

I found this post: http://www.snbforums.com/threads/how-to-implement-secure-vpn-authentication.15796/

I changed UDP to TCP and it connects! Strange...

I do appreciate your assistance.

Now working as:

 

[octopus]

Do you try connecting inside your lan or from wan side?
Anything from connected client log?
UDP/TCP must be same on both side.

 

[KrisS]

OMG, yes, I now realize I was on LAN. Stupid... Sorry...

But now it works on LAN or cloud.

Changed Auth Digest to SHA256 for better security.