ASUS RT-AC86U (Merlin) does not connect to OpenVPN (NordVPN)

[Nosha]

Hi,

I have recently bought Asus RT-AC86U as I was very impressed with its hardware specs. My main aim is to set this up as a dedicated VPN router (connect to NordVPN using OpenVPN client). I previously had N66U working for this purpose but due to hardware limitation the speed just isn't there when using VPN.

I configured the stock build and connected the OpenVPN client on AC86U without any issues. But as the stock firmware doesn't give much control (like no kill switch), I updated the firmware to Merlin's latest build yesterday. Since upgrading to Merlin, I am unable to connect to VPN. I have tried both UDP and TCP and tried connecting to many different NordVPN servers but it just doesn't work. The status stays on "connecting..." and nothing happens. Looking at the logs, here is what I got:
Jul 23 19:25:17 ovpn-client2[17324]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Jul 23 19:25:17 ovpn-client2[17324]: TCP/UDP: Preserving recently used remote address: [AF_INET]89.238.191.205:1194
Jul 23 19:25:17 ovpn-client2[17324]: Socket Buffers: R=[524288->524288] S=[524288->524288]
Jul 23 19:25:17 ovpn-client2[17324]: UDP link local: (not bound)
Jul 23 19:25:17 ovpn-client2[17324]: UDP link remote: [AF_INET]89.238.191.205:1194
Jul 23 19:26:17 ovpn-client2[17324]: TLS Error: TLS key negotiation failed to occur within 60 seconds (check your network connectivity)
Jul 23 19:26:17 ovpn-client2[17324]: TLS Error: TLS handshake failed
Jul 23 19:26:17 ovpn-client2[17324]: SIGUSR1[soft,tls-error] received, process restarting
Jul 23 19:26:17 ovpn-client2[17324]: Restart pause, 5 second(s)

and for TCP, here is what I get:
Jul 23 19:08:58 ovpn-client2[12337]: TCP connection established with [AF_INET]81.92.203.52:443
Jul 23 19:08:58 ovpn-client2[12337]: TCP_CLIENT link local: (not bound)
Jul 23 19:08:58 ovpn-client2[12337]: TCP_CLIENT link remote: [AF_INET]81.92.203.52:443
Jul 23 19:08:58 ovpn-client2[12337]: Connection reset, restarting [0]
Jul 23 19:08:58 ovpn-client2[12337]: SIGUSR1[soft,connection-reset] received, process restarting
Jul 23 19:08:58 ovpn-client2[12337]: Restart pause, 40 second(s)

After spending hours with NordVPN support team and unable to fix problem, I went back to stock firmware the OpenVPN connected first time.

Am I missing anything? Can someone help please?

Regards,
Nosha

 

[ColinTaylor]

https://www.snbforums.com/threads/nordvpn-problems-with-384-5-and-ac68u.46643/

 

[Nosha]

Hi,
Thanks for you reply. I have already tried many many different servers but nothing worked. Please note that I am using AC86U (not AC68U). At least about 20 times I have seen the recommended server and tried to change the server to connect to the new one and that too with UDP as well as TCP but nothing worked. Stock firmware connects without any issues but not Merlin.
Regards,
Usman

 

[Darryl]

I just connected successfully to two Nord VPN Servers without any difficulty. I can't speak for TCP but UDP is working. Looking at your log for UDP it seems that TLS is failing. When you try again look at your crypto settings in the GUI. Under TLS control channel security I have Outgoing Auth (1) and under Auth Digest I have SHA512. If you don't have these settings try with them. If not we can compare configurations as there are a few settings which can cause the connection to fail. Merlin is just light years ahead of stock with Open VPN. Policy based routing alone is worth it.

 

[Nosha]

I just connected successfully to two Nord VPN Servers without any difficulty. I can't speak for TCP but UDP is working. Looking at your log for UDP it seems that TLS is failing. When you try again look at your crypto settings in the GUI. Under TLS control channel security I have Outgoing Auth (1) and under Auth Digest I have SHA512. If you don't have these settings try with them. If not we can compare configurations as there are a few settings which can cause the connection to fail. Merlin is just light years ahead of stock with Open VPN. Policy based routing alone is worth it.

Thanks a lot Darryl. I'll try this today and will update you. As per NordVPN's tutorial, I was selecting default instead of SHA512 but then Nord support asked me to turn of cypto altogether. I'll try with Outgoing Auth (1) and SHA512 when I get home today.

 

[Darryl]

Okay. As well as the above check your Keys and Certificates. Pressing the edit button will bring up boxes for the various keys and certificates. In my configuration there are entries only in the first two boxes, which are static key and ca. The static key is as I understand it required for tls. Good luck.

 

[Nosha]

Okay. As well as the above check your Keys and Certificates. Pressing the edit button will bring up boxes for the various keys and certificates. In my configuration there are entries only in the first two boxes, which are static key and ca. The static key is as I understand it required for tls. Good luck.

Thanks. Yes, I did check the certificates and also tried pasting them from the ca and tls files from NordVPN but to no avail. And yes, I also had only the first two boxed filled so good to know that yours is the same. Thanks a lot for your help. I'll surely update you once I have tried your suggestions tonight.

 

[robahearts]

Jul 23 19:26:17 ovpn-client2[17324]: TLS Error: TLS key negotiation failed to occur within 60 seconds (check your network connectivity)
Jul 23 19:26:17 ovpn-client2[17324]: TLS Error: TLS handshake failed

I was having the same issue as you but with a different VPN provided. My issue was the Legacy/fallback cipher I was using was different to the one recommended by the opvn file I uploaded. To fix it, I downloaded the strong encryption from PIA and for Legacy/fallback cipher I used AES-256-CBC

GOOD LUCK.

 

[Nosha]

Hi,
Thanks a lot for all your help guys. I have managed to make it work. I had reverted back to the stock firmware and configured VPN on it. This time, when I upgraded to Merlin, I left the VPN configured on stock and even connected and guess what, after upgrade it still was working fine I did however then confirm the settings and SHA512 jumped up as I was selecting Default previously (as per NordVPN guidance). One thing to note is that Accept DNS needs to be set to Exclusive otherwise you get DNS leak. I have also managed to create the kill switch now so I guess I am all up and running. The router is brilliant and I am getting virtually no loss of speed with VPN

Thanks again !

 

[Darryl]

Congratulations. I agree you should be using the exclusive setting for DNS to avoid leaks. I'm not surprised that support couldn't get it sorted for you. In my experience most support from service providers is terrible when you get beyond the basics. Nord support is one of the better ones. It is good that they were prepared to spend all that time trying to help but the fact is that they should know what settings their own servers require and be able to assist with this sort of troubleshooting.

One caution. I seem to recall reading somewhere a while ago that some Nord servers still use SHA1. Hopefully not any more. But if you do come across one of their servers you can't connect to that is something to look at. Or better yet, select another one using SHA512.

 

[Nosha]

Congratulations. I agree you should be using the exclusive setting for DNS to avoid leaks. I'm not surprised that support couldn't get it sorted for you. In my experience most support from service providers is terrible when you get beyond the basics. Nord support is one of the better ones. It is good that they were prepared to spend all that time trying to help but the fact is that they should know what settings their own servers require and be able to assist with this sort of troubleshooting.

One caution. I seem to recall reading somewhere a while ago that some Nord servers still use SHA1. Hopefully not any more. But if you do come across one of their servers you can't connect to that is something to look at. Or better yet, select another one using SHA512.

Hi,
Thanks for the tip. I am planning to compiling a few screenshots and sending them to Nord support so they can refer to it should someone else need help. Their guide is from the older version and a few thing have changed since then. Also they do not list AC86U as their recommended router which I believe is wrong on their part. I'll make a case for it to appear on their list as well

 

[Marin]

Hi,
Thanks a lot for all your help guys. I have managed to make it work. I had reverted back to the stock firmware and configured VPN on it. This time, when I upgraded to Merlin, I left the VPN configured on stock and even connected and guess what, after upgrade it still was working fine I did however then confirm the settings and SHA512 jumped up as I was selecting Default previously (as per NordVPN guidance). One thing to note is that Accept DNS needs to be set to Exclusive otherwise you get DNS leak. I have also managed to create the kill switch now so I guess I am all up and running. The router is brilliant and I am getting virtually no loss of speed with VPN

Thanks again !

Hello,

Would you be willing to share your NordVPN configuration setup? I am having similar issues.

Thank you!

 

[Darryl]

Modified B&W screenshot of GUI for working config. Your certificates page should have entries for the first two items, Static Key and CA.

 

[Marin]

Modified B&W screenshot of GUI for working config. Your certificates page should have entries for the first two items, Static Key and CA.

Thank you! It is exactly what I have configured but my speeds are suffering quite a bit-not sure why. Will double check the configuration again. Appreciate your info!


Sent from my iPhone using Tapatalk

 

[Darryl]

Try some different servers. They can vary substantially between servers and even with the time of day. One good thing with Nord is they have plenty of servers to try.

Having said that, using this Nord configuration on an ASUS 88U does seem to have quite a bit of overhead. I just tried a little experiment. Running Speedtest gives:

Linux Desktop No VPN: Ping 11 Download 56.59 Upload 5.10
Linux Desktop VPN Client: Ping 15 Download 53.07 Upload 4.79
Linux Desktop Merlin VPN: Ping 17 Download 43.10 Upload 4.78

The Download Speed loss using the same Nord VPN server is 10 MBPS between using.the Merlin Client and using the Desktop client. My guess is that the Router finds this configuration much harder going than the Desktop and of course does not have near the same speed. I tried the test again with similar results. CPU use at the height of the test on the router Core 1 reached 90% and Core 2 20%. Memory use approached 100% at the height of the test.

 

[Marin]

Modified B&W screenshot of GUI for working config. Your certificates page should have entries for the first two items, Static Key and CA.

Thank you for sharing--still not getting the speeds I want in my RT-AC5300. I have researched most of the VPN servers in my area and have been able to test them with my app and....in my GT-AC5300. When I use the GT-AC5300, I get speeds at 98% of what the ISP offers. Once I switch to RT-AC5300 with Merlin's latest FW (384.6), speeds drop considerable (1/3 of ISP ones). I am sure this has to do with the difference in CPU's but I wish I could tweak the RT's settings somewhat so I could get good VPN speeds and at the same time take advantage of Merlin's FW able to use other programs such as AB-solution, Skynet, etc.

What scripts did you use in the custom config section of the VPN setup page? Did you keep the ones from Merlin's most recent FW release or did you use those from NordVPN's tutorial ? (they use an older Merlin FW 380.xx for their setup).

 

[Darryl]

I believe these entries come from the Nord configuration file imported when you setup the connection. My Custom Configuration entries are:

remote-random
tun-mtu 1500
tun-mtu-extra 32
mssfix 1450
ping 15
ping-restart 0
ping-timer-rem
explicit-exit-notify 3
remote-cert-tls server
pull
fast-io

 

[Marin]

Thank you very much!

 

[Nosha]

Hi,

Apologies for the delay in responding.

My RT-AC86U hardly drop about 1 mbps in speed when using VPN on merlin.

Here are my configuration screens:

Full custom configuration is:
remote-cert-tls server
remote-random
nobind
tun-mtu 1500
tun-mtu-extra 32
mssfix 1450
persist-key
persist-tun
ping-timer-rem
reneg-sec 0

#log /tmp/vpn.log

 

[Marin]

Hi,

Apologies for the delay in responding.

My RT-AC86U hardly drop about 1 mbps in speed when using VPN on merlin.

Here are my configuration screens:View attachment 13895 View attachment 13896 View attachment 13897
Full custom configuration is:
remote-cert-tls server
remote-random
nobind
tun-mtu 1500
tun-mtu-extra 32
mssfix 1450
persist-key
persist-tun
ping-timer-rem
reneg-sec 0

#log /tmp/vpn.log

Thank you so much for sharing!



Sent from my iPhone using Tapatalk