(Not specifically) VPNMON-R3 1.11 failure domino effect

[Ripshod]

I'll tag @Viktor Jaep to this as it's in their interest.
Had some really wierd events this morning. All my IoT devices were offline, which was caused by my vpn failing to connect. This got me concerned about my VPS as I host my own VPN, but the VPS and VPN server were up and running fine (confirmed with a connection from my phone).
My VPN client on the router was down and contantly connecting but never actually making a connection.
So via ssh I loaded VPNMON and that was detecting my WAN was down, which it wasn't so suspecting a problem with the default 8.8.8.8 dns server I tested with 1.1.1.1, 9.9.9.9 and 208.67.222.222 and everything sorted itself out -even going back to 8.8.8.8 was fine. However, my vpn client wouldn't reconnect. After disabling the vpn monitor in VPNMON and reimporting the config file and the client was stable again. Went back to VPNMON and re-enabled monitoring of the VPN slot and everything was fine again.
So, should VPNMON mess up my VPN configuration in that way?

From VPNMON log:

Jan 26 2024 09:36:24 ripshod VPNMON-R3[13487] - ERROR: WAN Connectivity Issue Detected
Jan 26 2024 09:36:30 ripshod VPNMON-R3[13487] - WARNING:  WAN Link Detected -- Trying to reconnect/Reset VPN
Jan 26 2024 09:37:39 ripshod VPNMON-R3[14214] - WARNING: VPN1 has disconnected
Jan 26 2024 09:37:46 ripshod VPNMON-R3[14214] - INFO: VPN1 Connection Restarted - New Server: Custom
Jan 26 2024 09:38:06 ripshod VPNMON-R3[14214] - INFO: VPN Director Routing Service Restarted
Jan 26 2024 09:42:36 ripshod VPNMON-R3[13487] - ERROR: WAN Connectivity Issue Detected
Jan 26 2024 09:42:42 ripshod VPNMON-R3[13487] - WARNING:  WAN Link Detected -- Trying to reconnect/Reset VPN
Jan 26 2024 09:43:14 ripshod VPNMON-R3[14214] - WARNING: VPN1 has disconnected
Jan 26 2024 09:43:21 ripshod VPNMON-R3[14214] - INFO: VPN1 Connection Restarted - New Server: Custom
Jan 26 2024 09:43:41 ripshod VPNMON-R3[14214] - INFO: VPN Director Routing Service Restarted

This looped constantly,

From Router log:

Jan 26 09:37:48 ripshod ovpn-client1[23088]: RESOLVE: Cannot resolve host address: Custom:31825 (Name or service not known)
Jan 26 09:37:48 ripshod ovpn-client1[23088]: RESOLVE: Cannot resolve host address: Custom:31825 (Name or service not known)
Jan 26 09:37:48 ripshod ovpn-client1[23088]: Could not determine IPv4/IPv6 protocol
Jan 26 09:37:48 ripshod ovpn-client1[23088]: SIGUSR1[soft,Could not determine IPv4/IPv6 protocol] received, process restarting
Jan 26 09:37:48 ripshod ovpn-client1[23088]: Restart pause, 1 second(s)
Jan 26 09:37:49 ripshod ovpn-client1[23088]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts

Again this continued to loop. IPv6 was disabled at this point.

This was constantly repeated even though the WAN was up.

Is there any way to grab a more detailed log?

EDIT: I've decided to go for a full reinstall - "it's about time" I hear you say.

 

[Viktor Jaep]

I'll tag @Viktor Jaep to this as it's in their interest.
Had some really wierd events this morning. All my IoT devices were offline, which was caused by my vpn failing to connect. This got me concerned about my VPS as I host my own VPN, but the VPS and VPN server were up and running fine (confirmed with a connection from my phone).
My VPN client on the router was down and contantly connecting but never actually making a connection.
So via ssh I loaded VPNMON and that was detecting my WAN was down, which it wasn't so suspecting a problem with the default 8.8.8.8 dns server I tested with 1.1.1.1, 9.9.9.9 and 208.67.222.222 and everything sorted itself out -even going back to 8.8.8.8 was fine. However, my vpn client wouldn't reconnect. After disabling the vpn monitor in VPNMON and reimporting the config file and the client was stable again. Went back to VPNMON and re-enabled monitoring of the VPN slot and everything was fine again.

Believe me, you're not the only one. I've had weird resolution issues, connectivity issues, etc... that could only be solved with a reboot. Albeit extremely infrequently, but I figure it's software... and software is software.

So, should VPNMON mess up my VPN configuration in that way?

VPNMON specifically does not touch your configuration. The only field(s) it touches are the VPN Client's "Server Address" (to change hostname/IP to a new server), and the description field (where it inserts "added by VPNMON-R3").

From VPNMON log:

Jan 26 2024 09:36:24 ripshod VPNMON-R3[13487] - ERROR: WAN Connectivity Issue Detected
Jan 26 2024 09:36:30 ripshod VPNMON-R3[13487] - WARNING:  WAN Link Detected -- Trying to reconnect/Reset VPN
Jan 26 2024 09:37:39 ripshod VPNMON-R3[14214] - WARNING: VPN1 has disconnected
Jan 26 2024 09:37:46 ripshod VPNMON-R3[14214] - INFO: VPN1 Connection Restarted - New Server: Custom
Jan 26 2024 09:38:06 ripshod VPNMON-R3[14214] - INFO: VPN Director Routing Service Restarted
Jan 26 2024 09:42:36 ripshod VPNMON-R3[13487] - ERROR: WAN Connectivity Issue Detected
Jan 26 2024 09:42:42 ripshod VPNMON-R3[13487] - WARNING:  WAN Link Detected -- Trying to reconnect/Reset VPN
Jan 26 2024 09:43:14 ripshod VPNMON-R3[14214] - WARNING: VPN1 has disconnected
Jan 26 2024 09:43:21 ripshod VPNMON-R3[14214] - INFO: VPN1 Connection Restarted - New Server: Custom
Jan 26 2024 09:43:41 ripshod VPNMON-R3[14214] - INFO: VPN Director Routing Service Restarted

This looped constantly,

View attachment 55967

So what you see here is that VPNMON was not able to validate a stable WAN connection. It does this by establishing a more complex HTTPS connection with 8.8.8.8. If it can't do that, then it assumes the WAN is down. Your screenshot shows that it got past the point where the router is indicating that the WAN connections are up (status code: 2), and is now in a 5 minute loop to allow the router/WAN to settle, before trying the HTTPS connection again in order to determine if the WAN is healthy enough to establish a VPN connection.

From Router log:

Jan 26 09:37:48 ripshod ovpn-client1[23088]: RESOLVE: Cannot resolve host address: Custom:31825 (Name or service not known)
Jan 26 09:37:48 ripshod ovpn-client1[23088]: RESOLVE: Cannot resolve host address: Custom:31825 (Name or service not known)
Jan 26 09:37:48 ripshod ovpn-client1[23088]: Could not determine IPv4/IPv6 protocol
Jan 26 09:37:48 ripshod ovpn-client1[23088]: SIGUSR1[soft,Could not determine IPv4/IPv6 protocol] received, process restarting
Jan 26 09:37:48 ripshod ovpn-client1[23088]: Restart pause, 1 second(s)
Jan 26 09:37:49 ripshod ovpn-client1[23088]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts

Again this continued to loop. IPv6 was disabled at this point.

This was constantly repeated even though the WAN was up.

Is there any way to grab a more detailed log?

EDIT: I've decided to go for a full reinstall - "it's about time" I hear you say.

When you are seeing these messages like "RESOLVE: Cannot resolve host address", to me it sounds like there was some catastrophic resolution error of some sort... which may have cascaded enough for the WAN connectivity check to not work either. The "could not determine IP4/IP6" sounds like a vpn config error. Not sure, but it does sounds something went kaput with OpenVPN. Unfortunately there's not any other logs I'm aware of other than what VPNMON puts out in combination with the syslog.

 

[Ripshod]

The reinstall improved nothing - same problem. It's soooo hard to get into VPNMON's configuration when this is happening, but I managed to get in and disable the WAN detection. Seems good for now so maybe something is happening on the WAN/DNS that isn't affecting normal browsing(?).

EDIT: synchronised posting

 

[Viktor Jaep]

The reinstall improved nothing - same problem. It's soooo hard to get into VPNMON's configuration when this is happening, but I managed to get in and disable the WAN detection. Seems good for now so maybe something is happening on the WAN/DNS that isn't affecting normal browsing(?).

EDIT: synchronised posting

From an SSH prompt, just type "vpnmon-r3 -setup". That should let you right in.

If it's still kicking back that it can't connect and reverts back to the 5 min timer, there's something else going on with WAN/DNS, or connectivity in general. Are you able to reach 8.8.8.8? Have you tried the reboot option yet?

 

[Ripshod]

I've gone the full reinstall with no change unfortunately.
I can ping 8.8.8.8 and the other dns servers just fine but I am using quad9 generally and there does appear to be some lag in domain resolution. Ping times are all over the place so I'm going to put the blame on my isp - even pings to their dns varies from 8 to 50+ ms.
I'm not going to look for any resolution until my isp answers some questions.
Disabling WAN detection will do for now.

EDIT: Pinging the same dns servers through my VPN gives much better times - definitely an ISP issue.

 

[Viktor Jaep]

I've gone the full reinstall with no change unfortunately.
I can ping 8.8.8.8 and the other dns servers just fine but I am using quad9 generally and there does appear to be some lag in domain resolution. Ping times are all over the place so I'm going to put the blame on my isp - even pings to their dns varies from 8 to 50+ ms.
I'm not going to look for any resolution until my isp answers some questions.
Disabling WAN detection will do for now.

EDIT: Pinging the same dns servers through my VPN gives much better times - definitely an ISP issue.

Let me know when you hear about any kind of resolution, or if it just "magically" all just starts working again (like it always does) lol

 

[Ripshod]

I have a resolution, and it's likely it's not what anyone expected.
After starting from scratch, adding scripts in intervals (took 4 hours on and off) I've finally tracked down the problem. You may no believe it but it was all down to a vpn client setting. Setting "Accept DNS Configuration" to "strict" killed DNS. Now what I'm not understanding though is VPN Director is set to only send my IoT devices to the VPN. Why it would slow down all DNS requests is the only mystery left now.

 

[Viktor Jaep]

I have a resolution, and it's likely it's not what anyone expected.
After starting from scratch, adding scripts in intervals (took 4 hours on and off) I've finally tracked down the problem. You may no believe it but it was all down to a vpn client setting. Setting "Accept DNS Configuration" to "strict" killed DNS. Now what I'm not understanding though is VPN Director is set to only send my IoT devices to the VPN. Why it would slow down all DNS requests is the only mystery left now.

I've actually have all my VPN slots set to "disabled" for that particular setting. Setting it to disabled would force utilization of what you have configured in your WAN settings, while also setting DNS director to "router". But nice sleuthing! It took a while for me to play with all these settings until I got acceptable results, but definitely strange that it would all of a sudden start acting up!

 

[Ripshod]

I've actually have all my VPN slots set to "disabled" for that particular setting. Setting it to disabled would force utilization of what you have configured in your WAN settings, while also setting DNS director to "router". But nice sleuthing! It took a while for me to play with all these settings until I got acceptable results, but definitely strange that it would all of a sudden start acting up!

Not quite all of a sudden. Strict was always good but this seems to coincide (loosely) with a firmware update, though the VPN update was with 3004.388.5.
Weird.

 

[cptnoblivious]

I have these kind of VPN issues on a semi regular basis, where vpnmon cycles through, but can't restart.

When I go into the router, I can't manually start or stop the client, only a reboot of the router helps.

Different than you you describe @Ripshod , because a router reboot fixes it. The issue is definitely in the router for me, because once the reboot has completed I'm generally find for another ... week, sometimes longer.

 

[Viktor Jaep]

I have these kind of VPN issues on a semi regular basis, where vpnmon cycles through, but can't restart.

This will happen to me as well... usually the VPN connection is made, but I find that it's not giving me a public IP address... usually it's blank. Then VPNMON kills it, and tries again. Sometimes multiple times until it finally settles. I thought it may have been NordVPN giving me trouble with running multiple connections (like a concurrent # of connections it's tracking, and if I'm exceeding my maximum allotted, make connection impossible). The more connections I run, the more likely this issue is. With 1 or 2 connections, the issue is pretty much non-existent.

 

[Ripshod]

I have these kind of VPN issues on a semi regular basis, where vpnmon cycles through, but can't restart.

When I go into the router, I can't manually start or stop the client, only a reboot of the router helps.

Different than you you describe @Ripshod , because a router reboot fixes it. The issue is definitely in the router for me, because once the reboot has completed I'm generally find for another ... week, sometimes longer.

I've seen this too (since 388.5). I noticed "Custom Configuration" gets wiped at the time, so I think we can assume the config gets messed up by something(?).

 

[Viktor Jaep]

I've seen this too (since 388.5). I noticed "Custom Configuration" gets wiped at the time, so I think we can assume the config gets messed up by something(?).

I have never seen the custom config getting deleted or manipulated on my end. That's a weird one. @Ripshod, what are you finding? Can you post a before and after of your custom config to see what changes, or does it just completely get blanked out?

 

[cptnoblivious]

This will happen to me as well... usually the VPN connection is made, but I find that it's not giving me a public IP address... usually it's blank. Then VPNMON kills it, and tries again. Sometimes multiple times until it finally settles. I thought it may have been NordVPN giving me trouble with running multiple connections (like a concurrent # of connections it's tracking, and if I'm exceeding my maximum allotted, make connection impossible). The more connections I run, the more likely this issue is. With 1 or 2 connections, the issue is pretty much non-existent.

Thanks for the response, I only have a single VPN slot configured on the router (using surfshark). It's not a huge deal, but I've gone back to having most systems running vpn clients on the endpoint so I _know_ if I'm no longer routed through the VPN

 

[cptnoblivious]

I've seen this too (since 388.5). I noticed "Custom Configuration" gets wiped at the time, so I think we can assume the config gets messed up by something(?).

I don't think the config's been wiped out, but next time it happens I'll definitely look at the custom settings!

 

[Ripshod]

I have never seen the custom config getting deleted or manipulated on my end. That's a weird one. @Ripshod, what are you finding? Can you post a before and after of your custom config to see what changes, or does it just completely get blanked out?

My custom config contains:

cipher AES-256-CBC
dev-type tun
remote-cert-tls server
tls-version-min 1.3
tun-mtu 1420
push-peer-info

When it occurs I first notice all my IoT devices drop. When I check the vpn client it can't connect and all the custom config has just disappeared. All the other settings appear fine so it should at least try to connect, but nothing in the logs.

 

[Viktor Jaep]

My custom config contains:

cipher AES-256-CBC
dev-type tun
remote-cert-tls server
tls-version-min 1.3
tun-mtu 1420
push-peer-info

Which VPN service do you use?

This is what I've been using for years with NordVPN, and worked equally as good with AirVPN:

remote-random
resolv-retry infinite
remote-cert-tls server
ping 15
ping-restart 0
ping-timer-rem
persist-key
persist-tun
reneg-sec 0
fast-io
disable-occ
mute-replay-warnings
auth-nocache
sndbuf 524288
rcvbuf 524288
push "sndbuf 524288"
push "rcvbuf 524288"
pull-filter ignore "auth-token"
pull-filter ignore "ifconfig-ipv6"
pull-filter ignore "route-ipv6"
explicit-exit-notify 3
tun-mtu 1500
tun-mtu-extra 32
mssfix 1450

When it occurs I first notice all my IoT devices drop. When I check the vpn client it can't connect and all the custom config has just disappeared. All the other settings appear fine so it should at least try to connect, but nothing in the logs.

That's really crazy... I'm not understanding how that just gets deleted. Do all the other vpn client fields look intact? I haven't been able to find any good reasoning for that. And is it only one VPN slot, or do they all no longer work, and all custom configs are gone?

 

[Ripshod]

I only run one slot, and it's my own remote openvpn access server. All the settings look intact bar the missing custom config - uploading the config file again brings it all back up and running.

 

[bencehavran]

Hi guys,

Does anyone else have a problem like mine where, for some reason, I assume it's the VPN server, it becomes unreachable and then the VPNmon freezes, as can be seen in my screenshot? I tried adding more servers with the script, but unfortunately, it still gets stuck, and nothing happens, no matter how long I wait. As you can see in the screenshot in the top left corner, it froze at 13:34 yesterday and hasn't responded since.

 

[bencehavran]

Okay, I've just realized that the list automation with the servers isn't working at all (TLS error). Unfortunately, it's extremely difficult to access the config, and I can't even get into the server list automation menu because the vpnmon closes, probably crashing.
I solved it manually by shutting down the vpnmon, fixing the servers in the GUI, and then starting the vpnmon again. This way, I can access the config because it doesn't start the restarting process, as all the monitored VPN servers are active.