Merlin 386.5.2 bug: OpenVPN Server does not start after router restart (scheduled restart, explicit restart, or power reset)

[David Kremer]

I run Merlin 386.5.2 on Asus RT-AC88U. I configured it to automatically reboot once every 24 hours. I noticed that sometimes after the reboot, the OpenVPN Server does not start, and I cannot connect to the VPN, but I can still log into the router. In this case, even rebooting the router manually does NOT restore the OpenVPN Server. The only solution is to manually turn off the OpenVPN Server and turn it on again, then it works. This bug does not occur often, in one year of automatic restarts every 24 hours this condition has only occurred twice.

Is this a known bug, or is this the first mentioning of it?

 

[L&LD]

Don't believe I've read about this issue before on these forums.

I would not be rebooting the router in the first place.

Is there a reason for doing so?

 

[GSpock]

The very rare cases I expiremented issue with OpenVPN Server not starting, it was due to the fact that (for whatever reasons) the router could not sync its date/time and was showing the usual boot date of 05-MAY ....

 

[David Kremer]

This is a serious problem. It also happens when the router restarts after an electric outage. Any fix?

 

[David Kremer]

Don't believe I've read about this issue before on these forums.

I would not be rebooting the router in the first place.

Is there a reason for doing so?

I wish that were an option, but for example when the electricity goes out the router boots up, and the same problem occurs: VPN does not start up.

 

[David Kremer]

The very rare cases I expiremented issue with OpenVPN Server not starting, it was due to the fact that (for whatever reasons) the router could not sync its date/time and was showing the usual boot date of 05-MAY ....

You're right, this is what happened to me! Router reboots due to electric outage, comes back and the time is May 5, 2018. That's why the VPN can't start.
Is there a fix for this "May 5, 2018" time bug?

 

[L&LD]

No, there is no fix. There is no backup battery for a real-time clock on (most) routers.

If the power goes out frequently, you need to invest in a good UPS.

 

[RMerlin]

Check your WAN DNS configuration (as a working DNS setup will be required for the router to be able to connected to the NTP server). You can also try either a different NTP server than the generic NTP pool, or specify an NTP server directly by its IP address.

 

[rijoy.manghat]

I have the same issue as well. I never reboot my router but this time I was out and there was a power outage I couldnt connect to my home network and later figured that the openvpn server didnt restart. Has this been always the case that the open vpnserver needs to be restarted manually?

 

[laracroftonline]

install scmerlin and activate Enable NTP boot watchdog script and set that to yes.

 

[David Kremer]

installing scmerlin might open a can of worms. The solution could be worse than the problem. How do we know scmerlin is stable, or won't cause OTHER problems?

 

[SoFluffy]

Obviously the right answer is if NTP works well. I don‘t reboot that often but have never had this issue on either Merlin-based OpenVPN servers I run. But presumably the time does get set if you can start it manually - I’d just set a cron job to start it up maybe 2 minutes after boot as a workaround.

 

[eibgrad]

AFAICT, this is NOT an OpenVPN server bug.

In my own testing w/ the RT-AC68U, it's clear to me the OpenVPN server will NOT run unless the NTP client has successfully set the time. And for obvious reasons; it can't validate either its own certs, or those of any potential OpenVPN clients w/o the correct time. So it waits in the background until the time is set.

Of course, the 64k question is why the time may be failing to be set. As Merlin suggested, you might want to include an explicit IP for the ntp client in cases it's a DNS issue. At least according to the OP, this is a very rare occurance, so it's difficult to know why it happens on any given reboot w/o deeper analysis.

 

[David Kremer]

AFAICT, this is NOT an OpenVPN server bug.

In my own testing w/ the RT-AC68U, it's clear to me the OpenVPN server will NOT run unless the NTP client has successfully set the time. And for obvious reasons; it can't validate either its own certs, or those of any potential OpenVPN clients w/o the correct time. So it waits in the background until the time is set.

Of course, the 64k question is why the time may be failing to be set. As Merlin suggested, you might want to include an explicit IP for the ntp client in cases it's a DNS issue. At least according to the OP, this is a very rare occurance, so it's difficult to know why it happens on any given reboot w/o deeper analysis.

What would be a good static NTP ip address to use? As you know, ip addresses are not static which is why hostnames are used.

 

[eibgrad]

What would be a good static NTP ip address to use? As you know, ip addresses are not static which is why hostnames are used.

Correct. But there is a secondary line that would provide the opportunity to use a explicit IP as a backup. We're just trying to circumvent the remote possibility of a DNS issue, particularly these days w/ some users depending on secure solutions like DoT, that need the correct time to verify their certs! A classic catch-22 situation.

I'm thinking whatever you might resolve from the likes of time.apple.com or time.microsoft.com.

 

[David Kremer]

@eibgrad I can confirm that the OpenVPN bug on Merlin is not related to NTP. I followed your suggestion and set static IP's for the NTP servers under Administration->System, and rebooted the router. The router started up with the correct time (today's date of July 22, 2022, not the incorrect date of May 5, 2018 when NTP fails to get the time). However, despite the correct date being set, OpenVPN could not accept new connections even though it was turned on. I had to manually stop OpenVPN and start it again, then it worked.

This points to a cause that's unrelated to NTP or the time on the router. How can we get the attention of the Merlin maintainer?

 

[eibgrad]

Initially you said this bug occurred quite rarely, about twice a year, and yet NOW you can produce it at will?

That's a big difference. It's going to be next to impossible to pin down the issue unless it can be reproduced reliably. I'm NOT doubting you have a problem, but at least here, w/ my RT-AC68U, I can't reproduce it.

FYI, the OpenVPN server does have a watchdog process that's supposed to restart a failed OpenVPN server. So normally I wouldn't expect this to be a problem. But the fact it is suggests the watchdog may be in a chicken and egg situation, where it doesn't start monitoring the OpenVPN server until it gets started. It might call for a separate, independent watchdog, similar to the one I wrote for the OpenVPN client. I had just assumed the built-in watchdog solved any such issues.

admin@lab-merlin1:/tmp/home/root# cat /tmp/etc/openvpn/server1/vpn-watchdog1.sh
#!/bin/sh
if [ -z $(pidof vpnserver1) ]
then
   service restart_vpnserver1
fi

It runs off the scheduler every 2 mins.

admin@lab-merlin1:/tmp/etc/openvpn/server1# cru l

*/2 * * * * /etc/openvpn/server1/vpn-watchdog1.sh #CheckVPNServer1#

You may need to be a bit of a sleuth here and see if you can determine what is and isn't there when the problem occurs. IOW, if things like the /tmp/etc/openvpn/server1 directory are or are NOT present, whether the scheduled task is there, etc.

Of course, a syslog would help as well.

Until the developer or I can reproduce the problem, this is going to be very difficult. As I said, in the end I suppose we could put together a script, but I'd like to avoid it since it's supposed to have a working watchdog already.

 

[David Kremer]

FYI, the OpenVPN server does have a watchdog process that's supposed to restart a failed OpenVPN server.

Actually the issue is not that the OpenVPN server is not started. The issue is that the OpenVPN server is running but does not accept new connections.
I can also confirm that it's not related to NTP or incorrect system time, since this occurs even when the router time is set correctly.
The only fix is to stop the OpenVPN server and restart it, which is not a viable fix because it requires frequent manual intervention.

 

[eibgrad]

Actually the issue is not that the OpenVPN server is not started. The issue is that the OpenVPN server is running but does not accept new connections.
I can also confirm that it's not related to NTP or incorrect system time, since this occurs even when the router time is set correctly.
The only fix is to stop the OpenVPN server and restart it, which is not a viable fix because it requires frequent manual intervention.

Whoa, that's a big difference! That's NOT what you said originally. No wonder I couldn't reproduce it. LOL

 

[David Kremer]

Yes, sorry, I mean to say that the VPN server doesn't start functionally, but according to the UI it is running. The green switch for VPN Server 1 shows ON, even though no connections are accepted.