OpenVPN Server Fails to Start on 386.1_2 on RT-AX88U

[NivekTheSizable]

The OpenVPN server fails to start at all on 386.1_2 on the RT-AX88U. This is using the same config that works fine on 386.1.

To attempt to troubleshoot, I tried to enable OVPN server 2, which I don't use, in its default configuration to see if that worked, but it also failed to start with the same errors in the logs.

Here's the error log (same errors for ovpn-server1):

Feb 26 11:14:17 ovpn-server2[12157]: --cipher is not set. Previous OpenVPN version defaulted to BF-CBC as fallback when cipher negotiation failed in this case. If you need this fallback please add '--data-ciphers-fallback BF-CBC' to your configuration and/or add BF-CBC to --data-ciphers.
Feb 26 11:14:17 ovpn-server2[12157]: Options error: --dh fails with 'dh.pem': No such file or directory (errno=2)
Feb 26 11:14:17 ovpn-server2[12157]: Options error: Please correct these errors.
Feb 26 11:14:17 ovpn-server2[12157]: Use --help for more information.
Feb 26 11:14:17 openvpn: Starting OpenVPN server 2 failed!

Overwriting the firmware to 386.1 solves the issue with no changes to the config. Here's the logs for the successful startup by comparison:

Feb 26 12:26:35 ovpn-server1[2432]: --cipher is not set. Previous OpenVPN version defaulted to BF-CBC as fallback when cipher negotiation failed in this case. If you need this fallback please add '--data-ciphers-fallback BF-CBC' to your configuration and/or add BF-CBC to --data-ciphers.
Feb 26 12:26:35 ovpn-server1[2432]: OpenVPN 2.5.0 arm-buildroot-linux-gnueabi [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [MH/PKTINFO] [AEAD] built on Jan 30 2021
Feb 26 12:26:35 ovpn-server1[2432]: library versions: OpenSSL 1.1.1i  8 Dec 2020, LZO 2.08
Feb 26 12:26:36 ovpn-server1[2444]: NOTE: your local LAN uses the extremely common subnet address 192.168.0.x or 192.168.1.x.  Be aware that this might create routing conflicts if you connect to the VPN server from public locations such as internet cafes that use the same subnet.
Feb 26 12:26:36 ovpn-server1[2444]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Feb 26 12:26:36 ovpn-server1[2444]: Diffie-Hellman initialized with 2048 bit key
Feb 26 12:26:36 ovpn-server1[2444]: Outgoing Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
Feb 26 12:26:36 ovpn-server1[2444]: Incoming Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
Feb 26 12:26:36 ovpn-server1[2444]: TUN/TAP device tun21 opened
Feb 26 12:26:36 ovpn-server1[2444]: TUN/TAP TX queue length set to 1000
Feb 26 12:26:36 ovpn-server1[2444]: /usr/sbin/ip link set dev tun21 up mtu 1500
Feb 26 12:26:36 ovpn-server1[2444]: /usr/sbin/ip link set dev tun21 up
Feb 26 12:26:36 ovpn-server1[2444]: /usr/sbin/ip addr add dev tun21 10.8.0.1/24
Feb 26 12:26:36 ovpn-server1[2444]: ovpn-up 1 server tun21 1500 1621 10.8.0.1 255.255.255.0 init
Feb 26 12:26:36 ovpn-server1[2444]: /usr/sbin/ip route add 10.8.0.0/24 via 10.8.0.2
Feb 26 12:26:36 ovpn-server1[2444]: ERROR: Linux route add command failed: external program exited with error status: 2
Feb 26 12:26:36 ovpn-server1[2444]: Could not determine IPv4/IPv6 protocol. Using AF_INET
Feb 26 12:26:36 ovpn-server1[2444]: Socket Buffers: R=[524288->524288] S=[524288->524288]
Feb 26 12:26:36 ovpn-server1[2444]: UDPv4 link local (bound): [AF_INET][undef]:61194
Feb 26 12:26:36 ovpn-server1[2444]: UDPv4 link remote: [AF_UNSPEC]
Feb 26 12:26:36 ovpn-server1[2444]: MULTI: multi_init called, r=256 v=256
Feb 26 12:26:36 ovpn-server1[2444]: IFCONFIG POOL IPv4: base=10.8.0.2 size=252
Feb 26 12:26:36 ovpn-server1[2444]: Initialization Sequence Completed

Anyone know if this is this a compilation issue in 386.1_2 or something I can fix myself without waiting for a new release?

 

[Jack Yaz]

Try reentering the dh section in certs, diffie hellman

 

[NivekTheSizable]

Try reentering the dh section in certs, diffie hellman

Looking at that in the web GUI, it appears that the key is there already. I tried a save of it, but the server still doesn't start.

 

[octopus]

Looking at that in the web GUI, it appears that the key is there already. I tried a save of it, but the server still doesn't start.

What do you have on this line?

 

[NivekTheSizable]

What do you have on this line?

View attachment 31396

For OVPN Server 1 (working on 386.1):
AES-128-GCM:AES-256-GCM:AES-128-CBC:AES-256-CBC

For OVPN Server 2 (in default config, not used):
AES-128-GCM:AES-256-GCM:BF-CBC

Interesting differences regardless of whether this is implicated in the problem with 386.1_2.

 

[octopus]

The OpenVPN server fails to start at all on 386.1_2 on the RT-AX88U. This is using the same config that works fine on 386.1.
To attempt to troubleshoot, I tried to enable OVPN server 2, which I don't use, in its default configuration to see if that worked, but it also failed to start with the same errors in the logs.

Here's the error log (same errors for ovpn-server1):

Feb 26 11:14:17 ovpn-server2[12157]: Options error: --dh fails with 'dh.pem': No such file or directory (errno=2)

Overwriting the firmware to 386.1 solves the issue with no changes to the config. Here's the logs for the successful startup by comparison:

Feb 26 12:26:36 ovpn-server1[2444]: Diffie-Hellman initialized with 2048 bit key

Anyone know if this is this a compilation issue in 386.1_2 or something I can fix myself without waiting for a new release?

Make sure you have that dh.pem key in config file seems its missing.

 

[NivekTheSizable]

Make sure you have that dh.pem key in config file seems its missing.

Apologies for the newbie question, but how would I do that?

 

[octopus]

Apologies for the newbie question, but how would I do that?

Under Keys and Certificates "edit" button. See if is't in there.

 

[NivekTheSizable]

Under Keys and Certificates "edit" button. See if is't in there.

OK, I upgraded to 386.1_2. and checked in there, all keys were blank. I had saved them before the upgrade and pasted them in and hit save. Once again, the VPN server didn't start - checked there again and all the keys are blank again.

Is there some sort of JFFS issue perhaps? In 386.1_2 I don't even have the option to save or restore JFFS:

This is what it looks like back in 386.1:

When I downgrade to 386.1, the JFFS option (and the OVPN keys) are back.

I'm a bit stumped as to where to go from here.

 

[RMerlin]

Check the state of your JFFS partition on the Tools -> Sysinfo page.

 

[NivekTheSizable]

Check the state of your JFFS partition on the Tools -> Sysinfo page.

OK, it's showing unmounted. Really, really weird that 386.1 mounts it without issue, even after repeated downgrades:

By comparison, the same screen in 386.1:

I see plenty of logs complaining about /jffs/ not being mounted, but no log entries that seem to say why.

Guessing I should just format the JFFS and restore it from backup at this point? Or is there some other way, perhaps from a SSH command line, to mount and try to repair/recover it?

 

[RMerlin]

Guessing I should just format the JFFS and restore it from backup at this point? Or is there some other way, perhaps from a SSH command line, to mount and try to repair/recover it?

Please post the result of this command:

nvram show | grep jffs2

After that, try maybe just clearing the size, and also setting both toggle flags (one is used by older versions the other one by Asus/newer Asuswrt-Merlin):

nvram unset jffs2_size
nvram set jffs2_enable=1
nvram set jffs2_on=1
nvram commit
reboot

Nothing will show up in the system log because the JFFS partition is mounted too early. The only way to get any debug info is through the serial console.

 

[NivekTheSizable]

Please post the result of this command:

nvram show | grep jffs2

Here is the output of that command from 386.1:

admin@hostredacted:/tmp/home/root# nvram show | grep jffs2
size: 76019 bytes (55053 left)
jffs2_enable=1
jffs2_format=0
jffs2_on=0
jffs2_scripts=0

Output on 386.1_2:
admin@hostredacted:/tmp/home/root# nvram show | grep jffs2
size: 71675 bytes (59397 left)
jffs2_enable=1
jffs2_format=0
jffs2_on=0
jffs2_scripts=0

After that, try maybe just clearing the size, and also setting both toggle flags (one is used by older versions the other one by Asus/newer Asuswrt-Merlin):

nvram unset jffs2_size
nvram set jffs2_enable=1
nvram set jffs2_on=1
nvram commit
reboot

Nothing will show up in the system log because the JFFS partition is mounted too early. The only way to get any debug info is through the serial console.

Executing these commands now allows the JFFS partition to mount and therefore the OpenVPN server to start normally; no JFFS reformat was needed. The "nvram show | grep jffs2" command now shows "jffs2_on=1".

Many thanks for your help.

 

[RMerlin]

jffs2_on is supposed to be set to "1" by default both on the stock firmware and my firmware, so I don't know how yours got set to 0.

 

[NivekTheSizable]

jffs2_on is supposed to be set to "1" by default both on the stock firmware and my firmware, so I don't know how yours got set to 0.

Yeah, who knows. This router has a relatively short history - it was bought about two months ago then I did a factory reset and new config build of it with your 386.1 firmware, if that helps narrow down where something might not have changed that setting.

 

[octopus]

This in mine on RT-AX86U

octopus@RT-AX86U-EA08:/tmp/home/root# nvram show | grep jffs2
size: 77334 bytes (53738 left)
jffs2_enable=1
jffs2_format=0
jffs2_on=1
jffs2_scripts=1
jffs2_size=49283072