What's new

Bug with corrupted passwords in OpenVPN Server

  • SNBForums Code of Conduct

    SNBForums is a community for everyone, no matter what their level of experience.

    Please be tolerant and patient of others, especially newcomers. We are all here to share and learn!

    The rules are simple: Be patient, be nice, be helpful or be gone!

bengalih

Senior Member
I had this issue which appears to have ben discussed at least once here:

However that was from a while ago and no definitive troubleshooting/root cause was listed.
I have some more specific information and I believe this is a a big enough problem it should have some more visibility for a possible fix, and at the very least to inform people who might come looking for the same issue.
I'm not sure if this is something that @RMerlin can fix, or if it is baked into the Asus code and therefore we are stuck with it.
At the very least I hope this helps others who are experiencing this strange issue, and if lucky we can find a fix (or at least someone with the knowledge can state how widespread and/or if it has indeed been fixed).

I want to preface that I am on a RT-AC68U with 386.3_2, which I realize is about 3 years old. The reason I have not upgraded is that last I checked (about a year ago) there were some problems with NVRAM space issues with the newer firmware which seemed like they were going to cause issues for me (large custom_clientlist). I don't know if any of these issues are fixed, I will have to research that to see if I can upgrade and if these OpenVPN issues are resolved. However, I have looked at the changelog and do not see anything which would indicate that these OpenVPN issues are fixed, and not much in the forums at all acknowledging the issue.

Issue:
As was mentioned in the linked thread above, the issue exhibits itself upon adding new users to the OpenVPN server. I have tested with only 2 users (not including the built-in account, which is apparently never affected). In short, what happens is that if you add a second user at a future time (most likely anytime after the router has been rebooted at least once) the password of the first user gets corrupted. If you then remove and readd the first user (as this is the only way to change/reset password), the second user would then get affected. The only solution is to add both users at once. I suspect this issue would occur on subsequent users, although I'm unsure if it would corrupt all users passwords or just some.

Some technical information and testing:
I won't claim to be definitive on this, but from my investigation it appears that the usernames and passwords for the OpenVPN server are saved in NVRAM.
Specifically in the vpn_serverx_clientlist variable.

In the below example, I show a user exists called jsmith exists with a password "ThisIsMyPassword".

Code:
myadmin@router-asus:/tmp/home/root#  nvram show | grep vpn_serverx_clientl
vpn_serverx_clientlist=<jsmith>ThisIsMyPassword
Now you will note that the password is unencrypted. It appears to remain this way initially, I believe until a reboot happens (whether this is a full reboot or a reboot just of the VPN services, I have not yet tested).
After a reboot however, the entry will look like this:
Code:
myadmin@router-asus:/tmp/home/root#  nvram show | grep vpn_serverx_clientl
vpn_serverx_clientlist=<jsmith>AaBbCc123
XxYyZz12345 is just my shorthand of the encrypted password. At this point, everything is fine with the VPN and will remain so indefinitely. I had been running mine for years with this configuration and only a single user (in addition to the built-in admin account).
If however you decide to add a second user. Let's say we want to add jdoe with password "Password123", this is what it will look like:
Code:
myadmin@router-asus:/tmp/home/root#  nvram show | grep vpn_serverx_clientl
vpn_serverx_clientlist=<jsmith>AaBbCc123<jdoe>Password123
The above makes sense based on what we have seen so far. The jsmith password is unchanged (and encrypted), but the new jdoe password has not yet been encrypted. We would expect/hope that after a reboot both would be properly encrypted, but this is only half true. The VPN will continue to work properly with both user account until a reboot occurs. At that time, the jsmith account will cease to function, and looking at the variable we can see why:
Code:
myadmin@router-asus:/tmp/home/root#  nvram show | grep vpn_serverx_clientl
vpn_serverx_clientlist=<jsmith>AFBgMc173<jdoe>XxYyZz456
"XxYyZz456" represents the proper encrypted form of jdoe's password. However "AFBgMc173" represents that the encrypted password of jsmith has been garbled and no longer represents the unencrypted "ThisIsMyPassword". Therefore, the jsmith user will now no longer be able to login.
You can solve this by deleting and re-adding jsmith with the proper credentials, which will then display the following:
Code:
myadmin@router-asus:/tmp/home/root#  nvram show | grep vpn_serverx_clientl
vpn_serverx_clientlist=<jdoe>XxYyZz456<jsmith>ThisIsMyPassword
And everything will continue to work until the next reboot:
Code:
myadmin@router-asus:/tmp/home/root#  nvram show | grep vpn_serverx_clientl
vpn_serverx_clientlist=<jdoe>XAPyZq596<jsmith>AaBbCc123
Here we can see that jdoe's encrypted password has been corrupted to "XAPyZq596", and that user will no longer be able to login.
This cycle will continue with only one solution: Delete both users and add them back at the same time.

This is obviously a major problem for this function if you have several users. Even if you could add all known users at once, if one user forgets their password, the only way to reset it is to add and delete the user which will affect at least one other user and then it is a chain effect from there. All users would need to be removed and re-added. Add to that the fact it is impossible to determine the unencrypted passwords of the current users, all users will have to have their passwords reset.

Again, I don't know what other models and firmware versions this affects, but as it is likely not tied to the model, and I can't find anything in the changelog about remedies to it, I expect it affects many devices out there and simply not many people are using multiple users on their VPN. Those that are however will be strongly affected.

NOTE:
One other side piece of technical information. As OpenVPN is authenticated via the underlying PAM subsystem, therefore the user accounts and passwords appear to also exist in the /etc/password and /etc/shadow files. These files are generated each time at boot. I can only assume that the data inside NVRAM (specifically vpn_serverx_clientlist) is used to create the data values in these files.
Even when the passwords in NVRAM match, the values in /etc/shadow may be different each time. This is due to the way the salts are used in the hash.
 
I want to preface that I am on a RT-AC68U with 386.3_2, which I realize is about 3 years old. The reason I have not upgraded is that last I checked (about a year ago) there were some problems with NVRAM space issues with the newer firmware which seemed like they were going to cause issues for me (large custom_clientlist). I don't know if any of these issues are fixed, I will have to research that to see if I can upgrade and if these OpenVPN issues are resolved. However, I have looked at the changelog and do not see anything which would indicate that these OpenVPN issues are fixed, and not much in the forums at all acknowledging the issue.
RMerlin made the following change 386.11 (14-May-2023) to address the low NVRAM issue on the RT-AC68U.
CHANGED: Reduce max OpenVPN clients to 2 for RT-AC68U and
DSL-AC68U due to lack of NVRAM on these two
models. Note that existing settings are not
automatically removed, you must run the following
command over SSH to remove them from nvram and
the /jffs/openvpn/ directory:

clear_vpnclients.sh

A backup will be saved in /jffs/openvpn_backup.tgz.
When you indicate you have checked the "change log" are you referring to the Asus-Merlin change log or to the OpenVPN change log?
https://github.com/OpenVPN/openvpn/blob/v2.6.10/Changes.rst

PS: You really should check the latest firmware to see if it resolves the issue you are discussing. OpenVPN has been updated repeatedly over the years since version 2.5.3 (now at 2.6.10). Not to mention all the other security fixes in the firmware that have been made between 386.3_2 (6-Aug-2021) and 386.13_2 (26-Apr-2024).
 
Last edited:
RMerlin made the following change 386.11 (14-May-2023) to address the low NVRAM issue on the RT-AC68U.

When you indicate you have checked the "change log" are you referring to the Asus-Merlin change log or to the OpenVPN change log?
https://github.com/OpenVPN/openvpn/blob/v2.6.10/Changes.rst

PS: You really should check the latest firmware to see if it resolves the issue you are discussing. OpenVPN has been updated repeatedly over the years since version 2.5.3 (now at 2.6.10). Not to mention all the other security fixes in the firmware that have been made between 386.3_2 (6-Aug-2021) and 386.13_2 (26-Apr-2024).
Yes, I acknowledge that I'm running an older version. However, as mentioned many people did not want to upgrade much past 386.3_2 due to the NVRAM issues. I suspect there are still many people on this older version. I will have to do additional research to see if there has been a better solution to the NVRAM issues with the latest releases for my model and if so I will likely upgrade.

That being said, while I was talking about checking the Merlin changelog (not the OpenVPN), this is unlikely an OpenVPN issue, but rather how OpenVPN is being implemented on Merlin (or stock ASUS). I say this because it seems pretty apparent that the corruption is happening in the variable stored in NVRAM which would be an ASUS specific thing. In fact all the mechanisms I discuss in the OP would seem to be ASUS/Merlin specific. If there was an issue writing/reading from /etc/shadow that would be more a PAM specific issue, but I don't think this issue is actually related to any core OpenVPN code, but rather how it is implemented on these models.

Even if the issue is resolved in later versions, I'm very surprised that there is/was not more people posting about this since the affected versions are/were actually used for a good period of time. I can only assume there were not that many users who actually configure more than one additional user for OpenVPN as I think anyone with this device would be able to recreate the issue.

It would be great if other users (especially those on RT-AC68U after 386.3_2) can state whether or not they can recreate the issue. If I am unable to upgrade past this due to NVRAM issues still existing, then I will just have to live with this. If it has been fixed post 386.3_2, it might not be affecting most models for users that upgraded, but those that have older routers who didn't want to have problems with later versions (e.g. NVRAM), might still be in my shoes.

My main purpose in posting is to make anyone else who might come searching for a similar problem to see my results and understand the workarounds. Ofc, if an upgrade is possible that should be the route to take, but until there is some type of confirmation that this was a known issue (i.e. root cause) and that it was addressed, no one has any reason to assume that the behavior is now changed.
 
RMerlin made the following change 386.11 (14-May-2023) to address the low NVRAM issue on the RT-AC68U.
Thanks... I somehow missed you addressing the fixes for NVRAM in my last response. Based on this I do plan to do the upgrade, but I am actually away for about 6 weeks (hence why I was messing more with the VPN and discovered the problem). I'll be sure to come back after I upgrade and test these password corruption issues to see if they still need to be addressed.

I will be pleasantly surprised if this issue is fixed since as mentioned this definitely wouldn't be addressed directly by OpenVPN because it is an issue with the NVRAM implementation of storing passwords. I looked at the stock ASUS changelogs and the last one which addresses OpenVPN is:

Version 3.0.0.4.385.20585 2020/06/19

As 386.3_2 was apparently released in 2021 it would have had those fixes. Since neither Merlin nor Asus addresses anything relating to this issue after those dates my guess is it still exists.

In the meantime, while I await my upgrade I would love to hear from anyone else who is adding multiple VPN users and what their model/FW revision is and if they come across (or ever came across) this issue.
 
I have eight defined in one, and two in another, and they aren't in nvram. I think they are stored in /etc, at least they appear there without passwords.
 
FWIW, I'm running 386.12_4 w/ my RT-AC68U, and from my testing, everything worked fine. In fact, the passwords were always encrypted *immediately*, before a reboot (perhaps that was part of the original problem). After a reboot, everything remained the same, correctly encrypted just as before the reboot.

BTW, I see the values are stored in vpn_serverx_clientlist, both in nvram and /jffs/nvram.
 
I have eight defined in one, and two in another, and they aren't in nvram. I think they are stored in /etc, at least they appear there without passwords.
I'm not sure what model you have, but I believe most models build the file systems from scratch on boot. This is why we need the /jffs partition to store data to customize those files as well as using the data stored in NVRAM.

The vpn_serverx_clientlist stores that data on my model (as mentioned in my OP) and applies them to the appropriate /etc files on boot.
That includes /etc/passwd and /etc/shadow. As per most standard linux-based systems the passwords are encrypted which is why you don't see them in /etc/passwd. If you look in shadow however you should see them.

You can also do:
Code:
nvram show | grep [your vpn usersname here]

And you will likely find what NVRAM variable it is stored in that gets applied.

Not that it helps much if I don't know your model/FW, but I would assume that with 8 users on your one VPN server you didn't create them all simultaneously?
 
FWIW, I'm running 386.12_4 w/ my RT-AC68U, and from my testing, everything worked fine. In fact, the passwords were always encrypted *immediately*, before a reboot (perhaps that was part of the original problem). After a reboot, everything remained the same, correctly encrypted just as before the reboot.

BTW, I see the values are stored in vpn_serverx_clientlist, both in nvram and /jffs/nvram.
Appreciate the testing. Well, hopefully that is good news and means it was resolved on some future FW.
I'll report back on if mine is resolved once I have the chance to upgrade.
 
I thought my model/firmware combos might have shown up in my signature. Anyway, both are 388.7; one is an AX88 and the other an AX86Pro. It could be a change in the codebase from 386 to 388.

The clientlists for both servers are the same, and I think always have been. My clients don't show up in nvram. Perhaps because I have client specific options enabled, so I have CCD directories for active clients. Those CCDs show up in nvram, but no passwords.
 
I'm experiencing this same issue... OpenVPN passwords other than the admin account are lost on reboot. Curious if anyone has developed a scripted solution to his issue.
 
I'm experiencing this same issue... OpenVPN passwords other than the admin account are lost on reboot. Curious if anyone has developed a scripted solution to his issue.

What release? Remember, the OP was running a very old release (386.3_2), whereas I tested using something much more recent (386.12_4) that did NOT have the problem. I suspect there was a bug and it was corrected. But if you're using something much older, then perhaps that's the issue.
 
What release? Remember, the OP was running a very old release (386.3_2), whereas I tested using something much more recent (386.12_4) that did NOT have the problem. I suspect there was a bug and it was corrected. But if you're using something much older, then perhaps that's the issue.
Current release (3004.388.8) on RT-AX86U. I've seen the passwords disappear 3-4x, but didn't associate it with a reboot until today, so the problem has probably existed for a few releases.
 
What release? Remember, the OP was running a very old release (386.3_2), whereas I tested using something much more recent (386.12_4) that did NOT have the problem. I suspect there was a bug and it was corrected. But if you're using something much older, then perhaps that's the issue.
Just completed a second reboot and passwords survived this time. Can't explain it. Passwords were there yesterday, gone after one reboot this morning, but present after a second one. Going to have to research further to replicate the issue.
 
Make sure your /jffs partition isn't full.
 
Make sure your /jffs partition isn't full.
No, 94% available.
/dev/mtdblock9 48128 2792 45336 6% /jffs

Since it's happened a few times, now, I'm going to have to keep a close eye on it and see if I can determine what triggers it. In troubleshooting, I've tried deactivating and reactivating OpenVPN, and the passwords are still there...and the latest reboot was also no issue. I definitely lost the passwords with the 388.7 to 388.8 upgrade earlier this morning, though, after I just put them back in yesterday. Going to have to track back in my archived syslogs to when I successfully used it last prior to yesterday and see if I can identify anything that occurred in between.
 

Similar threads

Latest threads

Support SNBForums w/ Amazon

If you'd like to support SNBForums, just use this link and buy anything on Amazon. Thanks!

Sign Up For SNBForums Daily Digest

Get an update of what's new every day delivered to your mailbox. Sign up here!

Members online

Top