Firmware bug found by changing Username & Password?

[Martinski]

To finish up my Friday night and start off the weekend, I decided it was time to take the plunge and upgrade my RT-AC86U router from "384.18_0" to "384.19_0" F/W version. After following all the recommended/required steps before *and* after the upgrade (e.g. make a backup of JFFS partition, reboot, upgrade, reset to factory defaults, reformat JFFS partition, reboot, restore JFFS partition, a couple of more reboots, etc.), so far the latest firmware appears to be working very well. I've been double-checking to make sure all the router services are indeed working as expected (e.g. Routing, DHCP, DoT, DDNS, two OpenVPN Servers, 2.4GHz & 5GHz wireless bands, Ethernet ports, etc.), and so far so good.

Then I decided to make one last change: modify my username and password credentials for the WebUI login. I had not changed these for almost 2 years since I got the Asus router, and lately I've been wanting to increase the entropy of my login password (since the max. number of chars is only 16, this limits its strength). Anyway, this change also went well, and after updating my OpenVPN & SSH clients with my new credentials (no change in TSL certs or PKI keys were made) everything is working as expected. However, while double-checking the router again, I discovered that there was one thing that was not working: all my cron jobs were *not* scheduled to run at all. Running the command "cru l" in my SSH session gave no output at all so no jobs were scheduled, including the built-in watchdogs for the 2 OpenVPN Servers and for the "Let's Encrypt" service.

I also noticed about a hundred of these lines in the system logs:

---------------------------------------------------------------
...
Nov 6 23:32:00 crond[1446]: can't get uid for AdminXXXXXXXXX
Nov 6 23:32:00 crond[1446]: can't get uid for AdminXXXXXXXXX
Nov 6 23:34:00 crond[1446]: can't get uid for AdminXXXXXXXXX
Nov 6 23:34:00 crond[1446]: can't get uid for AdminXXXXXXXXX
Nov 6 23:36:00 crond[1446]: can't get uid for AdminXXXXXXXXX
Nov 6 23:36:00 crond[1446]: can't get uid for AdminXXXXXXXXX
Nov 6 23:38:00 crond[1446]: can't get uid for AdminXXXXXXXXX
Nov 6 23:38:00 crond[1446]: can't get uid for AdminXXXXXXXXX
Nov 6 23:40:00 crond[1446]: can't get uid for AdminXXXXXXXXX
Nov 6 23:40:00 crond[1446]: can't get uid for AdminXXXXXXXXX
Nov 6 23:40:00 crond[1446]: can't get uid for AdminXXXXXXXXX
...
(where "AdminXXXXXXXXX" was my *OLD* Username, which means that for some reason the cron daemon was not aware of my change in login credentials.
---------------------------------------------------------------

At this point the obvious thing to do was to reboot the router one more time to make sure everything was restarted properly; and doing this did actually fix the "missing cron jobs" issue. However, IMO, this "hiccup" appears to be a bug caused by or related to changing the username & password. Granted, this change is not done often and it's easily fixed with a reboot, but had I not double-checked that things were working well, I could have easily left the router alone, and perhaps days would have gone by before I ever notice that none of the cron jobs were running at all.

Anyway, I hope this helps someone else as a heads-up in case they decide to change their router WebUI login credentials.

 

[Ronald Schwerer]

Wait. You changed the admin user name (and password) but didn't reboot, and you expected the old user's scripts to work? That's not a bug.

 

[dosborne]

That's not a bug.

While perhaps not strictly a bug, it would be beneficial if there was a notification at least that due to the credentials having been changed, that a reboot should be performed so reinitiate background processes. It 8s not necessarily intuitive that a reboot is required, and if this applies if you only change the password too (?), then I can see where many users would miss the reboot until they discovered things were not working. Although easily sorted with a reboot, it would / should be easy to display a notification vs the potential headache.

 

[Ronald Schwerer]

While perhaps not strictly a bug, it would be beneficial if there was a notification at least that due to the credentials having been changed, that a reboot should be performed so reinitiate background processes. It 8s not necessarily intuitive that a reboot is required, and if this applies if you only change the password too (?)

I agree that is probably an oversight in the design. As someone who has long time experience in using/configuring Unix/Linux systems, it just seems obvious to me. But I agree it might not seem so obvious to users with only consumer-level experience.

AFA reboot after a password change - no, I don't think a reboot is required since the UID (User IDentifier = a unique number assigned to a user name at time of creation) of the root/admin user didn't change. The UID/GID (GroupID) is used to distinguish ownership and privileges to all files and processes in the system.

However, it also occurs to me that there are many settings in the GUI that don't take full effect until the router is rebooted. I don't know which ones so I probably reboot more often than necessary. But those generally are things that I don't change often, or are just experimenting with. Regardless, the same case could be made that there should be warnings for those settings too. And I'll add that some setting changes effectively (and unexpectedly) restart the whole router from the kernel up without warning. Click "APPLY" and watch the countdown slowly from 100.....I can hear the wife screaming now "the network is DOWN!!!!".

 

[L&LD]

Making major or minor changes without rebooting any 'critical' system such as a router makes no sense at all to me.

'Supposed to work' without a reboot is a miserable consolation to an 'I need it to work' mission-critical aspect of any computer reliance.

In about a thousand years the 'supposed to work' will be figured out and everything we are playing with today will be mere appliances the size of a pinhead.

Today? Reboot.

 

[sbsnb]

One of the things that was always touted to me as making Linux superior to Windows was NOT having to reboot after making some little change. I've never in 30 years heard of rebooting after changing a password. Even Windows doesn't require that. You learn something new every day.

 

[L&LD]

Linux superior to Windows? Never heard that one before.

 

[Ronald Schwerer]

One of the things that was always touted to me as making Linux superior to Windows was NOT having to reboot after making some little change. I've never in 30 years heard of rebooting after changing a password. Even Windows doesn't require that. You learn something new every day.

No one said you needed to reboot after changing a password. The issue discussed is due to changing the administrator's login id. This is effectively adding a new login and deleting the old one. So ownership of critical processes has changed. That isn't a little change. Other config parameters that are changeable in the web GUI are not Linux - they are the router application or driver parameters. In a complex system it is far easier and more reliable to just restart everything by rebooting.

You're comparing apples to oranges. Linux is a real operating system designed for computing platforms. It is much more light-weight than Windows, making it more versatile and useful for embedded systems - like routers and other small computers. The down-side is it isn't natively as user friendly. Unless you are logging-in via ssh/telnet and using the command line interface, you are dealing with an application like the ASUS' GUI. That's not Linux. The graphical user interface is an after-thought to make it slightly more user friendly.

So bottom line Linux is superior to Windows in some areas and has much less overhead. It was designed for computing platforms. It is very useful for embedded systems. Windows was designed for the masses and is aimed at desktop computing.

 

[sbsnb]

Interesting. Windows just assigns an account to a GUID. You can change the username and/or password all you want, but it's still the same GUID. Since it works permissions based on the GUID you don't have this issue. I never knew changing the username associated with an account in Linux actually had the effect of deleting the old account and creating a new one.

 

[dave14305]

The problem really just seems to be the crontab is written based on the username, while file ownership and other ownership concerns are mapped with uid=0 (root) and wouldn’t care about the username.

It’s a one-off to me because most stock Asus users don’t even know cron exists.

 

[sbsnb]

It's a very minor bug, but it's interesting to understand the potential implications of a small mistake.

 

[Ronald Schwerer]

The problem really just seems to be the crontab is written based on the username, while file ownership and other ownership concerns are mapped with uid=0 (root) and wouldn’t care about the username.

Good point. I'm old-school Unix/Linux and this whole notion of changing the root (uid=0) user name is foreign to me. Root is root. Even calling it "admin" by default seems strange. But those systems were not exposed to the internet, so they didn't need one more layer of obfuscation (guessing the root name as well as password).
I didn't know crontab used username and not uid. That explains OP's original issue. In this case, a reboot isn't the issue as much as resetting the cron schedules. And in defense of ASUS, that is beyond the scope of their firmware. AFAIK, the router's stock firmware doesn't use cron to schedule anything. Login and run cru l and you won't find anything scheduled. Same for Merlin. That's 3rd party or user installed software. With just system firmware (no 3rd party or user added), there's probably no issue in changing the root user name since uid is still 0.

 

[RMerlin]

One of the things that was always touted to me as making Linux superior to Windows was NOT having to reboot after making some little change. I've never in 30 years heard of rebooting after changing a password. Even Windows doesn't require that. You learn something new every day.

A full-featured Linux distro, sure. You can easily restart every services. But a custom router firmware that only uses the Linux kernel and everything else around it is custom-made? That's more problematic. There's no systemd or initd to cleanly handle things.

 

[thecheapseats]

...There's no systemd or initd to cleanly handle things...

^^^that...

 

[thecheapseats]

toys require a reboot... real systems designed for uninterrupted uptime don't

 

[Martinski]

...
And in defense of ASUS, that is beyond the scope of their firmware. AFAIK, the router's stock firmware doesn't use cron to schedule anything. Login and run cru l and you won't find anything scheduled. Same for Merlin. That's 3rd party or user installed software. With just system firmware (no 3rd party or user added), there's probably no issue in changing the root user name since uid is still 0.

Actually, the Asus stock F/W adds some cron jobs for some built-in services/features when they're enabled. The "Let's Encrypt" service for the HTTPS/SSL certificate is one of them. See the "#LetsEncrypt#" cron job if you use that type of certificate.

In addition to that, the RMerlin F/W also adds a cron job (i.e. a watchdog) for each of the OpenVPN Servers that is enabled/configured (see "#CheckVPNServer1#" or "#CheckVPNServer2#").

Bottom line is that cron jobs are not only added by user-installed s/w or 3rd-party addons. They are part of the system as designed whenever some built-in features are enabled.

 

[Martinski]

Making major or minor changes without rebooting any 'critical' system such as a router makes no sense at all to me.

I'm fully aware that that is your perspective & opinion (to which you're certainly entitled) based on your own experience that's been collected, absorbed, and analyzed using your own personal & professional knowledge, background & lessons learned (however large or limited those may or may not be). But, just because your opinion has become your own "personal truth" it does not make it true for every case. And this, of course, it's the same for me and everyone else on this planet.

Here are some simple examples of "minor changes" that do NOT (and should NOT) require a system reboot:

1) Changing a fixed "Control Channel" setting in the 2.4GHz or 5GHz wireless band.
2) Changing the "DTIM Interval" & "Beacon Interval" values in the 2.4GHz or 5GHz wireless band.
3) Changing the "Time Zone" system setting.
4) Setting the "Enable Local NTP Server" option to NO and adding custom values for primary and/or secondary NTP servers.

If a full system reboot were necessary for all such minor changes, I'd say that the system is not well designed or well implemented. Yes, each of those changes does require a re-initialization of the affected service(s), so a series of "service restart" calls are usually made under the hood, and that's the way it should be. A well-designed critical system is not built as a monolithic structure where even minor changes affect the entire system to the point where a full system reboot is required. That would make the system very fragile and horrible to maintain and upgrade. A well-designed system is implemented as a structure of semi-independent services and modules that are, ideally, loosely couple as much as possible so that changing some parameterized options does not affect the whole system and, therefore, does not require a system reboot.

WRT major changes requiring a system reboot, that would depend on your definition of what constitutes a "major change." IMO, changing Username & Password is not a major change in the system. Yes, it's a "major change" to us as USERS because it's our access to the system so it's personally critical to have it working perfectly every single time; but it's not considered a major change within the embedded system because it should not affect its critical functional modules. Yes, some services which may depend on or use the login credentials would need to be re-initialized, and perhaps some "ownership permissions" may need to be reset, but a full system reboot should not be required. If a system reboot is required, it would indicate a flaw (however minor or trivial) in the software related to how such a change is handled and propagated throughout the affected services/modules. Perhaps the system was not designed with this in mind, and I can understand the oversight, but the flaw is still there.

Some will say that doing a system reboot is easy and simple to do so it's not a big deal to fix the problem, and I can certainly agree with that. However, that does not negate that there's some flaw in the way the system handles a change in login credentials as it relates to cron jobs, especially since those types of jobs tend to be scheduled because they may be important to the monitoring and/or functionality of a particular service or module.

Now, there are a couple of "mitigations" that could be implemented to, at least, try to minimize the potential effects of this particular issue on users (especially to those not technically savvy and for whom a reboot is not intuitive) without making major changes to the software architecture.
For instance:

1) In the GUI notification where it's clearly stated that the new login credentials will be required on the next login, an addition can be made to the message stating that a "system reboot is necessary for the change to take full effect." That would make it the requirement known and clear, putting the responsibility on the users to follow through with the full system reboot at their discretion (thanks to @dosborne for the suggestion).

2) In the same GUI notification, it could be also be stated that a system reboot is required and will be performed when clicking on the "OK" button. A "Cancel" button can be added as well so users can choose whether to do the system reboot right away. Again, this makes the requirement known and clear to users, and they take responsibility for following through.

I haven't looked thru the available source to understand the underlying system and know what changes may be needed to actually avoid doing the system reboot when changing the Username (which is actually the main cause for the "missing cron jobs"), but I suspect the modified GUI notifications described above as alternatives would not be a major change within the system.

Anyway, that's my perspective & opinion based on my experience as a s/w developer working on embedded systems & user applications for almost 30 years now.

 

[L&LD]

@Martinski, I am not opposed to any of your points.

Rebooting and then testing for the desired/expected results is just so much easier and less hassle when doing these things for customers or for myself.

I trust what I have built. Everything else is not left to chance as much as I can control.

 

[Martinski]

Rebooting and then testing for the desired/expected results is just so much easier and less hassle when doing these things for customers or for myself.

@L&LD,

I can understand and respect your personal stance WRT rebooting and then testing [often], which seems to come more from a system user's perspective with the ultimate goal of getting the device well tested, verified, and completed for you and your customers. I'm certainly not opposed to that, and that's the kind of attitude and thinking we look for in our own system test engineers who take our software releases and put them through the wringer to try to find any bugs, strange behaviors, or unexpected results from either the internal components or the user interface.

My perspective comes from looking at the system as both a user and s/w developer, with the latter often taking precedence because of my real-time embedded system experience, where rebooting unnecessarily often is just not considered best practice for our critical components (e.g. satellites, radar, telecoms). Most, if not all, of our customers would be violently opposed to a full system reboot every time a change is made, especially when the change (or perhaps an update) is being done to a remote system, and there's no person on site to bring it up if it fails to restart. Those critical systems need to run uninterruptedly 24/7 and with very little downtime, if any, so from the initial design we always look for ways to avoid doing full system restarts/reboots.

I realize that it's not fair to compare a consumer-grade router device (worth $100 to $400) to, say, a satellite component system because the levels of H/W and S/W design, code reviews, implementation procedures, testing & verification rigor, certification, etc. are just at very different scales (not to mention the money invested), but this type of background & experience does influence my personal perspective when using consumer-grade appliances like routers.

Lastly, I got to say that, FWIW, I find the RMerlin F/W to be a great product which I've been using for about 5 years now (my 1st ASUS router was the RT-AC68U, followed by the RT-AC86U), and any comments/posts about possible s/w bugs or design flaws are not meant to diminish his efforts and hard work in putting the F/W together all these past years.

My hat's off to you Mr. RMerlin, as a fellow s/w developer I can appreciate the amount of time & effort, the numerous late nights or the long weekends poring over source code trying to find that sneaky, pernicious bug that doesn't show up on command and only happens intermittently, but, most of all, all those glorious, magnificent days when the software works just as designed and intended, and it just purrs like a kitten...

 

[thecheapseats]

@L&LD,

Lastly, I got to say that, FWIW, I find the RMerlin F/W to be a great product...

the RMerlin firmware is trick and adds value as do the third party scripts... to be kind, the physical ASUS hardware implementation is an ongoing hobby in practice and admin... if you're an owner/user - accepting that at their price points is just the way it is...

I like drilling holes and soldering-up hacks to get the out-of band admin features I miss from real network hardware - but I'll tire of it soon and move up the appliance food chain... after ten years and several RMerlin powered routers it was fun...