Seeking Feedback & Contributions: Merlin Auto Update Solutions

[Yota]

I still don’t understand how a simple shell script would “know” there’s a problem with the firmware or wifi or vpn, and it’s now necessary to downgrade the firmware.

Internet access checks are easy to implement by having a script download a file over the Internet.

Detecting WiFi is also easy to implement. The wl command can check the running status of the WiFi driver and connected clients.

It is also possible to detect whether the VPN server/client is active.

But you're right, the idea of downgrading was introduced in the first place to deal with these potential problems.

Starting small, the first version would simply login and upload the firmware file like a person would do. What happens after that would be the same as if a human did it.

OK, I understand. Yeah I'm probably overthinking this because I did read way too many complaints at the beginning of this post and then trying to address their concerns.

The more limited the scope of a script like this, the more trustworthy it can become. Do one thing and do it well.

Yes, totally agree, do one thing and do it well.

 

[Yota]

Ok, you guys are right, we should focus on updating and then fixing the issues we have, I agree let's put the firmware downgrade aside.


Now there are two topics I look forward to seeing all of you discuss.

The first is updating across multiple versions.

Then how to protect the admin password?

@ExtremeFiretop @dave14305

 

[dave14305]

We do not rule out that people who use this script initially installed a very old version of the firmware.

I rule them out. People who are on older than N-1 firmware probably aren’t interested in (or going to be aware of) an auto-upgrade add-on.

Personally, I would not let the script upgrade any router older than 386.11 or 388.2, as a hard version floor. But that’s just my opinion.

Once they’re current via other means, they would stay current with each incremental firmware release.

Then how to protect the admin password?

We don’t even try. Anything we do to encrypt/decrypt it would be in plain sight in the script.

 

[Yota]

I rule them out. People who are on older than N-1 firmware probably aren’t interested in (or going to be aware of) an auto-upgrade add-on.

Personally, I would not let the script upgrade any router older than 386.11 or 388.2, as a hard version floor. But that’s just my opinion.

Once they’re current via other means, they would stay current with each incremental firmware release.

Good idea, when the script installs it can check the differences between the current version and the latest version, if there are more than 2 or 3 versions, then the installation will be refused.

We don’t even try. Anything we do to encrypt/decrypt it would be in plain sight in the script.

You are right, any secret we store in the script is not a secret. anyone can eventually find a way to read it, no matter how hard we protect it. But at least don't let us save clear text password that are easy to retrieve or base64 encoded that can be used directly for login.

I'm still curious about how Asus implements encrypted password? Are they just a hash of the password and not the real password? Or is it encrypted and can the real password be decrypted?

 

[dave14305]

I'm still curious about how Asus implements encrypted password? Are they just a hash of the password and not the real password? Or is it encrypted and can the real password be decrypted?

They seem to use a mix of md5, base64 and aes to do a one-way encryption, somehow using the LAN MAC address as a salt or encryption key. http_passwd is sitting (encrypted) in /data/.kernel_nvram.setting.

 

[Yota]

They seem to use a mix of md5, base64 and aes to do a one-way encryption, somehow using the LAN MAC address as a salt or encryption key.

Sounds like a mess. Why not just hash the password on the browser and just save the hash on the router?

Using these methods to mix means that there must be a weakness in the entire encryption method, and while they are actually still saving the clear text password in the router, these encryptions are just an obfuscation measure.

As far as I know, the scenarios where they must use clear text password include: SSH, SMB, FTP, and AiCloud login verification. This is probably why Asus doesn't use a more secure hashing method to store password.

All it takes is a hacker studying it to figure out a way to decrypt it, since they're just obfuscating rather than actually mathematically protecting the secret.

http_passwd is sitting (encrypted) in /data/.kernel_nvram.setting.

That might not be encrypted, but the file format is different, read here.

 

[RMerlin]

In fact, please @RMerlin also join the discussion of this question, because I think most people on this forum do not have unanimous opinions and expertise on how often should factory reset be performed, and @RMerlin is one of the few who can express professional opinions on this.

My original opinion still stands: automated updates is not a good idea, as it has no way of taking into account any exception scenario such as what you describe. Or when there's an important note that's written in the changelog, which may require user intervention or at the very least user awareness, such as when an update reduced the number of supported OpenVPN clients to 2 on the RT-AC68U.

 

[Yota]

My original opinion still stands: automated updates is not a good idea, as it has no way of taking into account any exception scenario such as what you describe. Or when there's an important note that's written in the changelog, which may require user intervention or at the very least user awareness, such as when an update reduced the number of supported OpenVPN clients to 2 on the RT-AC68U.

OK, you are right. I did forget about the RT-AC68U's feature reduction (haha, I seemed to be the only one against that at the time), and it's a case of good reason.

So I have a solution for this.


Under normal circumstances, automatic updates follow the steps I mentioned before (check for updates, wait a week, then download the firmware, verify the firmware, and update automatically)

But we will add an extra step. After discovering the update, the router accesses the project's github repository before downloading the update. The repository maintains a list of models/firmware versions that require user participation (blocklist for firmwares and models that disable automatic updates). If there is a match, automatic updates will not be performed.

So, yes, the maintainers of this project will have to pay attention to each changelog to make judgments for users. If something like RT-AC68U 386.11 happens that requires users to decide, 386.11-RT-AC68U will be add to the block list, the script will inform the user in the system log and system notifications why automatic updates have stopped, and what users need to do. once they manually upgrade to 386.11, automatic updates will work again. Until the next time a firmware version that requires manual confirmation by the user is included in the block list again.

I wonder @RMerlin do you think this solution will alleviate your concerns?

Do you guys have any other solutions for the situations RMerlin mentioned that require user intervention?

@ExtremeFiretop @dave14305



EDIT:

Another solution is to work with you (@RMerlin ). When you decide that it is not appropriate to automatically update a firmware version, you add a fixed-format message to the changelog.

for example:

386.11 (14-May-2023)
[Auto Update: this version requires user confirmation]
  - UPDATED: curl to 8.0.1.
  - UPDATED: openvpn to 2.6.3.
  - CHANGED: Reduce max OpenVPN clients to 2 for RT-AC68U and
             DSL-AC68U due to lack of NVRAM on these two
             models.  Note that existing settings are not
             automatically removed, you must run the following
             command over SSH to remove them from nvram and
             the /jffs/openvpn/ directory:

                clear_vpnclients.sh

             A backup will be saved in /jffs/openvpn_backup.tgz.

  - FIXED: DDNS was being refreshed every time IPv6 bound6() event
           occured even if IPv6 DDNS update was disabled.
  - FIXED: Wireless Log wouldn't properly show IPv6 address
           for clients with multiple addresses.
  - FIXED: dnsmasq may crash if no DNS server is configured
                (fix backported from dnsmasq upstream)

The automatic update script will not automatically update after matching the content ([Auto Update: this version requires user confirmation]). Instead, as I mentioned before, it will trigger a notification that requires user intervention.

 

[dave14305]

After discovering the update, the router accesses the project's github repository before downloading the update. The repository maintains a list of models/firmware versions that require user participation (blocklist for firmwares and models that disable automatic updates). If there is a match, automatic updates will not be performed.

Probably a bad idea because it will rely on one person with commit access to maintain such a list in a timely manner. In the long run, all these scripts become abandoned (and in the longer run, we’re all dead).

Do you guys have any other solutions for the situations RMerlin mentioned that require user intervention?

Users don’t read changelogs anyway in real life. The script can download the release note and email it to the user if they have setup email in amtm. But most probably would not have done it to begin with.

Another solution is to work with you (@RMerlin )

LOL. He wants nothing to do with this, or else he would have done it himself. He won’t likely endorse it in any way, nor make special accommodations for it.

Can’t we just hack along and see how far we get before deciding we have to account for every corner case and remote router in a cabin in the deep woods?

 

[Yota]

Probably a bad idea because it will rely on one person with commit access to maintain such a list in a timely manner. In the long run, all these scripts become abandoned (and in the longer run, we’re all dead).

I initially considered an allow list, but that would be much more difficult to maintain since each firmware would need to be approved. Blocklists have no such concerns. If we die, automatic updates will go on forever lol.

Users don’t read changelogs anyway in real life. The script can download the release note and email it to the user if they have setup email in amtm. But most probably would not have done it to begin with.

Yes.

LOL. He wants nothing to do with this, or else he would have done it himself. He won’t likely endorse it in any way, nor make special accommodations for it.

Well, I just had to put my thoughts out there to see what he and others thought about it.

Can’t we just hack along and see how far we get before deciding we have to account for every corner case and remote router in a cabin in the deep woods?

Yes, let's move on! hahaha. Hopefully by Christmas we'll have a stable script so a lot of people can go to their cabins and install this script there. HAHAHAHHAHA

 

[Tech9]

He may have similar usage scenarios.

I do have VPN server in another country, it used to be an Asus router, but was replaced by something else after Asus screwed up the ASD update.

 

[RMerlin]

I'm not adding any further steps to my existing workload. I've been trying to reduce it over the years, not add to it, sorry.

 

[sfx2000]

Yes, let's move on! hahaha. Hopefully by Christmas we'll have a stable script so a lot of people can go to their cabins and install this script there

I still think this is a very bad idea...

Even in managed environments, things can go wrong - there, the provider has to eat the cost of replacements...

This is not a managed environment - @Yota @ExtremeFiretop @dave14305 - are you willing to pay up for any bricked devices?

 

[ExtremeFiretop]

Ok, you guys are right, we should focus on updating and then fixing the issues we have, I agree let's put the firmware downgrade aside.

The first is updating across multiple versions.

Then how to protect the admin password?

@ExtremeFiretop @dave14305

I rule them out. People who are on older than N-1 firmware probably aren’t interested in (or going to be aware of) an auto-upgrade add-on.

Hey team!

Sorry for the delay, work was nuts today!

I fully agree with this by dave. Once they’re current via other means, they would stay current with each incremental firmware release of our script.

I don't know many people going out of their way to setup a router without first updating it, however we can try to implement as many safeties as possible, (such as a hard min version) but look, at the end of the day, an update is always the responsibility of the user.
I am not sure how many times we've said this, but naturally if your service is *relying* on some shacks VPN connection out in the woods, then maybe auto update isn't for you, no problem, no one is forced to use it.

I expect people to display some understanding of the risks involved with your network and decide accordingly. Same as stock firmware with it being enabled or disabled.
Simply giving the users the option is not enough of a reason for me not to be interested.

I can guarantee my home service is not so sensitives as a banks, etc, and I have the confidence that if I initiate a firmware update over the web GUI, for a single version increase, it should be fine 98% of the time.

These hypotheticals where a company like SourceForge is hijacked is nice to think about, and develop some understanding of the risks before moving forwards is required!.
But if that really happened and someone came to me complaining, I would simply laugh. How is SourceForge being hihacked my problem?

It really isn't, using my script or not, if it was hijacked you likely downloaded (automatically or manually) compromised firmware. As long as it's offered on SourceForge, that risk has already been calculated in the project.
As already discussed a delayed release cycle will help combat this situation.

EDIT:

Another solution is to work with you (@RMerlin ). When you decide that it is not appropriate to automatically update a firmware version, you add a fixed-format message to the changelog.

for example:

386.11 (14-May-2023)
[Auto Update: this version requires user confirmation]
  - UPDATED: curl to 8.0.1.
  - UPDATED: openvpn to 2.6.3.
  - CHANGED: Reduce max OpenVPN clients to 2 for RT-AC68U and
             DSL-AC68U due to lack of NVRAM on these two
             models.  Note that existing settings are not
             automatically removed, you must run the following
             command over SSH to remove them from nvram and
             the /jffs/openvpn/ directory:

                clear_vpnclients.sh

             A backup will be saved in /jffs/openvpn_backup.tgz.

  - FIXED: DDNS was being refreshed every time IPv6 bound6() event
           occured even if IPv6 DDNS update was disabled.
  - FIXED: Wireless Log wouldn't properly show IPv6 address
           for clients with multiple addresses.
  - FIXED: dnsmasq may crash if no DNS server is configured
                (fix backported from dnsmasq upstream)

The automatic update script will not automatically update after matching the content ([Auto Update: this version requires user confirmation]). Instead, as I mentioned before, it will trigger a notification that requires user intervention.

I like the sentiment, but I wouldn't have even asked, clearly if merlin had interest in auto update he would of done it himself or displayed interest in the project once we got it working through Web UI.
This really is going to be a fun project of troubleshooting our way through most of the steps, including this one.

Can’t we just hack along and see how far we get before deciding we have to account for every corner case and remote router in a cabin in the deep woods?

Love it! On the same page! Move on The point at the end of the day is if we can get this to work reliability.

Something I or you can use personally at home, where I can trust 98% of the time it will be fine, and the 2% where something could potentially happen (such as hijacked SourceForge or a bad/broken Final release)
I will have a USB with backup tools to restore quickly and effectively, and that still cuts down my human interaction by most of the time.

I did make note though of some needed hard coded min version.

 

[dave14305]

are you willing to pay up for any bricked devices?

No sir. Why are you assuming this is going to turn into some mass-distributed feature? Any potential users are going to be nerd's nerds and seek it out and install it knowingly, understanding the risks.

 

[Tech9]

I'm interested in bricked heavy routers. I run electronics recycling business, goes by weight.

 

[ExtremeFiretop]

I'm interested in bricked heavy routers. I run electronics recycling business, goes by weight.

Oh really?

What was the address to ship stuff too?

My 26 story government office is renewing their printer fleet.
I'm pretty sure that should meet your weight requirement.

I'll make sure to send the hundreds of dead printers there

 

[Viktor Jaep]

I'm interested in bricked heavy routers. I run electronics recycling business, goes by weight.

I will beat @Tech9's price by offering 50% more per lb/kg! Any day! Crazy Viktor's Electronics Recycling, Inc.!

 

[Tech9]

My 26 story government office is renewing their printer fleet.

You don't have to ship anything. I can explain how it works on private, if interested.

Crazy Viktor's Electronics Recycling, Inc.

I did donate 12x AIO routers last year. You can't qualify as business and outside of the country. If you are really interested in some free development hardware though, let me know and I'll watch for something potentially interesting and ship it to you. Let me know on private what to watch for.

 

[Viktor Jaep]

I did donate 12x AIO routers last year. You can't qualify as business and outside of the country. If you are really interested in some free development hardware though, let me know and I'll watch for something potentially interesting and ship it to you. Let me know on private what to watch for.

Definitely outside of country. We actually do most our recycling efforts in space, near-earth orbit. There's a lot of junk out there, but Crazy Viktor's Electronics Recycling, Inc. takes care of business!