Seeking Feedback & Contributions: Merlin Auto Update Solutions

[L&LD]

Right.. But any change to the GUI would just fail on the curl request and we would update the curl request. It wouldn't be dangerous. It would just fail like a failed http web request. I don't know many failed web requests that are "dangerous" do you? Usually the page just refreshes or times out? etc?

So the safety argument doesn't hold much fruit here anymore too the remaining testing this.

If something failed with old hnd-write, then you had a more serious concern. This method is easy to work with, since we are just scripting the Web GUI process and still invoking all the same safety measures as the GUI normally does.

That is the assumption, right now.

To put this in context, I don't agree with or enable anything Asus does with regard to 'auto-updates' either. And they're in full control of the process. From the start, under the hood, to the user view side. Because even there it isn't 100% safe (proven, in these very forums).

Keep plugging away. Like I've wished you before. Glad you didn't give up.

 

[glens]

The hack? Use nohup on the curl.

nohup curl 'http://ww ... ad_response.txt 2>&1 &

Why the ampersand at the tail end? If there's more to do later in the script, doesn't it RELY on the file being wholly present? If nothing follows (haven't looked at the script) then what's wrong with the script blocking for the duration?

There was something on page 9 I'd thought to address but it's not jumping out at me during a quick scan now... Perhaps that's where I saw the iteration of the *nix philosophy of "do one thing and do it well." Something I wholeheartedly embrace.

Regarding the "not enough space" issue, I have to ask "where do you think tempfs comes from?" Seems typical it's half the RAM here. There's more than one in use. If you think you can fill them both to declared capacity and still use anything else, you deserve a cookie or some other prize.

As for my personal opinion about this whole thing, I'd never even use the /native/ autoupdate from a remote location and have spent more time reading through this thread already than it would take to manually update, with hard resets each, for the next 5 years! But that's not to say I never waste time chasing ideas like this myself otherwise...

I know you kids are having fun, and it's keeping you out of other trouble, so carry on.

 

[dave14305]

Why the ampersand at the tail end? If there's more to do later in the script, doesn't it RELY on the file being wholly present? If nothing follows (haven't looked at the script) then what's wrong with the script blocking for the duration?

Once the upload completes, the firmware takes over and the life of the script can no longer be guaranteed.

 

[ExtremeFiretop]

That is the assumption, right now.

To put this in context, I don't agree with or enable anything Asus does with regard to 'auto-updates' either. And they're in full control of the process. From the start, under the hood, to the user view side. Because even there it isn't 100% safe (proven, in these very forums).

Keep plugging away. Like I've wished you before. Glad you didn't give up.

I understand your point now your saying it's more reliable doing it manually, I don't disagree on that front. (Keeping in mind we are still working on making it as reliable as possible)
I just wanted to make the point that currently the troubleshooting is easy and not dangerous on the curl, because if we don't construct the request correctly the UI simply refreshes like an invalid web request.

Incoming curl patches hopefully very soon to shore up some critical bugs...

Patches Prepared for 'Probably Worst' cURL Vulnerability

A high-severity vulnerability in the data transfer project cURL will be addressed with libcurl and curl updates this week.

www.securityweek.com

Recently and relevant news! I like it!
Thank you, was a valuable read.

Why the ampersand at the tail end? If there's more to do later in the script, doesn't it RELY on the file being wholly present? If nothing follows (haven't looked at the script) then what's wrong with the script blocking for the duration?

Once the upload completes, the firmware takes over and the life of the script can no longer be guaranteed.

I even tried playing with a way to keep the script open for some reporting after the last curl, and as discussed in this thread, the best thing I found is to make sure the curl is the last function on the actual flash since as Dave says anything after the curl/upload is started cannot be guaranteed, the system shuts down many things in the flashing process to be safe/conserve memory it seems.
Basically anything we need to do for the flash has to be handled before the last curl.

But that's not to say I never waste time chasing ideas like this myself otherwise...

I know you kids are having fun, and it's keeping you out of other trouble, so carry on.

Thank you Glens It makes me happy to know that while you may not find value in the script itself, that you find some of the information value at the very least.

 

[glens]

Once the upload completes, the firmware takes over and the life of the script can no longer be guaranteed.

I follow that. Backgrounding the curl call has no effect when it's the last line of the script, unless when its parent, the shell, then closes and takes it with, which evidently was occurring. Telling backgrounded curl not to hang up in that case would would be all the same as simply 'exec'ing it in the foreground instead, right? I mean, unless one wanted to free up some memory by letting the interpretor close and be cleaned-up-after, there'd be no harm in letting curl block 'til done from within. That's all I'm saying.

 

[glens]

I even tried playing with a way to keep the script open for some reporting after the last curl, and as discussed in this thread, the best thing I found is to make sure the curl is the last function on the actual flash since as Dave says anything after the curl/upload is started cannot be guaranteed, the system shuts down many things in the flashing process to be safe/conserve memory it seems.
Basically anything we need to do for the flash has to be handled before the last curl.

I'd hope the firmware doesn't start shutting stuff down until it's ascertained the validity of the payload. Did you try adding something like, instead of the ampersand

 || { sleep 3; echo "upload unsuccessful"; }

after?

 

[ExtremeFiretop]

I'd hope the firmware doesn't start shutting stuff down until it's ascertained the validity of the payload. Did you try adding something like, instead of the ampersand

 || { sleep 3; echo "upload unsuccessful"; }

after?

The problem is the script is run as user interactive or as a background script triggered by a cron (non-user interactive)
If we dropped the user interactive requirement we could drop nohup, but if we want it to function when the user calls the script in an interactive fashion, then we need the nohup for it to complete successfully.

If the script is run user-interactively on SSH, and the user triggers an update, the firmware update shuts down ssh session at about 4% of the upload.
Causing the script to fail the rest of the upload. (Since the script was running in an interactive session which is now closed.)

If it's not run user-interactively, then it succeeds.

Was discussed here:

Great, you found a breakthrough.

You don't need to add nohup in the final user-facing script, because the SSH session being killed only happens during our debugging, when the script is invoked by a custom script or a cron job, their parent process is not dropbear, so it will not be killed.

The nohup is redundant when the script run by the cron job, but required when running user-interactive mode to keep the curl alive when when the ssh session is closed.

If the script relies on interaction over SSH then retained nohup will be necessary.

But if it's going to be a purely non-interactive script, where updates will only be triggered by custom scripts and cron, then nohup may be redundant. and, I'm not sure when nohup became available again because it never worked in my memory (2017-2023).

 

[glens]

On second thought, that (what I'd last posted) would only announce the failure of the curl call itself (which would be informative). Probably would omit the braces, semicolons, and the sleep time. Then on another line add a sleep and then another echo declaring the script itself had against expectations proceeded beyond the firmware validation / loading.

 

[glens]

the firmware update shuts down ssh session at about 4% of the upload.

I guess I missed that part. Seems awfully premature...

 

[ExtremeFiretop]

I guess I missed that part. Seems awfully premature...

No worries, I appreciate the questions and interest in how the code functions now

Agreed 4% does seem premature, however this nohup was simply due to the restrictions of the environment, it was something we had to factor in.

 

[Yota]

Once the upload completes, the firmware takes over and the life of the script can no longer be guaranteed.

There are several ways to ensure that the script continues to run after uploading.

Use cron job. the update part of the script is called by the cron job.

The minimum cron time interval is 1 minute. If the user uses SSH to interact with the script, it may wait up to 1 minute before updating.

Or create a service-start script and then restart an insignificant service to trigger the script.

 

[Yota]

Regarding the "not enough space" issue, I have to ask "where do you think tempfs comes from?" Seems typical it's half the RAM here. There's more than one in use. If you think you can fill them both to declared capacity and still use anything else, you deserve a cookie or some other prize.

Testing confirmed that even for extreme models such as the RT-AC68U, automatic updates work smoothly with the aid of a USB drive.

The firmware of RT-AC68U consists of two files, and the appropriate version will be selected when updating, so its firmware has a size of more than 90MB, and its RAM is only 256MB, leaving us with /tmp space, as you said, just half the RAM, only 128MB. It is impossible to download the zip in /tmp of RT-AC68U and unzip the firmware in place. For such an extreme model, a USB drive must be used to complete the automatic update.

In previous testing, since it was not filled (93MB/128MB), but the error was triggered, filled it just to reproduce the error.

 

[Yota]

I understand your point now your saying it's more reliable doing it manually, I don't disagree on that front. (Keeping in mind we are still working on making it as reliable as possible)
I just wanted to make the point that currently the troubleshooting is easy and not dangerous on the curl, because if we don't construct the request correctly the UI simply refreshes like an invalid web request.

Asus once provided broken firmware during automatic updates and bricked many RT-AX86Us. So that's what he and others are afraid of, but in that case it was the firmware that was broken, not the auto-update itself, in fact if the auto-update was broken, that broken firmware would never be flashed to their router XD

 

[Yota]

3004.388.4 (21-Aug-2023)
  - NOTE: In preparation for the new 3.0.0.6 codebase, the version
          string will now start with 3004 or 3006 to match with
          upstream.

It should be pointed out that the firmware version number may contain the following different formats. The script must be able to determine the firmware version and set the appropriate minimum version.

• 3004.388.4
• 3004.388.4_2
• 3006.xxx.x
• 388.3
• 386.12

 

[L&LD]

No, that is not what I am 'afraid' of. At least, not only that.

Auto Updates: dumb mechanical method to upgrade to the latest firmware. Never a good idea, IMO.

Current update method:

• Read about new firmware released (RMerlin's, in my case).
• Read all forums (in full), about how many have updated, what issues were discovered, and what steps are required to make it 'right'.
• Download the firmware and install it on my test router(s) if the above research warrants it.
• Do my tests on those routers and decide, based on observing the network performance, stability, responsiveness, and any other data points I track, if this warrants running the firmware onto my main router.
• Flashing the firmware onto a main router on my own network will still be put off until I have enough time to jump as needed, if needed, to get my network up and running again if a catastrophic event due to the firmware, were to happen. This includes having a spare router (or two) to quickly get online again.

When an Auto update includes the above, I may begin to consider using it. Depending on other's reported experience with it too (for an extended time period), of course. That may happen when AI is embedded into our router/networking hardware.

While I offer encouragement and celebrate each 'win' here along with everyone else, I fundamentally do not believe in 'auto' anything. Those options are for the masses who can't be bothered to learn, or who can't even understand why they should (learn), to do it themselves. Not that it makes it 'okay' in that case, but it does help, at large.

A one-time broken firmware may seem like a small thing to be 'afraid' of. I'm not worried about that.

What I'm worried about is a system/process where millions and millions of routers are auto-updated by default. And then learning someone hacked the server(s) that were delivering infected firmware that bricked those routers, or worse.

I will repeat myself, I don't ever see myself or my customers using this 'feature', ever. No matter if RMerlin, Asus, or anyone else is 'backing' it. Just seems like an illogical way to give up control of your network to me.

 

[Yota]

No, that is not what I am 'afraid' of. At least, not only that.

Auto Updates: dumb mechanical method to upgrade to the latest firmware. Never a good idea, IMO.

Current update method:

• Read about new firmware released (RMerlin's, in my case).
• Read all forums (in full), about how many have updated, what issues were discovered, and what steps are required to make it 'right'.
• Download the firmware and install it on my test router(s) if the above research warrants it.
• Do my tests on those routers and decide, based on observing the network performance, stability, responsiveness, and any other data points I track, if this warrants running the firmware onto my main router.
• Flashing the firmware onto a main router on my own network will still be put off until I have enough time to jump as needed, if needed, to get my network up and running again if a catastrophic event due to the firmware, were to happen. This includes having a spare router (or two) to quickly get online again.

When an Auto update includes the above, I may begin to consider using it. Depending on other's reported experience with it too (for an extended time period), of course. That may happen when AI is embedded into our router/networking hardware.

While I offer encouragement and celebrate each 'win' here along with everyone else, I fundamentally do not believe in 'auto' anything. Those options are for the masses who can't be bothered to learn, or who can't even understand why they should (learn), to do it themselves. Not that it makes it 'okay' in that case, but it does help, at large.

A one-time broken firmware may seem like a small thing to be 'afraid' of. I'm not worried about that.

What I'm worried about is a system/process where millions and millions of routers are auto-updated by default. And then learning someone hacked the server(s) that were delivering infected firmware that bricked those routers, or worse.

I will repeat myself, I don't ever see myself or my customers using this 'feature', ever. No matter if RMerlin, Asus, or anyone else is 'backing' it. Just seems like an illogical way to give up control of your network to me.

Frankly speaking, you are a professional user, the potential users of auto-update may be those who are too lazy to update, they may be running old firmware from 1 year ago or even 3 years ago, because they don't have the time or want to do the steps you do every 4 months.

The security benefits of automatic updates outweigh the potential risks.

But this is an open source project, and users choose whether to use it or not, and can customize the code in their own way.

So we're not forcing people to use this feature like Asus does (Yes Asuswrt's auto-updates used to be enabled by default and couldn't be turned off). we also wouldn't shock you if it was powered by a cat under the hood.

Because everything is open, public and voluntary.

People choose to use this script because they have a need, you don't choose because you have a better way to update the firmware, but you can't force everyone to treat a router as carefully as you do.

Please note that no animals were harmed in this project

So, have a nice day.

 

[L&LD]

No, I'm not suggesting my method is the only right method, nor that there are no benefits (as I already alluded to) in having routers be on a semi-current version either.

Just trying to make clear my perspective and the logic behind it.

To me, it is perplexing why anyone would want to have this script installed. Because, as you said, a higher level of user is required to even know about this script on firsthand. But that may just be me. I know it takes all kinds to make the world turn, but if I didn't believe I was 'right', I'm not true to myself, and others.

 

[Yota]

No, I'm not suggesting my method is the only right method, nor that there are no benefits (as I already alluded to) in having routers be on a semi-current version either.

Just trying to make clear my perspective and the logic behind it.

To me, it is perplexing why anyone would want to have this script installed. Because, as you said, a higher level of user is required to even know about this script on firsthand. But that may just be me. I know it takes all kinds to make the world turn, but if I didn't believe I was 'right', I'm not true to myself, and others.

You have indeed helped many people on this forum. including me.

In the early days, I was having issues with my router and I, like other newbies, posted here to complain.

But you taught me how to diagnose the problem, troubleshoot it, and then to this day I also do a factory reset every time I update the firmware and then spend 2 hours manually setting everything up again (including manually entering dozens of static IP lists).

So, I owe you a huge thank you. Thank you, and I know your recommended steps will help many more people as well.

A few years ago, I was tired, so I tried to find a way to automate factory reset and resetting after every update, but that quest ultimately failed.

This post reminded me of something I once thought about trying, and then it could significantly reduce the burden on users to maintain their routers, of course the consequence of this is that they might have more problems, but I'm open to it, And be ready to help them find solutions to their problems, just like you helped me.

I don't have the time to educate people and help them as much as you do, but I do hope that I can contribute back to this excellent forum in some way.

So, that's it, how about give us a chance and see what we can do, okay?

 

[L&LD]

Already open to that as every post in this thread has stated.

Go!

 

[Yota]

Already open to that as every post in this thread has stated.

Go!

Thank you!