Request: Compile firmware using GitHub Actions

[Yota]

Simply put, Actions is a function provided by Github, which can compile the required code to binary files directly on the Github cloud, supporting all languages and platforms, without downloading the repository and configuring the local compilation environment by yourself. It's mature enough now, and the free version offers ample usage quota. I've seen some forks of OpenWRT compile using this form.

If we can, anyone can fork Merlin's repository to compile the latest changes at any time, and since each compilation is done in a completely clean environment, there will be no compilation problems caused by errors in the compilation environment configuration.

The free version has ample time of 6 hours per build (2 Cores, 7G RAM, 14GB SSD) and can build 20 different firmwares/models at same time.

I think compiling firmware is the biggest problem for non-professional developers who want to contribute to the project. If you want to test changes, you must configure the whole compilation environment by yourself, then download and try to compile. If you're not familiar with these, or just have bad luck, you may run into compile errors. You may need to spend a few hours troubleshooting the build environment. We do see a lot of questions about compilation on the forums, and if all these things are handed over to GitHub, we just need to modify the code, then a few mouse clicks in the browser, wait, and then download the compiled firmware, it all becomes very simple, many non-professional developers can benefit from it.


Learn more: https://docs.github.com/en/actions/learn-github-actions
About quotas: https://docs.github.com/en/actions/learn-github-actions/usage-limits-billing-and-administration
About performance and environment: https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners


What we need to do is provide the configuration files needed for compilation based on how Actions work, without modifying the code.
How to create the workflows configuration file required for compilation see here: https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions

 

[dave14305]

RMerlin didn’t think it was doable.

amcfwm - AsusWRT-Merlin Custom Firmware Manager

Thanks for pointing it up. I was not aware self-compiled firmware did not have 100% of the functionality. If I understood correctly the images compiled by RMerlin do have all functionality enabled, correct? The main reason I am complining my own firmware is to generate some drivers for the...

www.snbforums.com

 

[Yota]

RMerlin didn’t think it was doable.

amcfwm - AsusWRT-Merlin Custom Firmware Manager

Thanks for pointing it up. I was not aware self-compiled firmware did not have 100% of the functionality. If I understood correctly the images compiled by RMerlin do have all functionality enabled, correct? The main reason I am complining my own firmware is to generate some drivers for the...

www.snbforums.com

I know some proprietary code is missing from the GitHub code and never contains the signatures used by Asuswrt and Merlin, but that doesn't mean it won't compile.

What I mean is that a build environment can be provided, which also allows others to fork the repository and add the necessary files to compile without having to add those files locally.

And RMerlin can still compile his version natively because of that signature.

The advantage of this is that more people can participate in the development of Asuswrt-Merlin, especially those who are not familiar with the compilation process and developers, which is very meaningful.


Even if the compiled firmware lacks these proprietary components and code, I think it's valuable to a lot of people because the real contributors are always focused on perfecting the open source parts rather than trying to hack the closed source components. All I wish was that it could be compiled, not that I need to download the whole repository, then look online for the Asuswrt source code that hasn't removed the secret, then compare the differences and replace the missing files, then compile, which is for too many people a pain.

 

[RMerlin]

As of 386.5, the repo contains everything required for people to build their own fully-working firmware.

The free version has ample time of 6 hours per build (2 Cores, 7G RAM, 14GB SSD) and can build 20 different firmwares/models at same time.

That's not gonna cut it. A source tree where you have completed a build takes 12 Go (checked with an RT-AX88U). Add to that the OS, the toolchain crosscompile tools, and so on.

An RT-AX88U takes a bit over 20 minutes to build on my Zen3-based 20-threads 16 GB RAM VM. 2 threads is going to take forever, and you will have (as of 386.5) 14 different firmware to build, each of them off a clean tree. You will be compiling non-stop on such a build environment...

 

[Yota]

As of 386.5, the repo contains everything required for people to build their own fully-working firmware.

Glad to know this, and glad that you have made significant progress in negotiations with Asus, which is very good news for the open source community, thank you so much for what you do! and I believe it will bring returns to Asus as well.

That's not gonna cut it. A source tree where you have completed a build takes 12 Go (checked with an RT-AX88U). Add to that the OS, the toolchain crosscompile tools, and so on.

An RT-AX88U takes a bit over 20 minutes to build on my Zen3-based 20-threads 16 GB RAM VM. 2 threads is going to take forever, and you will have (as of 386.5) 14 different firmware to build, each of them off a clean tree. You will be compiling non-stop on such a build environment...

Please forgive me for not explaining clearly, I'll use "Virtual Machine" as this is easier to understand, although this may not fully explain Github Actions.

I meant GitHub free version allows up to 20 virtual machines at the same time, each virtual machine is equipped with Dual-core 2.1-2.4GHz CPU [1] and 7GB RAM and 14GB SSD [2], of course each virtual machine needs reconfigure before starting, but each virtual machine has 6 hours of usage. And since they can run at same time, I guess it's more efficient than building locally.

For the build environment configuration of each virtual machine, this is the role of the workflows.yaml configuration file [3]. Only one configuration of this needs to be written, and the virtual machine will prepare the required build environment according to the configuration file each time it is started.

No matter how resource-intensive this is, it is feasible to configure the build environment and build a firmware that takes no more than 6 hours. So I thought 6 hours was quite enough.

 

[RMerlin]

and 14GB SSD

That`s too small. The OS itself will probably require 4-6 GB of disk space. The toolchain takes 1.8 GB. And to fully build an RT-AX88U requires 12 GB of disk space. That means in total you'll need about 20 GB of SSD space per build environment.

 

[Yota]

That`s too small. The OS itself will probably require 4-6 GB of disk space. The toolchain takes 1.8 GB. And to fully build an RT-AX88U requires 12 GB of disk space. That means in total you'll need about 20 GB of SSD space per build environment.

No, I think GitHub Actions' 14 GB SSD is free space, the OS doesn't count. In addition, they provide systems such as Ubuntu 20 LTS or Ubuntu 18 LTS.

As the GitHub Actions help documentation says Each job in a workflow executes in a fresh instance of the virtual machine, this means that each virtual machine has a separate 14 GB space. And many compilation dependencies are already built into the system, no need to reinstall, these components will not be counted in the 14 GB free space, learn more about the software installed in the build environment: https://github.com/actions/virtual-environments/blob/main/images/linux/Ubuntu2004-Readme.md

 

[RMerlin]

No, I think GitHub Actions' 14 GB SSD is free space, the OS doesn't count. In addition, they provide systems such as Ubuntu 20 LTS or Ubuntu 18 LTS.

As the GitHub Actions help documentation says Each job in a workflow executes in a fresh instance of the virtual machine, this means that each virtual machine has a separate 14 GB space. And many compilation dependencies are already built into the system, no need to reinstall, these components will not be counted in the 14 GB free space, learn more about the software installed in the build environment: https://github.com/actions/virtual-environments/blob/main/images/linux/Ubuntu2004-Readme.md

The toolchain is not part of Ubuntu. You need to check it out off its own github repo. That's an extra 1.4 GB that needs to be added on top of the 12 GB space needed for compiling (A bit less if you were to just copy the toolchain without using the github repo, and only storing the binaries).

 

[Yota]

The toolchain is not part of Ubuntu. You need to check it out off its own github repo. That's an extra 1.4 GB that needs to be added on top of the 12 GB space needed for compiling (A bit less if you were to just copy the toolchain without using the github repo, and only storing the binaries).

I see, it seems space is the problem, maybe we can optimize the code, precompile some files, like a toolchain?

Of course, even if everything we do can be limited to 14 GB, we have to switch to local or paid GitHub Actions someday, given future code growth. It seems that choosing GitHub Actions is not a good choice

 

[RMerlin]

I see, it seems space is the problem, maybe we can optimize the code, precompile some files, like a toolchain?

Of course, even if everything we do can be limited to 14 GB, we have to switch to local or paid GitHub Actions someday, given future code growth. It seems that choosing GitHub Actions is not a good choice

For such a large project, you are better off hooking to some other type of CI integration, or simply running a VPS that can do automated builds for you. A 5$ VPS might be enough if you are just dealing with one ore two models.

Anyone expecting to get nightly builds out of something like this however may be disappointed. Sometimes development gets shifted to a separate branch for a while, for instance. This project isn't always developed as linearly as your usual FOSS project.

 

[JGrana]

As of 386.5, the repo contains everything required for people to build their own fully-working firmware.

That is indeed good news.
Is amcfwm still usable to build a full copy?

 

[RMerlin]

That is indeed good news.
Is amcfwm still usable to build a full copy?

Yes.

 

[Yota]

For such a large project, you are better off hooking to some other type of CI integration, or simply running a VPS that can do automated builds for you. A 5$ VPS might be enough if you are just dealing with one ore two models.

Anyone expecting to get nightly builds out of something like this however may be disappointed. Sometimes development gets shifted to a separate branch for a while, for instance. This project isn't always developed as linearly as your usual FOSS project.

ok i got it, thanks for the explanation