MerlinWRT - compile instructions?

[kvic]

@sfx2000

Perhaps you can also include this ARM toolchain that I created three years ago for AsusWRT 56U/68U/87U in your VM image.

It's a drop-in replacement of the stock toolchain and provides NPTL and a little bit extra performance.

 

[sfx2000]

@sfx2000

Perhaps you can also include this ARM toolchain that I created three years ago for AsusWRT 56U/68U/87U in your VM image.

It's a drop-in replacement of the stock toolchain and provides NPTL and a little bit extra performance.

Still thinking about things - seems to be some interest from the community for a reference image - but @RMerlin makes a great point about who will support it - I've got some free time now, but who knows in the future.

Once the image is built and confirmed good - good progress there - but once things are working, how to distribute? I'm not presently in a situation where I can host download - I could seed a torrent perhaps...

 

[mokodi]

Docker is not really a good choice here because of multiple paths and the like, and for many folks where Linux is not their primary OS, the subtle aspects of getting docker up and running is likely more work than it is worth.

I was thinking about maybe building a VirtualBox box VM image with everything needed, and everything in it's proper place with a reproducible build environment.

Hit "like" if you're interested...

I think there are a few challenges with the approach of distributing a VM image. There's just no good way to version control or distribute. Instead of downloading a <1MB text file everyone has to download the full image which is hundreds of MBs. Even minor updates to the VM image will require redownloading the entire image again.

It's also going to be a challenge for the next maintainer because there won't be an easy way to figure out what packages were installed and what configuration files were changed. Sure they can keep building off the existing image but what happens if there's an OS upgrade or other big changes?

You can work around that by providing an image and documenting all the steps but by that point you might as well go full infrastructure as code. Someone has already posted Vagrantfiles on Github (it's in the wiki). It's a few years old and probably doesn't work but I think it's a better approach than trying to distribute VM images.

 

[sfx2000]

You cannot heavily customize a build. Too many precompiled components, these will not work if various settings are changed at the kernel or at the build level. That's why for instance AiMesh cannot be fully disabled at build time, or AiProtection cannot be removed either.

Asuswrt is not designed to be end-user customizable. It's a proprietary product developed in-house by Asus for their specific devices, not an open sourced project like DD-WRT where everyone can customize everything at build time. It just happens to contain lots of open-sourced parts, allowing someone to recompile it, and do some tweaks here and there.

If you need something fully customizable, your only alternative right now is OpenWRT or DD-WRT.

I absolutely agree - if one has taken time to dig under the hood of AsusWRT, it's a bit of a hot mess there - step carefully or things break in unexpected ways.

Responding to other comments in this thread -

1) adding OpenWRT/DDWRT to the image - erm, no, the AsusWRT build environment and the purpose here, OpenWRT and DDWRT are out of scope - let's just build asuswrt-rmerlin

2) Alt-toolchains for AsusWRT-RMerlin - again, out of scope for this project - note the "hot mess" comment above and how to maintain and manage/support it

3) Financial Support - there's no desire for compensation, as there's no guarantee that one might be able to provide support moving forward - I have no objections, and I would suggest that any contributions go in a similar direction to the AsusWRT-RMerlin thread - this is my contribution.

 

[Starbuck99]

@sfx2000, if you have some spare time, reading all the comments in this section, why don't you just re-write a proper guide from a to z with the latest LTS Ubuntu build?

One guide specifically for legacy and a separate guide for HND. Not a combination guide which might also be confusing.

 

[RMerlin]

Or make distribution easier: just create a script that will download and setup the toolchain and required packages on top of a pristine Ubuntu 18.04 LTS system. Setting up a Linux VM is really easy, anyone unable to do something as trivial should NOT be messing with firmware customization.

Such a script would be really trivial to do really:

sanity check - Ubuntu version
sanity check - ensure not doing this as root (building as root does not work with HND for starter)
apt install xxx yyy zzz
cd ~
git clone toolchain
git clone repo
cp -ar repo buildtree
set up toolchain symlinks

There you go. Ubuntu system turned into a functional build environment, without having to distribute a 5 GB VM.

 

[Starbuck99]

Thanks @RMerlin.
Now we novices just need one of the guru's on this forum to write the script and post it.....

 

[sfx2000]

There you go. Ubuntu system turned into a functional build environment, without having to distribute a 5 GB VM.

I could - don't really have skin in this game - not my distro

I think one of the challenges here is the host distro, and the packages there - some are specifically called out, or things break.

The whole VM idea was to freeze the host build environment to something that actually works - it's not so simple as git clone this and that, cp a toolchain, set up the symlinks - and going from Ubuntu 14.04 to 16.04 to 18.04 (and that's just the LTS versions) - all the support libs can and do change and break things.

Yeah, one can do a bash/vagrant/chef/puppet script... that's a bit beyond my free time to support all the dependencies -

Building a static VM and sorting the build platform, that's a contribution to some where building AsusWRT might be a bit over their heads - one might know code - but the AsusWRT platform is a bit complicated (e.g. hot mess) and very brittle in some areas.

 

[Starbuck99]

Well, there are a few options on the table, however, I am still stuck with my original question.

We still have no workable build environment for the community.
Is someone in this community going to help us out? Whether a VM or a script, I really don't care. As long as we have the ability to compile and play a little around with the code.

Or is the compilation only going to be a mystery in the hands of the few?

 

[kfp]

Docker is not really a good choice here because of multiple paths and the like, and for many folks where Linux is not their primary OS, the subtle aspects of getting docker up and running is likely more work than it is worth.

Docker actually works alright for compiling, and pretty mature on all platforms (Linux, macOS, Windows) at this point.

I support the notion of this endeavour and am willing to infuse vast sums of capital in order to build on demand as well as remove modules like LPD , NAS, u2ec and the puppies that infest the asuswrt.

That'll be easier if all components are open-sourced, randomly removing binaries you are basically praying that one of the closed-sourced components is not expecting it (and you're thorough enough to check all the available source code that might be referencing it). It'll probably cause some instability.

Or is the compilation only going to be a mystery in the hands of the few?

It's not some secret sauce that needs to be guarded per se, just the associated time and responsibility required for maintaining it and answering people's questions about it. The information in the wiki, while outdated, is enough to get you started. If you feel that's way over your head, well, these kind of errors are exactly what you'll encounter once you start messing with the code anyways, so you'll be stuck again.

 

[mokodi]

I've been playing around with this and Docker has been an amazing tool for quickly iterating and trying things out. Since each layer of the docker image is cached I can make changes and try them quickly without having to reinstall packages or checkout the repo over and over again. @RMerlin wasn't kidding when he said 5GB, my docker image is over 5GB now which started as a minimum 20MB Ubuntu base.

In the end I wasn't able to successfully compile the AC68U firmware and what really confused me is that I thought I wouldn't need to checkout the legacy repo anymore because the AC68U is on the next gen firmware. Turns out you might still need the old repo for some tools but it wasn't very clear to me. I thought at first the doc was just out of date and I needed to update the symlinks to reference the ng repo.

There are also some symlinks in the instructions referencing am-toolchains but the instructions doesn't mention checking out that repo.

It would be nice if the next gen firmware could be built without any dependencies on the legacy codebase. Seems like that shouldn't be too difficult to do. If I'm misunderstanding something and it doesn't (maybe everything that's needed has already been pulled out to am-toolchains) then updating the instructions would be nice.

Anyway I'm not planning on spending more time on this but good luck to you guys.

 

[plun]

Hi all

Maybe assarbad's building script can be cloned and modified? (saves time)

https://github.com/assarbad/build-asuswrt-merlin

Used that script several times for building earlier versions with Ubuntu 14.04.

(The vagrant stuff can be excluded)

 

[RMerlin]

It would be nice if the next gen firmware could be built without any dependencies on the legacy codebase.

There are no dependencies between the old and the NG branch. You only need the NG branch and the am-toolchain repo.

 

[sfx2000]

Or make distribution easier: just create a script that will download and setup the toolchain and required packages on top of a pristine Ubuntu 18.04 LTS system. Setting up a Linux VM is really easy, anyone unable to do something as trivial should NOT be messing with firmware customization.

Such a script would be really trivial to do really:

snip for brevity

There you go. Ubuntu system turned into a functional build environment, without having to distribute a 5 GB VM.

I was working on a script - but you make a very good point - what happens when the script breaks because of some distro-level update that breaks a dependency in the AsusWRT buildroot - it happens, and has happened in the past.

Looking back - might just be best to clean up the wiki pages - there's a bit of info that is out of date, and that's what is likely tripping up folks that might be interested.

 

[john9527]

Looking back - might just be best to clean up the wiki pages - there's a bit of info that is out of date, and that's what is likely tripping up folks that might be interested.

Hanging my head....the other problem is that some of the necessary info is now on the git repro pages instead of the wiki (for the ng toolchain setup). I ran into it when I set up to compile ng but never got back to do the cleanup.

 

[kvic]

Looking back - might just be best to clean up the wiki pages - there's a bit of info that is out of date, and that's what is likely tripping up folks that might be interested.

lol. somehow I anticipated that this will happen. I bet you haven't compiled asuswrt for years. Let the active users do their work and help each other.

 

[sfx2000]

lol. somehow I anticipated that this will happen. I bet you haven't compiled asuswrt for years. Let the active users do their work and help each other.

Hehe... I did actually get things to build - I just look at the mess of symlinks and older dependencies that make the environment very fragile....

I can appreciate patience of Eric and John - working with the mess that it is...

No reason why there should be things 5 layers down and splash symlinks around a file system... nothing against asuswrt, but this is the result of decisions from long time back and the legacy they inherited - a good example of how to do things right, look at OpenWRT and what they've done there as a public example...

 

[Starbuck99]

Well guys, if the guru's are having trouble, just think about us folks who aren't linux guru's.
I now have 2 questions, while reading the latest posts...

1... Can the code be cleaned up a little?
2... Can someone then post a script for everybody to use?

I still think it is of benefit to everybody if more than one or two persons are actually able to compile..

 

[mokodi]

Thanks to @RMerlin and @john9527 for the helpful tips. I successfully compiled the RT-AC68U image from trunk in my Docker image. It was easier than I thought and it's pretty straightforward. Hardly a mess.

Here's the Dockerfile I used. I only set it up and tested it for non HND but it should give a good idea of what needs to be done. Obviously anyone wanting to use this will either have to run similar commands in Ubuntu or modify the Dockerfile to use multi stage builds.

Since the docker image runs as a root user you'll have to create a user first and make the necessary changes for HND.

Looking forward to what the community contributes now that it can be compiled

Creating ASUS RT-AC68U firmware to image/RT-AC68U_384.7_alpha1.trx

TRX Image:
 Total Size .... : 37310464 (36436.0 KB) (35.6 MB)
 CRC-32 ........ : 2DA6EDD8

#md5sum image/RT-AC68U_384.7_alpha1.trx > image/RT-AC68U_384.7_alpha1.md5

make[3]: Leaving directory '/asuswrt-merlin.ng/release/src-rt-6.x.4708'
make[2]: Leaving directory '/asuswrt-merlin.ng/release/src-rt-6.x.4708'
make[1]: Leaving directory '/asuswrt-merlin.ng/release/src-rt-6.x.4708'

FROM ubuntu:18.10

RUN apt-get update && apt-get -y upgrade

ENV DEBIAN_FRONTEND noninteractive
ENV DEBCONF_NONINTERACTIVE_SEEN true

RUN echo "tzdata tzdata/Areas select Etc" > /tmp/preseed.txt; \
    echo "tzdata tzdata/Zones/Europe select GMT" >> /tmp/preseed.txt; \
    debconf-set-selections /tmp/preseed.txt && \
    apt-get update && \
    apt-get install -y tzdata

RUN apt-get install -y \
  libtool-bin \
  cmake \
  libproxy-dev \
  uuid-dev \
  liblzo2-dev \
  autoconf \
  automake \
  bash \
  bison \
  bzip2 \
  diffutils \
  file \
  flex \
  m4 \
  g++ \
  gawk \
  groff-base \
  libncurses-dev \
  libtool \
  libslang2 \
  make \
  patch \
  perl \
  pkg-config \
  shtool \
  subversion \
  tar \
  texinfo \
  zlib1g \
  zlib1g-dev \
  git-core \
  gettext \
  libexpat1-dev \
  libssl-dev \
  cvs \
  gperf \
  unzip \
  python \
  libxml-parser-perl \
  gcc-multilib \
  gconf-editor \
  libxml2-dev \
  g++-multilib \
  gitk \
  libncurses5 \
  mtd-utils \
  libncurses5-dev \
  libvorbis-dev \
  git \
  autopoint \
  autogen \
  sed \
  build-essential \
  intltool \
  libelf1 \
  libglib2.0-dev \
  xutils-dev \
  lib32z1-dev \
  lib32stdc++6 \
  xsltproc \
  gtk-doc-tools

RUN git clone https://github.com/RMerl/asuswrt-merlin.ng

RUN git clone https://github.com/RMerl/am-toolchains

RUN ln -s /am-toolchains/brcm-arm-sdk/hndtools-arm-linux-2.6.36-uclibc-4.5.3 /opt/brcm-arm
RUN rm /asuswrt-merlin.ng/release/src-rt-6.x.4708/toolchains
RUN ln -s /am-toolchains/brcm-arm-sdk  /asuswrt-merlin.ng/release/src-rt-6.x.4708/toolchains

ENV PATH=$PATH:/opt/brcm-arm/bin

RUN dpkg --add-architecture i386
RUN apt-get update
RUN apt-get install -y libelf1:i386

WORKDIR /asuswrt-merlin.ng/release/src-rt-6.x.4708
RUN make clean
RUN make rt-ac68u

 

[RMerlin]

Can the code be cleaned up a little?

No, because that would make it a pain to keep code in sync with Asus.