BRT-AC828 Questions wrt firmware compilation and fixing OpenVPN issue

[diederik]

I have a BRT-AC828 with Firmware version 3.0.0.4.382_70348 and like others I've run into the problem that recent versions of OpenVPN client (compiled with OpenSSL 3.x) fail to connect to the OpenVPN server in my router.

admin@BRT-AC828:/tmp/home/root# openssl x509 -noout -text -in /etc/openvpn/server1/ca.crt | grep Signature
    Signature Algorithm: sha1WithRSAEncryption
    Signature Algorithm: sha1WithRSAEncryption

According to https://github.com/RMerl/asuswrt-merlin.ng/commit/cbae028cd8ed3a5853eb4ddf275fb3e0e3a75bc2 the fix was really easy, just replace sha1 with sha256 in pkitool.

So this sounded like a good reason to actually try to compile firmware from source code. I followed the instructions from README.TXT and created a Debian Jessie VM (matches Ubuntu 16.04 pretty decently), installed the build dependencies, cd-ed into release/src-qca-ipq806x and ran 'make brt-ac828'. First try failed. Second try failed as the instructions missed the tcl build dependency. Third try was successful but if I would upload that to my router, I wouldn't know if I was running Asus' or my own firmware.
So for the 4th try, I modified "src/router/extendno.conf" to EXTENDNO=diederik-sha256-1 and applied the aforementioned patch to pkitool. That built succeeded too, but upon trying to upload it to my router, I got this error message

Invalid Firmware Upload
Firmware upgrade unsuccessful. This might result from incorrect image or error transmission, please check the model name BRT-AC828 and version of firmware from support site and try again.

So I went for a 5th try where I only applied Merlin's patch. I should still be able to determine whether I was running my own compiled firmware as it should then return "sha256WithRSAEncryption". Built of the firmware was successful again and now also the upload to my router succeeded \o/

So my first question is then: How or where can change the version number so it's easy to determine what version of "my" code I'm running, but still gets accepted when uploading it to my router?
My assumption here is that maybe my new version number was 'wrong' and therefor rejected. I think a numbering scheme with <CCYYMMDD>-<seq-nr> could be useful, but I'm open to (other) suggestions.

I'm quite certain that I was running my own compiled firmware:

admin@BRT-AC828:/tmp/home/root# grep sha /rom/easy-rsa/pkitool
        $OPENSSL req $BATCH -days $CA_EXPIRE $NODES_REQ -new -newkey rsa:$KEY_SIZE -sha256 \
                -in "$FN.csr" $CA_EXT -md sha256 -config "$KEY_CONFIG" ) && \

By making a change in OpenVPN's config and back again, so nothing changed, I saw that new files were generated in "/etc/openvpn/server1/". So I exported a new ovpn file, loaded that onto my phone and tried to connect to my router with OpenVPN client ... which failed. Grepping ca.crt showed that the Signature algorithm was still sha1.

Did I do something wrong? Should something more or sth else be changed so that sha256 is being used to generate ca.crt?

I have a lot to learn and plan/hope to experiment a lot. But I'm assuming that the (flash?) storage which holds the firmware only has a limited nr of write cycles. How 'worried' should I be wrt that?
Is it possible to write/deploy things to an USB stick for example, so I don't have to constantly upload new firmware?

Any other tips wrt development are also much appreciated

 

[RMerlin]

release/src-rt/version.conf

 

[diederik]

Thanks. I was already looking at your repo's history/commits for possible clues wrt the OpenVPN issue and then saw the 6th commit about your versioning scheme, which I'll very likely adopt (too)

 

[RMerlin]

But I'm assuming that the (flash?) storage which holds the firmware only has a limited nr of write cycles. How 'worried' should I be wrt that?

I don't know about that particular model, but all high-end Broadcom models for the past 7-8 years are using high endurance flashes which are rated for either 10K or 100K rewrites. Far more than you will ever use. If in doubt you could check what NAND chip they use, and look it up online, but I suspect it's very similar to what they use in the other high-end BCM models.

You can imagine how many flashes I have done myself over the years on my various routers. Probably like 40-50 firmware flashes on some of these. Basically there is nothing to be worried about.

 

[diederik]

Thanks, that's quite reassuring
How can I find out what NAND chip my router is using? There are various photo's with the FCC submission. Would it mean inspecting those photo's to identify the chip(s) and then look that up?

I plan to update the various programs to newer versions and I don't expect much troubles with that.
But I also plan/hope to update the kernel and I expect that task to be HUGE. (And I'll likely fail, but you never know until you try)
I'll likely do/try that in small increments, which will result in many firmware flashes, hence my concern.

But before I even get to any updates, I have to learn a LOT, much more then I anticipated. I usually do that by changing 'something' and see what/how/if things change. I have a whole bunch of 16G USB3 'sticks' which I'd like to use to redirect various/most writes to. Is there a way to accomplish that?
Is there a way to virtualize (much of) my router, so I can do most of my learning on that?

There is an Asus tool which you can use to transfer an image to the router if f.e. it won't start. But that is a Windows only tool and I only have Linux. Is there a comparable tool (tftp?) I can use instead?

Thanks for all the work you do When I found out my current router wouldn't be supported by your firmware (I totally understand why), I was quite bummed as I VERY much enjoyed your firmware on my previous router :-D

 

[RMerlin]

How can I find out what NAND chip my router is using? There are various photo's with the FCC submission. Would it mean inspecting those photo's to identify the chip(s) and then look that up?

Correct, you'd need to be able to read the part number on the chip. Or maybe Wikidevi has the info.

But I also plan/hope to update the kernel and I expect that task to be HUGE. (And I'll likely fail, but you never know until you try)

If Qualcomm is anything like Broadcom, the kernel is most likely heavily patched. Plus, you have a bunch of pre-compiled kernel components that would fail to link, and modules that would fail to load if you changed the kernel. Don't waste your time there.

Is there a way to virtualize (much of) my router, so I can do most of my learning on that?

No. You have to build the firmware, and flash it.

Is there a comparable tool (tftp?) I can use instead?

Don't know how the Qualcomm platform works, but if it's like Broadcom and its CFE, that restoration tool is just a tftp client.

 

[diederik]

Correct, you'd need to be able to read the part number on the chip. Or maybe Wikidevi has the info.

I already had the wikidevi page bookmarked: https://wikidevi.wi-cat.ru/ASUS_BRT-AC828/M2 and it looks like that's where I got the FCC link from. And I'm guessing the FLA1 and FLA2 are the relevant ones.

If Qualcomm is anything like Broadcom, the kernel is most likely heavily patched.

What I understand so far is that Qualcomm is (much?) better then Broadcom in that there's much more upstreaming taking place, so it seems I may be lucky in that regard. Searching for 'IPQ806x' in Linus' tree returns quite some results and many of them quite recently.
Having said that, the diff seems to be HUGE. I created the following script to compare upstream 3.4.103 with Asus' kernel for the BRT-AC828 from `asuswrt/release/src-qca-ipq806x/linux/linux-3.4.x/` and that resulted in quite some patch files.

#!/bin/sh

DIR_ORG="linux-3.4.103"
DIR_ASUS="linux-3.4.x-BRT-AC808"
DIR_DIFF="linux-3.4.x-diff"

for file in $(find $DIR_ASUS/ -type f -print)
do
    #echo "$file"
    file_path=$(echo "$file" | sed "s/$DIR_ASUS//")
    #echo "$file_path"
    if ! cmp -s ${DIR_ORG}${file_path} ${DIR_ASUS}${file_path} ; then
        #echo "$file_path is different"
        diff -N --unified ${DIR_ORG}${file_path} ${DIR_ASUS}${file_path} > ${DIR_DIFF}${file_path}
    fi
done

But this is also meant as a learning experience and I intend to use `quilt` to manage the patches. I 'played' with it a little bit wrt the Debian kernel, but that is nothing compared to this 'patch set'.
I hope that when going to higher kernel versions, I am able to keep dropping patches. If not, I'll probably get bored/fed up at some point and bail out.

Plus, you have a bunch of pre-compiled kernel components that would fail to link, and modules that would fail to load if you changed the kernel. Don't waste your time there.

This is indeed my biggest worry. But getting my router to run on a Super Long Term Support kernel is actually my main goal.
How can I find the pre-compiled kernel components?

I just did the following:

diederik@bagend:~/dev/routers/linux-3.4.x-BRT-AC808$ find . -type f | perl -lne 'print if -B'
./Documentation/logo.gif
./drivers/staging/ft1000/ft1000-usb/ft3000.img
./drivers/staging/ft1000/ft1000-pcmcia/ft1000.img
./drivers/char/.objs/rtl8370mb.obj

The first 3 are also in the upstream kernel, so that leaves only one :-O
But it's also the most crucial part afaict ...
If I do `find . -type f -name *.obj | grep -vE "samba-*|\/lzo\/" | wc -l` in the whole extracted GPL archive, I get 59 results. Of those, there's 57 for u-boot (`src-qca-ipq806x/uboot/drivers/rtl8370mb`), the above mentioned char driver and `./asuswrt/release/src/router/rc/prebuild/ctl.obj`; don't know yet what that last one if for.
All (very) crude metrics, but I do hold out some hope.
I plan to experiment with kernels at the very last stage, so if it doesn't work out, I should still be in a better position then I am now.

The first thing I want to upgrade, is the compiler (?) contained in 'openwrt-gcc463.arm.tar.bz2'.
Do you have experience/tips wrt that?

No. You have to build the firmware, and flash it.

Ok, thanks

Don't know how the Qualcomm platform works, but if it's like Broadcom and its CFE, that restoration tool is just a tftp client.

Excellent, I had a feeling that would be the case and I think that https://openwrt.org/toh/netgear/r7800#debricking looks like a way to do it
What is 'CFE'?

 

[RMerlin]

This is indeed my biggest worry. But getting my router to run on a Super Long Term Support kernel is actually my main goal.
How can I find the pre-compiled kernel components?

Look for any .o files that are part of Asus's GPL drop, and which are specific to the SDK itself, and not part of Asus's userland code (i.e. not the rc/prebuild or httpd/prebuild content). I don't know how the QCA SDK is organized within Asuswrt, so I can't tell you where to look. You will have to find it yourself.

Anything that is used to build a .ko will be problematic. For example, anything that touches the skbuf struct in the kernel cannot be changed in any way when dealing with network-related prebuild objects. And fully compiled kernel modules (like the Tuxera filesystem modules or the TrendMicro udp engine) will also expect a specific kernel magic number to be loadable. Kernel patches that changes this will not work.

All of these are why I'm saying you will not be able to upgrade the kernel. At best you might be able to backport some specific patches, provided they don't touch any of these sensitive areas.

The first thing I want to upgrade, is the compiler (?) contained in 'openwrt-gcc463.arm.tar.bz2'.
Do you have experience/tips wrt that?

Again, since some stuff is precompiled, your options are limited there, as the ABI needs to remain identical, same for any static library that might be linked with these prebuilt components.

What is 'CFE'?

Common Firmware Environment. It's an alternative to uboot or UEFI that is used by Broadcom.

 

[diederik]

Thanks so much for your insight
I have to figure out what sources exactly are relevant for my router, but I've already identified 2 directories by which I'm quite likely affected: "router/libbcmcrypto/prebuilt.qca.ipq806x/" and "router/shared/prebuild/", whereby especially the former seems rather crucial as afaik there is a HW crypto/hashing chip which is important for (decent) performance :-/

Bummer that even openwrt-gcc463 will be difficult/impossible. I found a openwrt-sdk-21.02.0-ipq806x-generic_gcc-8.4.0_musl_eabi.Linux-x86_64.tar.xz and that looked promising.
I do have 'access' to people knowledgeable about Linaro and keeping ABI stability, so hopefully I can still achieve something in that area.

Why can't anything be simple or even doable ;-P LOL

 

[RMerlin]

I have to figure out what sources exactly are relevant for my router, but I've already identified 2 directories by which I'm quite likely affected: "router/libbcmcrypto/prebuilt.qca.ipq806x/" and "router/shared/prebuild/", whereby especially the former seems rather crucial as afaik there is a HW crypto/hashing chip which is important for (decent) performance :-/

router/shared/prebuild/ is only used for libshared.so, an Asuswrt library, so that's kernel-agnostic.

Look for stuff either within release/src-rt-qXXXX/ (which contains that platform's SDK), or things like release/src/router/qca*/ (like the NAT acceleration component).

I found a openwrt-sdk-21.02.0-ipq806x-generic_gcc-8.4.0_musl_eabi.Linux-x86_64.tar.xz and that looked promising.

I don't think Asus uses musl for that platform, it's probably glib or uclibc.

My advice would be to focus on updating the userspace components that would make the most sense, like OpenVPN and dnsmasq. You can then look at possibly backporting some of my own changes, like dnsfilter. These are easier, and will give you a chance to gradually familiarize yourself with the Asuswrt environment.

If you feel the need for a challenge, then look into backporting the Wireless Log page. That will require you to familiarize yourself with QCA's API to retrieve connected STA info - that code would be in httpd/sysdep/web-*.c and httpd/data_arrays.c (my part of it).

 

[diederik]

router/shared/prebuild/ is only used for libshared.so, an Asuswrt library, so that's kernel-agnostic.

Cool, thanks.

I don't think Asus uses musl for that platform, it's probably glib or uclibc.

Asus uses uclibc, version 0.9.33.2 is used in openwrt-gcc463.arm

My advice would be to focus on updating the userspace components that would make the most sense, like OpenVPN and dnsmasq.

I will do that.

If you feel the need for a challenge.

I don't mind a challenge. But the challenge I am/was after is making my router (more) secure and ready for the next 5-10 years (OpenSSL 3 issue wrt OpenVPN is a prime example). And the way I want to achieve that is by using software that is not severely outdated and preferably supported (in some way).

According to wikidevi, the BRT-AC828 was released on 2016-05-31.
The IMO most important components used are

• GCC 4.6.3 released 2012-03-01
• kernel 3.4(.103) released 2012-05-20, EOL 2016-10 (~4 months after router was released)

While I can understand that using kernel 4.4, the first SLTS release (also used in Ubuntu 16.04, referenced in README.TXT), may have been too ambitious, 3.16 released on 2014-08-03 and used in Debian Jessie should've been entirely doable. I'm a fan of Debian, but it also has other benefits like security support by first the security team and then by the LTS and then by the ELTS teams. All you have to do is leverage the work that is already been done by the Debian project.
If someone wants to forward-port things to f.e. Debian Stretch levels there should be plenty of article/tutorials/etc on the internet as more people would've upgraded from Jessie to Stretch, thereby leveraging their work.

I realize that the situation for my BRT-AC828 won't change, but I really do want to encourage Asus to use SLTS kernels in the future. Routers should easily last 10+ years imo and that really fits the targets of the SLTS kernel effort.
I also realize that the situation is crap across the board, but that also presents Asus with an opportunity ... by being not crap ;-)

You know better then anyone (here) that things don't improve unless someone puts in the work.
In the case of the BRT-AC828 I decided it might as well be me (and it would be a useful learning opportunity).
As the first step I wanted to upgrade the toolchain. As I prefer to leverage the knowledge and work of others and uclibc looks like a wasteland (I found out today that uclibc-ng does seem maintained), using the same toolchain as OpenWRT uses for boards like mine, seems like a wise choice. I would also not have to reinvent the wheel and create yet another (tech) island.

As I then assumed I had everything in source code, that should not be a problem.
Under that same assumption I thought upgrading the kernel was a LOT of work, but doable (given enough patience/time).
As my primary goal is to improve security, I'm now seriously considering disabling the HW encryption/hashing modules. But I'll have plenty of time to mull about that as my plan is now to create a cleaned up source tree just for my router (I don't need/want 4 copies of samba f.e.) and then gradually upgrade userspace.
After that I can experiment all I want (in a separate branch) with upgrading the toolchain and kernel.

 

[RMerlin]

I realize that the situation for my BRT-AC828 won't change, but I really do want to encourage Asus to use SLTS kernels in the future.

Tell that to Qualcomm and Broadcom. Asus does not chose the kernel, the SoC manufacturer does when they generate the SDK for their product. Asus cannot change that.

This is the same situation with Android phones. Notice how a phone's kernel does not change even if the phone manufacturer upgrades to a new Android release - the kernel is set in stone by Qualcomm's SDK, not by Google or the phone manufacturer.

This is a common thing with the embedded market in general, where SoC manufacturers never upgrade their toolchain or kernels over the lifetime of a product. They simply see no financial incentive to do so, and would rather have people switch to a newer product.

 

[diederik]

Tell that to Qualcomm and Broadcom. Asus does not chose the kernel, the SoC manufacturer does when they generate the SDK for their product. Asus cannot change that.

I did bring this exact point up recently in a thread on debian-arm ML: Feedback from the community -> ARM
Asus may not decide it, but they should have some influence. Problem is that their incentives are (more) aligned with Qualcomm/Broadcom then with end users.

This is the same situation with Android phones. Notice how a phone's kernel does not change even if the phone manufacturer upgrades to a new Android release - the kernel is set in stone by Qualcomm's SDK, not by Google or the phone manufacturer.

This is a common thing with the embedded market in general, where SoC manufacturers never upgrade their toolchain or kernels over the lifetime of a product. They simply see no financial incentive to do so, and would rather have people switch to a newer product.

Yep. The whole "planned obsolescence" crap annoys me greatly.
And I've adjusted my buying criteria accordingly, so there's a good chance I've already bought my last Android phone.

 

[diederik]

But I'll have plenty of time to mull about that as my plan is now to create a cleaned up source tree just for my router (I don't need/want 4 copies of samba f.e.) and then gradually upgrade userspace.

I've now started to put things into git and I'm trying to figure out which folders (under `release/src/router`) I should 'import' and which I can just disregard.
I created a log file by doing `make --debug=b brt-ac828 | tee -a brt-ac828-build-debug-basic.log` (`--debug=b = basic debugging).
By doing `grep "Entering directory" brt-ac828-build-debug-basic.log | awk '{ print $4 }' | uniq` I get a list of directories which make entered into, which seems like good starting list of directories I need for my router.

Note: I'm a novice wrt Makefile, so there's a good chance I'm making mistakes.

What surprises me and don't understand is that I get 2 different folders for some components:
1) `release/src/router/iproute2` and `release/src/router/iproute2-3.x`
It looks like the former directory (which is truly ancient) is only used to make 'iproute2-clean', while the latter is actually used in the router (AFAICT). And if you want to make the 'clean' target, I'd expect it's more useful to clean the one that's actually used...

I always assumed that those multiple directories of the same component was because the (whole) source tree is used to build various different models with different configurations. And I could therefor get rid of the copies that were not used by the brt-ac828.
But here I have a case where it appears that I need both copies. Why is that?

In this particular case I could opt to include both copies as the directory is ~1.5MB in size. But ...
2) `release/src/router/samba-3.5.8` and `release/src/router/samba-3.6.x_opwrt`
`smbd --version` on my router gives `Version 3.6.25` which indicates the latter is actually used.
In the 'iproute2' case I didn't want it from a principle stand point, but in the case of 'samba' it's (also) a practical matter as `samba-3.5.8` is ~108MB in size. As I want to keep the sizes of my git repos manageable, I do think this is problematic.

What can I do about this? (So I don't have to put multiple copies of samba f.e. in my git repo(s))

When doing `grep -r "iproute2-clean"`, I only get results from the 'master' Makefile, so I'm 'guessing' that it calls the 'clean' target in some iproute2 folder, but how does it choose which iproute2 folder?
At first I thought it would take the first one alphabetically, but that theory doesn't hold when applying it to the samba (folder) case as there's also a 'samba' folder, but it doesn't use that one.
What's the logic that `make` uses here?

 

[RMerlin]

What surprises me and don't understand is that I get 2 different folders for some components:
1) `release/src/router/iproute2` and `release/src/router/iproute2-3.x`
It looks like the former directory (which is truly ancient) is only used to make 'iproute2-clean', while the latter is actually used in the router (AFAICT). And if you want to make the 'clean' target, I'd expect it's more useful to clean the one that's actually used...

Asus uses an unified Asuswrt codebase for all of their routers. So within the code there may be different component versions that will be selected based on which router is being compiled. You have to study the router/Makefile file to see which one gets compiled for your specific model.