ARM toolchains with NPTL for Asuswrt/Merlin

[kvic]

Earlier we found (through little exercises in the Pixelserv thread) that the ARM toolchain supplied in Asuswrt GPL does not support native pthread. We also discovered that vendors do not build their own toolchains from Build Root (anecdotal evidence: Asus and Netgear have a few binaries identical from their toolchains).

The past weekend I spent sometime building an ARM toolchains from the Build Root source shipped in Asus' 9135 GPL codes for AC56U. To the credit of Asus (and by perhaps Broadcom), they actually include the codes... the configs & codes unfortunately aren't build-able out of the box. With some effort and a few rounds of back and forth between building the toolchain and using the new toolchain building the firmware (v378.55), now I have a workable toolchain that supposed to yield NPTL support.

I tried the resultant image i.e the new firmware binary on my AC56U. Didn't boot to completion. Had to rescue myself with Asus recovery software. To eliminate any residual traces due to previous toolchain. I started again from scratch and builded a new image which I haven't tried.

The challenge for me is I only have one Asus router and it's a "production" set. So the frequency is very limited that I can try and work out any issues in the new toolchain.

My immediate goal is to have the toolchains to support native pthread in Asuswrt/Merlin. Please drop a line here if any dev is interested in collaborating on this.

 

[sfx2000]

Been a while since I dug into any of the toolchains for ASUSwrt, they use glibc or uclibc?

There was an issue a long time back with ARMv7 (Cortex-A9) where threads could get deadlocked, e.g. Threads 1 and 2 running, and pop in a new thread and one of the previous threads would stall...

If I recall correctly, the fix was in libc...

There were some other oddities with pthreads on ARMv7 - might be the reason why some of the OEM's have specifically not built in NPTL..

 

[RMerlin]

Doing low-level changes to the toolchain is a bad idea, as there are a lot of precompiled libraries and binaries in the firmware, which might not be compatible with your modified toolchain.

Been a while since I dug into any of the toolchains for ASUSwrt, they use glibc or uclibc?

uclibc.

 

[sfx2000]

Doing low-level changes to the toolchain is a bad idea, as there are a lot of precompiled libraries and binaries in the firmware, which might not be compatible with your modified toolchain.

I agree...

OpenWRT might be a better option - and entware, if I recall, has a libpthread package that is installed as part of the entware package manager.

 

[kvic]

Doing low-level changes to the toolchain is a bad idea, as there are a lot of precompiled libraries and binaries in the firmware, which might not be compatible with your modified toolchain.

It's a valid concern..

OpenWRT might be a better option - and entware, if I recall, has a libpthread package that is installed as part of the entware package manager.

OpenWRT/Entware use glibc. The code size is generally not as compact. My bottom line is uClibc /w NTPL could exist as a third run-time on Asuswrt/Merlin. If could rebuilt the whole firmware, will be a bonus.

 

[kvic]

Hooray!

After much pre-flight check, I decided to take another flash. Very delighted to report that the rebuilt firmware runs very well. My AC56U now is running 378.55 with latest stable uClibc (0.9.33.2) with support of native pthread. The pace is a bit beyond my expectation!

All apps (except one) run perfectly. Accelerated NAT ok. DPI engine ok. Adaptive QoS ok. Traffic analyser ok...Anything in kernel space run as usual. That's within my expectation.

Here is the screenshot of top.

Usually I don't have that many processes (as I kill off most junks on startup). The rows in green are now native pthreads. E.g., the stock minidlna is now only one process of two threads. The row with argument is the main thread, the one in green is the second thread. In the old firmware, they will be two heavy-weight processes + one additional heavy-weight process to manage the prior two. Three processes in total.

data_colld from Trend Micro also shows up as threads. Apparently Trend Micro code it with pthread (who won't in yester-years?). Hence, it benefits from the support of native pthread without re-compilation. For the close source binaries I only spot one, wred in the rogue. It seg fault.

wred like data_colld is written with pthread APIs. Not sure what magic Trend Micro put in, it crashes on load. Here is the backtrace:

phaeo:/jffs$ gdb `which wred`

(gdb) run
Starting program: /usr/sbin/wred
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
[New LWP 2670]

Program received signal SIGSEGV, Segmentation fault.
0x401412b8 in free () from /lib/libc.so.0
(gdb) bt
#0  0x401412b8 in free () from /lib/libc.so.0
#1  0x400971dc in pthread_attr_destroy () from /lib/libpthread.so.0
#2  0x0000ae30 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb)

Personally not affecting me as I don't run any DPI stuff. I expect to be a quick fix for Trend Micro. No more excuse for Asus not to do a bit more decent ground work!

 

[kvic]

I have uploaded this toolchain to Github. Interested folks can find it here:

• https://github.com/kvic-z/brcm-arm-toolchains

Wish some of you may enjoy as much as I do.

 

[ASAT]

I have uploaded this toolchain to Github.

I'm currently running Asuswrt-Merlin 378.55 built with this updated Buildroot. And I'm in the process of re-compiling all my programs. No issues thus far. I don't use the Trend Micro.

However, one thing that always bites me, when compiling the NTPD program, is that I forget to copy timepps.h to the /usr/include folder of the toolchain. Without timepps.h, everything compiles fine and it appears to work. However, it will show poor precision timekeeping, as in milliseconds versus microseconds.

To fix this: Copy timepps.h to /usr/include of the ARM toolchain. Otherwise NTPD is built without PPS support for high-precision timekeeping.

Copy from here (timepps.h):
https://github.com/ago/pps-tools

To here:
https://github.com/kvic-z/brcm-arm-...-brcm-linux-uclibcgnueabi/sysroot/usr/include

See instruction at bottom of page about timepps.h:
http://www.linuxpps.org/wiki/index.php/LinuxPPS_installation

 

[kvic]

Nice and thanks for the patch.

Added timepps.h. Also created a ChangeLog.

I'm currently running Asuswrt-Merlin 378.55 built with this updated Buildroot. And I'm in the process of re-compiling all my programs. No issues thus far. I don't use the Trend Micro.

However, one thing that always bites me, when compiling the NTPD program, is that I forget to copy timepps.h to the /usr/include folder of the toolchain. Without timepps.h, everything compiles fine and it appears to work. However, it will show poor precision timekeeping, as in milliseconds versus microseconds.

To fix this: Copy timepps.h to /usr/include of the ARM toolchain. Otherwise NTPD is built without PPS support for high-precision timekeeping.

Copy from here (timepps.h):
https://github.com/ago/pps-tools

To here:
https://github.com/kvic-z/brcm-arm-...-brcm-linux-uclibcgnueabi/sysroot/usr/include

See instruction at bottom of page about timepps.h:
http://www.linuxpps.org/wiki/index.php/LinuxPPS_installation

 

[shibby20]

Really nice work. I will try compile Tomato for RT-AC3200 using your toolchain.
Will let you know.

Best Regards.

 

[shibby20]

Debian Jessie x64

/opt/hndtools-arm-linux-2.6.36-uclibc-4.5.3/bin/../libexec/gcc/arm-brcm-linux-uclibcgnueabi/4.5.3/cc1: error while loading shared libraries: libmpc.so.2: wrong ELF class: ELFCLASS32

any advise?

 

[kvic]

Debian Jessie x64
any advise?

Seems cc1 cannot load the correct version of libmpc. Could you try below..see if it helps?

mkdir -p /projects/hnd/tools
ln -s /opt /projects/hnd/tools/linux

 

[john9527]

Sending this via my 374 fork compiled with your toolchain... I don't have to worry about any of the TrendMicro components.

I did have to make one edit to get the compile to complete....

error occurred in

/root/brcm-arm-toolchains/hndtools-arm-linux-2.6.36-uclibc-4.5.3/bin/../arm-brcm-linux-uclibcgnueabi/sysroot/usr/include/ctype.h

ctype.h was linked from rc.h from auth.c
error at ctype.h:73.12 complaining of expected '('
I couldn't see what it was complaining about.

I then looked at my original toolchain and that line was commented out, so I did the same to get the successful compile. So far I don't see any big change in the way the router is running, maybe web pages load a bit snappier (or maybe the placebo effect )

 

[ASAT]

My Asuswrt-Merlin build environment is Linux Mint 13 "Maya" 64-bit on VMware.

 

[lancethepants]

My one concern is that uClibc doesn't guarantee compatibility between different versions. If the firmware uses an older version, you can't do anything with binary blobs You may not immediately encounter some issues, or you may be lucky.

Sent from my XT1049 using Tapatalk

 

[john9527]

Duh...sorry.....mine is Ubuntu 12.04 x64 on VMware Player running on Windows 7 Pro x64
Followed the instructions from Merlin's wiki.

 

[kvic]

I did have to make one edit to get the compile to complete....

error occurred in

/root/brcm-arm-toolchains/hndtools-arm-linux-2.6.36-uclibc-4.5.3/bin/../arm-brcm-linux-uclibcgnueabi/sysroot/usr/include/ctype.h

ctype.h was linked from rc.h from auth.c
error at ctype.h:73.12 complaining of expected '('
I couldn't see what it was complaining about.

Looking at those few lines, we have

     69 /* ISO C99 introduced one new function.  */
     70 #ifdef  __USE_ISOC99
     71 __BEGIN_NAMESPACE_C99
     72
     73 extern int isblank(int __c) __THROW;
     74
     75 __END_NAMESPACE_C99
     76 #endif

Seems related to C99 enabled or not but I have no idea what may have caused the error in your environment. Will keep an eye on this.

My build environment is a headless Debian Wheezy 64bit on VirtualBox. I didn't have this error. Btw, I believe Wheezy provides better compatibility in general for building asuswrt or merlin.

So far I don't see any big change in the way the router is running, maybe web pages load a bit snappier (or maybe the placebo effect )

I had the exact experience and told myself the same thing! lol

 

[kvic]

My one concern is that uClibc doesn't guarantee compatibility between different versions. If the firmware uses an older version, you can't do anything with binary blobs You may not immediately encounter some issues, or you may be lucky.

Sent from my XT1049 using Tapatalk

It's a valid concern as Merlin raised the same thing at the beginning. I don't have an idea where are those blobs in asuswrt or merlin. Nor how they were linked in. Or else we could have a closer look...

So far everything runs smoothly except that wred.

 

[shibby20]

Debian Jessie x64

/opt/hndtools-arm-linux-2.6.36-uclibc-4.5.3/bin/../libexec/gcc/arm-brcm-linux-uclibcgnueabi/4.5.3/cc1: error while loading shared libraries: libmpc.so.2: wrong ELF class: ELFCLASS32

any advise?

problem solved. Debian Jessie doesn`t have libmpc2 package but libmpc3. I had to make symlink from /usr/lib/x86_64-linux-gnu/libmpc.so.3.0.0 to /usr/lib/x86_64-linux-gnu/libmpc.so.2 and compilation went but failed on

/opt/hndtools-arm-linux-2.6.36-uclibc-4.5.3/bin/../arm-brcm-linux-uclibcgnueabi/sysroot/usr/include/ctype.h:73:12: error: expected identifier or ‘(’ before ‘int’
/opt/hndtools-arm-linux-2.6.36-uclibc-4.5.3/bin/../arm-brcm-linux-uclibcgnueabi/sysroot/usr/include/ctype.h:73:12: error: expected ‘)’ before ‘==’ token

ctype.h line 73:
extern int isblank(int __c) __THROW;

In asus`s toolchain this line is commented, well i commented it:
//extern int isblank(int __c) __THROW;

next i had to add some symlinks in toolchain/bin directory:

lrwxrwxrwx 1 shibby shibby 31 lis 7 15:39 arm-linux-ar -> arm-brcm-linux-uclibcgnueabi-ar
lrwxrwxrwx 1 shibby shibby 32 lis 7 15:39 arm-linux-c++ -> arm-brcm-linux-uclibcgnueabi-c++
lrwxrwxrwx 1 shibby shibby 31 lis 7 15:39 arm-linux-cc -> arm-brcm-linux-uclibcgnueabi-cc
lrwxrwxrwx 1 shibby shibby 32 lis 7 15:39 arm-linux-cpp -> arm-brcm-linux-uclibcgnueabi-cpp
lrwxrwxrwx 1 shibby shibby 32 lis 7 15:39 arm-linux-g++ -> arm-brcm-linux-uclibcgnueabi-g++
lrwxrwxrwx 1 shibby shibby 32 lis 7 15:40 arm-linux-gcc -> arm-brcm-linux-uclibcgnueabi-gcc
lrwxrwxrwx 1 shibby shibby 31 lis 7 15:40 arm-linux-ld -> arm-brcm-linux-uclibcgnueabi-ld
lrwxrwxrwx 1 shibby shibby 35 lis 7 15:41 arm-linux-ranlib -> arm-brcm-linux-uclibcgnueabi-ranlib
lrwxrwxrwx 1 shibby shibby 36 lis 7 15:39 arm-linux-readelf -> arm-brcm-linux-uclibcgnueabi-readelf
lrwxrwxrwx 1 shibby shibby 33 lis 7 15:39 arm-linux-size -> arm-brcm-linux-uclibcgnueabi-size

next compilation stopped on miniupnpd
natpmp.c: In function ‘ReceiveNATPMPOrPCPPacket’:
natpmp.c:211:36: error: dereferencing pointer to incomplete type
natpmp.c:212:15: error: dereferencing pointer to incomplete type
natpmp.c:214:35: error: dereferencing pointer to incomplete type
natpmp.c:215:39: error: dereferencing pointer to incomplete type

well i taked file arm-brcm-linux-uclibcgnueabi/sysroot/usr/include/netinet/in.h from asus`s toolchain

after those changes i finally compiled Tomato. Flashed to RT-AC3200 and looks all works.

11 root 0 SW [kworker/u:1]
53 root 0 SW [sync_supers]
55 root 0 SW [bdi-default]
56 root 0 SW< [kblockd]
119 root 0 SW [kswapd0]
166 root 0 SW [fsnotify_mark]
174 root 0 SW< [crypto]
246 root 0 SW [mtdblock0]
251 root 0 SW [mtdblock1]
256 root 0 SW [mtdblock2]
261 root 0 SW [mtdblock3]
288 root 0 SW [kworker/1:1]
289 root 0 SW [kworker/0:1]
292 root 0 SW [mtdblock4]
689 root 656 S hotplug2 --persistent --no-coldplug
734 root 0 SW [dhd_watchdog_th]
738 root 0 SW [dhd_watchdog_th]
742 root 0 SW [dhd_watchdog_th]
770 root 1564 S buttons
772 root 1568 S blink_5g
773 root 1500 S console
775 root 1512 S /bin/sh
776 root 1504 S syslogd -L -s 50 -b 1
778 root 1504 S klogd
794 root 0 SW [khubd]
868 root 0 SW< [usbhid_resumer]
1023 root 1504 S telnetd -p 23
1046 root 1044 S dropbear -p 22 -a
1071 root 1032 S eapd
1086 root 1672 S nas
1119 root 1512 S crond -l 9
1123 root 0 SW [scsi_eh_2]
1124 root 0 SW [usb-storage]
1127 root 0 SW [scsi_eh_3]
1128 root 0 SW [usb-storage]
1149 root 1056 S rstats
1150 root 1000 S cstats
1507 root 1112 S dropbear -p 22 -a
1584 root 1516 S -sh
2267 nobody 1344 S dnsmasq -c 1500 --log-async
2623 root 2812 S httpd
2624 root 1516 S udhcpc -i eth4 -b -s dhcpc-event -H unknown -m

but i need more time to test all features. Will let you know.
Best Regards.

 

[kvic]

Great to hear you're making progress! What an adventure..

problem solved. Debian Jessie doesn`t have libmpc2 package but libmpc3. I had to make symlink from /usr/lib/x86_64-linux-gnu/libmpc.so.3.0.0 to /usr/lib/x86_64-linux-gnu/libmpc.so.2 and compilation went but failed on

It's one way of solving the problem. There is an easier way I believe i.e. the one I suggested in #12.

Build root is self-contained for critical components for the host. libmpc is one of them. I'm not 100% certain because I haven't met the error as I created the link the first time I built the asus firmware.

I think it's one of the ugliness left behind by Broadcom in the whole thing. I did't attempt to justify it because every tarball of ASUSWRT and variants depend on it..

Anyhow, glad to hear you solved the problem otherwise. I'm very interested in your progress. Keep us updated.

Cheers