RT-AX88U Internal security

[AdrianSz]

I wanted to raise the concern of potentially lacking internal router hardening in cases where router is hijicked/exploited by any means.
Such means could be attacker getting into router via software exploit/bug of any of the services that are enabled in the router. Routers are open for customizations and execution of partner software/plugins and majority of people trust too much on the 3pps; including trusting non stock firmware.

Note: I know too little about the this router internal FS structure hence I am not sure listed things below are really a big deal.

File permissions
First thing I wonder why file permissions are so wide open on various files? Example of /tmp.
In case of attacker gets into router, he can easily delete audit trail logs:
-rw-rw-rw- 1 admin root 1048600 Feb 24 13:59 syslog.log-1
-rw-rw-rw- 1 admin root 99541 Feb 24 14:09 syslog.log
-rw-rw-rw- 1 admin root 47 May 5 2018 resolv.conf

When I see executables that are world writeable I get always really nervous.
-rw-rw-rw- 1 admin root 237 May 5 2018 run_lldpd.sh

I guess something like su -l nobody rm -rf /; reboot would be a good test.

Kernel version
I feel personally bit bothered buying new router, having kernel 4.1 from mid 2015. Which is indeed LTS, but hey, way too old with respect to optimizations, security hardening and feature set. Why not picking 4.14 LTS as baseline and later switch to 4.19 LTS / 5.x LTS at end of the year.

Daemon execution
All daemons seem to execute under single admin user.
10441 admin 2696 R dropbear -p 5022 -a
1103 admin 3192 S avahi-daemon: running [RT-AX88U-3958.local]
1148 admin 9620 S erp_monitor
985 admin 3184 S /sbin/syslogd -m 0 -S -O /tmp/syslog.log -s 1024 -l 6

Kernel module bloat
Modules loaded, but not really used.
When I saw huawei_cdc_ncm loaded I had to quickly investigate what the heck this is, and that this is used in 3G/4G WAN access via mobile use cases. Which I am not using and most users wont be using.

Sandboxing & confinement
Connected to above as well. seccomp / selinux / cgroups / containers not used. Perhaps performance hit is too big ?
Perhaps the embedded SW mindset in the past considered this is not needed, but now with the SW extendability its less obvious running without any confinements.

Server socket bindings including WAN interface
admin@RT-AX88U:/# netstat -antu | grep 0.0.0.0:[[:digit:]]
tcp 0 0 0.0.0.0:5473 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:18017 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:3394 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:7788 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:56461 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:5022 0.0.0.0:* LISTEN
udp 0 0 0.0.0.0:67 0.0.0.0:*
udp 0 0 0.0.0.0:18018 0.0.0.0:*
udp 0 0 0.0.0.0:7788 0.0.0.0:*
udp 0 0 0.0.0.0:59000 0.0.0.0:*
udp 0 0 0.0.0.0:5353 0.0.0.0:*
udp 0 0 0.0.0.0:9999 0.0.0.0:*
udp 0 0 0.0.0.0:5474 0.0.0.0:*
udp 0 0 0.0.0.0:1900 0.0.0.0:*
udp 0 0 0.0.0.0:55178 0.0.0.0:*

ubuntu@ax88u-compile:~/$ nmap -p 1-60000 -A -T4 ***masked wan ip***
Nmap scan report for ***masked*** (***masked***)
Not shown: 59994 closed ports
PORT STATE SERVICE VERSION
3394/tcp open d2k-tapestry2?
5022/tcp open ssh Dropbear sshd 2016.74 (protocol 2.0)
5473/tcp open apsolab-tags?
7788/tcp open tcpwrapped
18017/tcp open http Asus wanduck WAN monitor httpd
|_http-server-header: wanduck
56461/tcp open upnp MiniUPnP 1.9 (AsusWRT 9.0.0.4; UPnP 1.1)
Service Info: OS: Linux; Device: WAP; CPE: cpe:/o:linux:linux_kernel, cpe:/o:asus:asuswrt:9.0.0.4

Depending how the thread goes perhaps slogan can be renamed.

 

[RMerlin]

Kernel version

Because 4.14 did not exist when that platform was developed by Broadcom, and the vast majority of manufacturers do not upgrade the kernel, as it requires a LOT of work to re-test everything, update driver code that might not work on newer kernel, and so on. This isn't unique to Asus (or router manufacturers even) but to almost every embedded devices. Look at your smartphone kernel for instance, it does not get updated even when jumping to a new major Android revision. The kernel version is dictated by the SDK for the SoC used on the device, not by the software stack built by the device manufacturer.

4.1 is actually quite recent for an embedded device, as the vast majority of ARM-based devices out there still run on 3.x, including many devices released in 2018.

Sandboxing & confinement

Such things require much more complex software design and higher end hardware to run it. You won't find that on a router that targets home users, this falls in the realm of business-class products, where paying a 30% premium to account for the increased development costs is the norm.

It always comes down to the intended target market. Everything comes at a cost - software development is not free. Just like your bank has a much better door lock than your house, these home routers won't match the security of high-end business class products. For home users, this is "good enough". The real issues are not those you pointed out, but poorly written software from the device manufacturers, most frequently the web server itself which has to run as root to be able to manage the router configuration.

Asuswrt is designed to run on things as low-powered as 400 MHz routers with 128 MB of RAM and 32 MB of flash. You won't be able to fit what you are asking for in a 25$ router with such hardware specifications.

 

[ColinTaylor]

This is a question that comes up again and again, although not specifically for the RT-AX88U (search the forums for more detailed answers). The bottom line is the router is an appliance. It is not a multi-user Linux distribution and should not be viewed as such. If people start running non-standard software on their router then they have to take responsibility for the security implications of that.

 

[AdrianSz]

This is a question that comes up again and again, although not specifically for the RT-AX88U (search the forums for more detailed answers). The bottom line is the router is an appliance. It is not a multi-user Linux distribution and should not be viewed as such. If people start running non-standard software on their router then they have to take responsibility for the security implications of that.

When devices start to have 4 x 1.8 Ghz CPU cores, capability of running vpn, nfs, tor, download client capability, cloud file sharing and more (even only for home usage) it becomes quite of advanced networking unit. It is the feature set which motivates the price of premium router and users want to use them in secure way. Not necessary multi user system, but having daemons confinement and separated by different users will reduce the damage in case software exploits.
I think when the routers emerge with NAS capability and sits on WAN it is fair to expect certain level of security, especially when so much technology and knowledge is available for companies.
I would not expect that from a 25$ router, because I will not even buy such thing. Its a 400$+ router which increases expectations on security, stability and technology. At least I am expect things to be secure, as many users expect the WPA3 in near future. And yes, even for home network.

 

[AdrianSz]

Because 4.14 did not exist when that platform was developed by Broadcom, and the vast majority of manufacturers do not upgrade the kernel, as it requires a LOT of work to re-test everything, update driver code that might not work on newer kernel, and so on. This isn't unique to Asus (or router manufacturers even) but to almost every embedded devices. Look at your smartphone kernel for instance, it does not get updated even when jumping to a new major Android revision. The kernel version is dictated by the SDK for the SoC used on the device, not by the software stack built by the device manufacturer.

4.1 is actually quite recent for an embedded device, as the vast majority of ARM-based devices out there still run on 3.x, including many devices released in 2018.

Maybe they are. I was perhaps comparing to recent androids (samsung s9 and onward), I think they have 4.9 something if I am not wrong. My raspberries, minnowboard have also 4.14+ something. They are also pretty embedded devices just powerful enough to run more powerful stuff than 400 mhz board did 10 years ago.

It always comes down to the intended target market. Everything comes at a cost - software development is not free. Just like your bank has a much better door lock than your house, these home routers won't match the security of high-end business class products. For home users, this is "good enough".

Yes, we are not talking about banking system. So perhaps, selinux, NIST, CISCAT compliance is out of the question. But it is still a not cheap premium router and I can imagine smaller companies could buy as wifi infrastructure.

The real issues are not those you pointed out, but poorly written software from the device manufacturers, most frequently the web server itself which has to run as root to be able to manage the router configuration.

I think everything is always poorly written in early days of software. Over time it gets better. That is what confinement partially protects against. But coding is not the only thing that can be poor and I didn't mentioned it because I didn't look at it. Poorly written software, poor compilation flags, poor configuration and poor integration of appliances all contribute too poor security.

Asuswrt is designed to run on things as low-powered as 400 MHz routers with 128 MB of RAM and 32 MB of flash. You won't be able to fit what you are asking for in a 25$ router with such hardware specifications.
s.

So you are saying we can fit it in the $400 router with slightly better spec?

 

[RMerlin]

Maybe they are. I was perhaps comparing to recent androids (samsung s9 and onward), I think they have 4.9 something if I am not wrong.

Those devices were released in 2018 however. Look at them again two years from now, and they will still be on the same kernel version.

So you are saying we can fit it in the $400 router with slightly better spec?

Sure, provided you delete Asuswrt, and rewrite it from the ground up, and forget about using it for any of your other models with lower hardware specs, are willing to delay the next router model by 2-3 years while you rewrite the whole operating system, and then add an extra 150$ to the price tag...

That 400$ pricetag is the premium you pay for 802.11ax. You'd expect to pay around 600$ for a similar product within the business class market and with a more security-focused firmware.

 

[Frankflash]

I wanted to raise the concern of potentially lacking internal router hardening in cases where router is hijicked/exploited by any means.
Such means could be attacker getting into router via software exploit/bug of any of the services that are enabled in the router. Routers are open for customizations and execution of partner software/plugins and majority of people trust too much on the 3pps; including trusting non stock firmware.

Note: I know too little about the this router internal FS structure hence I am not sure listed things below are really a big deal.

File permissions
First thing I wonder why file permissions are so wide open on various files? Example of /tmp.
In case of attacker gets into router, he can easily delete audit trail logs:
-rw-rw-rw- 1 admin root 1048600 Feb 24 13:59 syslog.log-1
-rw-rw-rw- 1 admin root 99541 Feb 24 14:09 syslog.log
-rw-rw-rw- 1 admin root 47 May 5 2018 resolv.conf

When I see executables that are world writeable I get always really nervous.
-rw-rw-rw- 1 admin root 237 May 5 2018 run_lldpd.sh

I guess something like su -l nobody rm -rf /; reboot would be a good test.

Kernel version
I feel personally bit bothered buying new router, having kernel 4.1 from mid 2015. Which is indeed LTS, but hey, way too old with respect to optimizations, security hardening and feature set. Why not picking 4.14 LTS as baseline and later switch to 4.19 LTS / 5.x LTS at end of the year.

Daemon execution
All daemons seem to execute under single admin user.
10441 admin 2696 R dropbear -p 5022 -a
1103 admin 3192 S avahi-daemon: running [RT-AX88U-3958.local]
1148 admin 9620 S erp_monitor
985 admin 3184 S /sbin/syslogd -m 0 -S -O /tmp/syslog.log -s 1024 -l 6

Kernel module bloat
Modules loaded, but not really used.
When I saw huawei_cdc_ncm loaded I had to quickly investigate what the heck this is, and that this is used in 3G/4G WAN access via mobile use cases. Which I am not using and most users wont be using.

Sandboxing & confinement
Connected to above as well. seccomp / selinux / cgroups / containers not used. Perhaps performance hit is too big ?
Perhaps the embedded SW mindset in the past considered this is not needed, but now with the SW extendability its less obvious running without any confinements.

Server socket bindings including WAN interface
admin@RT-AX88U:/# netstat -antu | grep 0.0.0.0:[[:digit:]]
tcp 0 0 0.0.0.0:5473 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:18017 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:3394 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:7788 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:56461 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:5022 0.0.0.0:* LISTEN
udp 0 0 0.0.0.0:67 0.0.0.0:*
udp 0 0 0.0.0.0:18018 0.0.0.0:*
udp 0 0 0.0.0.0:7788 0.0.0.0:*
udp 0 0 0.0.0.0:59000 0.0.0.0:*
udp 0 0 0.0.0.0:5353 0.0.0.0:*
udp 0 0 0.0.0.0:9999 0.0.0.0:*
udp 0 0 0.0.0.0:5474 0.0.0.0:*
udp 0 0 0.0.0.0:1900 0.0.0.0:*
udp 0 0 0.0.0.0:55178 0.0.0.0:*

ubuntu@ax88u-compile:~/$ nmap -p 1-60000 -A -T4 ***masked wan ip***
Nmap scan report for ***masked*** (***masked***)
Not shown: 59994 closed ports
PORT STATE SERVICE VERSION
3394/tcp open d2k-tapestry2?
5022/tcp open ssh Dropbear sshd 2016.74 (protocol 2.0)
5473/tcp open apsolab-tags?
7788/tcp open tcpwrapped
18017/tcp open http Asus wanduck WAN monitor httpd
|_http-server-header: wanduck
56461/tcp open upnp MiniUPnP 1.9 (AsusWRT 9.0.0.4; UPnP 1.1)
Service Info: OS: Linux; Device: WAP; CPE: cpe:/o:linux:linux_kernel, cpe:/o:asus:asuswrt:9.0.0.4

Depending how the thread goes perhaps slogan can be renamed.

if you dont like it you could always go and complain or moan to broadcom about the kernel not getting updated

 

[AdrianSz]

Those devices were released in 2018 however. Look at them again two years from now, and they will still be on the same kernel version.



Sure, provided you delete Asuswrt, and rewrite it from the ground up, and forget about using it for any of your other models with lower hardware specs, are willing to delay the next router model by 2-3 years while you rewrite the whole operating system, and then add an extra 150$ to the price tag...

That 400$ pricetag is the premium you pay for 802.11ax. You'd expect to pay around 600$ for a similar product within the business class market and with a more security-focused firmware.

Right, I was comparing from 2018 because the AX88U is from that year.
Well if that is the pain point here to evolve into more secure networking unit, perhaps a change should be planned by asus.
RMerlin, I am also working for software company, so I do understand what change of platform means and that it is not done very often. And how one source base target same range of products. It is challenging and it also bites one in the butt until you perform some "generation" change. (Legacy eats your tail)

I am not sure why discussion goes so much around the pricing here. 400-600$ is for me same price range.
I was trying to get some insight and understanding how these embedded devices are just bit out of date when it comes to security adherence. But right now the response on the thread is slightly defensive.

 

[RMerlin]

But right now the response on the thread is slightly defensive.

It's just that the same question comes up quite regularly, and often they come from users with no real low-level knowledge, so we have to go at length in making them understand that upgrading a kernel isn't just a matter of copying new files on top of the old ones and call it a day.

While the RT-AX88U came out in 2018, note that it's based on Broadcom's HND platform which was designed a few years before (only the wifi SoC is actually new). The RT-AC86U was the first model to use that new platform. What Broadcom did is simply update to the latest LTS version with the RT-AX88U (updating from 4.1.27 to 4.1.51) with the RT-AX88U. Upgrading to a completely different kernel version would have required too much revalidation/update work.

Manufacturers should devote more time in updating the other components however, as these are the real source of worries IMHO. Back in 2015 I created a table showing the versions of various critical components used by various manufacturers on their flagships routers at that time. The end result was much more scary than the kernel version... I never updated the table, but it's still interesting to check:

https://docs.google.com/spreadsheet...CJ2kdQILQXydpaFBv9CW4/edit?pref=2&pli=1#gid=0

This is what truly worries me (and these are all components that CAN be updated without too much trouble by the device manufacturers, they aren't tied to the SoC's SDK). Check out the OpenSSL versions used at the time by these...

Asus has been doing a good job at keeping at least openssl and dnsmasq up-to-date. I hope that the other manufacturers have started doing the same since that table was created, but somehow I doubt it.

I'd like to eventually update that table, see if things have improved over the past few years, I just lack the time to do so these days.

 

[RMerlin]

For the record, I've done a bit of hardening on my firmware, within what was possible without changing the actual firmware architecture. I've tightened various config file permissions, and have "downgraded" some daemons to run as unprivileged users instead of root. In addition of various component updates, of course - I always update OpenSSL/OpenVPN/Dropbear to the latest version as soon as they are released. I'm still trying to push Asus toward updating OpenVPN on their end, but for some reason they still haven't moved on that front, even tho I've already done all the heavy-lifting involved in moving from 2.3.2 to 2.4.6 (now 2.4.7).

 

[Frankflash]

It's just that the same question comes up quite regularly, and often they come from users with no real low-level knowledge, so we have to go at length in making them understand that upgrading a kernel isn't just a matter of copying new files on top of the old ones and call it a day.

While the RT-AX88U came out in 2018, note that it's based on Broadcom's HND platform which was designed a few years before (only the wifi SoC is actually new). The RT-AC86U was the first model to use that new platform. What Broadcom did is simply update to the latest LTS version with the RT-AX88U (updating from 4.1.27 to 4.1.51) with the RT-AX88U. Upgrading to a completely different kernel version would have required too much revalidation/update work.

Manufacturers should devote more time in updating the other components however, as these are the real source of worries IMHO. Back in 2015 I created a table showing the versions of various critical components used by various manufacturers on their flagships routers at that time. The end result was much more scary than the kernel version... I never updated the table, but it's still interesting to check:

https://docs.google.com/spreadsheet...CJ2kdQILQXydpaFBv9CW4/edit?pref=2&pli=1#gid=0

This is what truly worries me (and these are all components that CAN be updated without too much trouble by the device manufacturers, they aren't tied to the SoC's SDK). Check out the OpenSSL versions used at the time by these...

Asus has been doing a good job at keeping at least openssl and dnsmasq up-to-date. I hope that the other manufacturers have started doing the same since that table was created, but somehow I doubt it.

I'd like to eventually update that table, see if things have improved over the past few years, I just lack the time to do so these days.

what about the fritzs box and Ubiquiti Edge routers

 

[RMerlin]

what about the fritzs box and Ubiquiti Edge routers

No idea how the software in these two look like, I don't think either of these provide complete GPL drops for their firmware code.

 

[Frankflash]

openwrt.org Gpl code for the fritz box can be found

 

[AdrianSz]

Manufacturers should devote more time in updating the other components however, as these are the real source of worries IMHO. Back in 2015 I created a table showing the versions of various critical components used by various manufacturers on their flagships routers at that time. The end result was much more scary than the kernel version... I never updated the table, but it's still interesting to check:

https://docs.google.com/spreadsheet...CJ2kdQILQXydpaFBv9CW4/edit?pref=2&pli=1#gid=0

This is what truly worries me (and these are all components that CAN be updated without too much trouble by the device manufacturers, they aren't tied to the SoC's SDK). Check out the OpenSSL versions used at the time by these...

I agree that user space software shouldn't be a big problem. But I think its all a consequence going for non main stream linux distro in first place (stripping stuff, tuning etc) due to HW constraints. Any rebase of sw version pulling in new deps creates work, trigger updates of build systems and new extended regression scope must be added. That is why today modern embedded devices (Rpi3, Odriod ...) having decent RAM/CPU make peoples life rather simple because choosing main distro such as debian or fedora one gets the packages secure, both due to CVE's fixes and updated frequently. With the snap/appimage/lxc/docker/flatpack movement further allows software providers to increase the frequency of software delivery since sw is decoupled from the OS libs. Containerized confinement is improving security. Not because confinement technique is super secure, but because software can be patched in more frequent way and independent of any other piece of software.

Asus has been doing a good job at keeping at least openssl and dnsmasq up-to-date. I hope that the other manufacturers have started doing the same since that table was created, but somehow I doubt it.

I believe the whole industry was more or less forced doing the same after heart bleed. Mobiles, TVs, routers had all to update openssl due to heartbleed.