After many tests, here is my understanding of the "No space left on device" crash on RT-AC68U:
Affected models: Asus RT-AC68U, RT-AC1900P and RT-AC66U_B1
Affected firmware: 386.7_2 and earlier versions
Fixed firmware: 386.9
Tested firmware: 386.7 to 386.12
Test environment:
One RT-AC68U
Restore factory settings every time when testing, only turn off wifi and turn on SSH, no other settings.
Problem Description:
The RT-AC68U series models have 256MB of RAM, like most models, 50% of the RAM is dynamically divided into
/tmp
storage, so there should be 128MB of
/tmp
storage space on the RT-AC68U.
Although
/tmp
reports more than 120MB free space before uploading, it returns "No space left on device" error after uploading or writing more than 90MB files to
/tmp
, after which the shell crashes and a GUI (httpd) crash is observed.
Crash manifests itself in the following two situations:
1) Entering any command in SSH will return "-sh: can't fork" and the command will not be executed,
2) Sometimes the GUI can be logged in, but in some tests it cannot be logged in. The browser reports that the website is damaged or currently unavailable. In cases where the GUI can be successfully logged in, the CPU, RAM and other information on the website map cannot be loaded, but the reboot button works.
Steps to reproduce:
On 386.7 and 386.7_2 firmware (earlier versions not tested yet) you can reproduce this way:
1) Use scp to upload files larger than 90MB to the
/tmp
directory of the router
2) It can be observed that the "No space left on device" error is returned during the upload process
3) Log into SSH (if this is possible at this time)
4) Entering any command will return "-sh: can't fork"
Or use this method:
1) Copy files larger than 90MB to a USB drive and plug the drive into the RT-AC68U USB port
2) Log in to SSH and use the
cp
command to copy the file to the
/tmp
directory
3) "No space left on device" error will be returned during the copy process
4) After returning this error, entering any command will return "-sh: can't fork".
Problem causes:
Unknown
Checked
RMerlin's change log, and checked
Asuswrt's change log, there is no information mentioning that this error is fixed. What is currently known is that the error does not affect Asuswrt-Merlin 386.9 and Asuswrt 386_50757 versions and above.
Influence level:
High. It will affect the normal operation of the router.
This error can only be fixed by rebooting the router (freeing up RAM) since
rm
and any delete commands cannot be executed. If passed through a script, note that the
reboot
command will also fail, in which case the user will be required to perform a cold reboot via the power button.
Please note that wifi and internet access were not tested in the test.
Mitigation measures:
When the error occurs, reboot the router. As mentioned above, the
reboot
command may not be available, but the GUI reboot button will be available, It's worth pointing out that the GUI itself is not always available.
Or:
Update to 386.9 and later versions. 386.9 to 386.12 (the latest versions so far) have fixed this error.
Some screenshots:
Screenshot of unsuccessful login to the router via SSH after the "No space left on device" occurred.
In another test, no commands could be executed after successfully logging in via SSH.
For more information see earlier tests:
https://www.snbforums.com/threads/s...ns-merlin-auto-update-solutions.87044/page-12
References:
https://www.downtowndougbrown.com/2023/05/what-happened-with-asus-routers-this-morning/
https://www.snbforums.com/threads/router-lockup-cant-fork-on-command-line.32097/