Menu
What's new
By:
Filters
Better Search

RT-AX88U 386.3_2 goes about 3-4 weeks before RAM fills up.

  • SNBForums Code of Conduct

    SNBForums is a community for everyone, no matter what their level of experience.

    Please be tolerant and patient of others, especially newcomers. We are all here to share and learn!

    The rules are simple: Be patient, be nice, be helpful or be gone!

theirongiant

theirongiant

Regular Contributor
I started sending all my RT-AX88U logs to a syslog server running in a docker container. Everything was fine for about a month, until I noticed one day when I logged into the router page the RAM and CPU bars weren't animating anymore. When I tried to ssh into the router, it would say 'connection reset by peer.'

So I rebooted the router and checked the logs that were captured on the syslog server.

For about a week, the logs indicated "no space left on device."
 
Last edited:
Really frustrating... I keep trying to add a comment and it says: "Oops! We ran into some problems. Please try again later. More error details may be in the browser console."
 
I'm going to add a cron job to my router to send the "top" command once an hour to the syslog server. Unfortunately, I can't seem to show this here because the forum keeps throwing a 403 Forbidden (I had to check the javascript console to figure this out). Gonna send a message to the admins.
 
What client devices are throwing these errors? No issues here from desktop and laptop computers.
 
theirongiant said:
Really frustrating... I keep trying to add a comment and it says: "Oops! We ran into some problems. Please try again later. More error details may be in the browser console."
Click to expand...

The website filters key words that are related to administration commands of Linux/Windows and key file references. If you are using mentioning key commands or listing a full pathname to certain directories, insert spaces in the filename path and/or commands.

e.g. "/ e t c /filename" to get the point across

I don't know of any single document that explains all the "hit" words/phrases.
 
Jeffrey Young said:
The website filters key words that are related to administration commands of Linux/Windows and key file references. If you are using mentioning key commands or listing a full pathname to certain directories, insert spaces in the filename path and/or commands.

e.g. "/ e t c /filename" to get the point across

I don't know of any single document that explains all the "hit" words/phrases.
Click to expand...
Don’t you think it’s a bit strict for a forum dedicated to troubleshooting small business network equipment, some of which encourages or even requires remote access and scripting via ssh? How are people supposed to share tips on how to automate things?
 
Here's a little script you can try that can handle shrinking an open log file in most cases.....two parameters....the name of the logfile and the number of lines to keep in the log...

Code: 
#!/bin/sh

#check-log

scr_name="$(basename $0)[$$]"

# Shrink logfile to last 100 lines

LOGFILE="$1"
if [ "$2" == "" ]; then
        LINES=100
else
        LINES=$2
fi

if [ -f "$LOGFILE" ]; then
        lines=$(wc -l < $LOGFILE)
        if [ $lines -gt $LINES ]; then
                echo "$(tail -n $LINES $LOGFILE)" > $LOGFILE 2>&1
                logger -s -t $scr_name "log: $LOGFILE $lines total lines $((lines-LINES)) lines pruned"
#       else
#               logger -s -t $scr_name "log: $LOGFILE $lines total lines"
        fi
else
        logger -s -t $scr_name "log: $LOGFILE not found"
fi

exit 0
 
That script will fail at some point when the content of however many lines overruns the maximum command buffer size for `/bin/sh` as part of the `echo $(tail ...)`

Safest is to do something like:

Bash: 
tmpfilename="/tmp/mktemp.$$"
tail -n $LINES $LOGFILE > $tmpfilename
cat $tmpfilename > $LOGFILE
rm -f $tmpfilename

The important thing is to NOT remove $LOGFILE, because the logger is holding that file open. You have to ensure that you overwrite the existing file (which the original script does) with the content you wish to persist.
 
ccleeland said:
That script will fail at some point when the content of however many lines overruns the maximum command buffer size for `/bin/sh` as part of the `echo $(tail ...)`
Click to expand...
May be due to the busybox implementations and the way the do/not do buffering, but I've never run into any problems. The problem with your solution, is that you are putting the intermediate file in /tmp, which is most likely where he is already running out space.
 
ColinTaylor said:
Sorry, I didn't mean "what router" but what storage device as indicated in the full error message, e.g. /var.
Click to expand...

Thanks for the clarification. Here is a sample of the 'no space left' messages. It would seem that they are all in the /var/ mount point. The filesystem is tmpfs which I assume means a RAM drive. And the culprit does appear to be the strongswan.charon.log file.

Code: 
2021-09-19T23:30:02+00:00 RT-AX88U-27B8-464C3A0-C hostapd: eth7: STA 92:52:4b:14:b6:f2 WPA: group key handshake completed (RSN)
2021-09-19T23:30:03+00:00 RT-AX88U-27B8-464C3A0-C hostapd: eth7: STA 38:f9:d3:1c:e0:1c WPA: group key handshake completed (RSN)
2021-09-19T23:38:40+00:00 RT-AX88U-27B8-464C3A0-C kernel: br0: port 7(eth7) neighbor 8000.2c:30:33:d3:04:9d lost
2021-09-19T23:38:40+00:00 RT-AX88U-27B8-464C3A0-C kernel: br0: topology change detected, propagating
2021-09-20T00:00:21+00:00 RT-AX88U-27B8-464C3A0-C cfg_server: Error locking /var/lock/wiredclientlist.lock: 28 No space left on device
2021-09-20T00:00:21+00:00 RT-AX88U-27B8-464C3A0-C cfg_server: Error unlocking -1: 9 Bad file descriptor
[...]
2021-09-20T01:00:23+00:00 RT-AX88U-27B8-464C3A0-C cfg_server: Error locking /var/lock/wiredclientlist.lock: 28 No space left on device
2021-09-20T01:00:23+00:00 RT-AX88U-27B8-464C3A0-C cfg_server: Error unlocking -1: 9 Bad file descriptor
2021-09-20T01:00:28+00:00 RT-AX88U-27B8-464C3A0-C cfg_server: Error locking /var/lock/cfg_mnt.lock: 28 No space left on device
2021-09-20T01:00:28+00:00 RT-AX88U-27B8-464C3A0-C cfg_server: Error locking /var/lock/wiredclientlist.lock: 28 No space left on device
2021-09-20T01:00:28+00:00 RT-AX88U-27B8-464C3A0-C cfg_server: Error unlocking -1: 9 Bad file descriptor
2021-09-20T01:00:30+00:00 RT-AX88U-27B8-464C3A0-C cfg_server: Error unlocking -1: 9 Bad file descriptor
2021-09-20T01:00:30+00:00 RT-AX88U-27B8-464C3A0-C cfg_server: Error locking /var/lock/clientlist.lock: 28 No space left on device
2021-09-20T01:00:30+00:00 RT-AX88U-27B8-464C3A0-C cfg_server: Error unlocking -1: 9 Bad file descriptor
2021-09-20T01:00:33+00:00 RT-AX88U-27B8-464C3A0-C cfg_server: Error locking /var/lock/wiredclientlist.lock: 28 No space left on device
2021-09-20T01:00:33+00:00 RT-AX88U-27B8-464C3A0-C cfg_server: Error unlocking -1: 9 Bad file descriptor
[...]
2021-09-20T01:00:34+00:00 RT-AX88U-27B8-464C3A0-C conn_diag[2069]: Error locking /var/lock/conn_diag.lock: 28 No space left on device
2021-09-20T01:00:34+00:00 RT-AX88U-27B8-464C3A0-C conn_diag[2069]: Error locking /var/lock/conn_diag.lock: 28 No space left on device
2021-09-20T01:00:34+00:00 RT-AX88U-27B8-464C3A0-C conn_diag[2069]: Error locking /var/lock/conn_diag.lock: 28 No space left on device
[...]
2021-09-20T01:00:34+00:00 RT-AX88U-27B8-464C3A0-C conn_diag[2069]: Error locking /var/lock/conn_diag.lock: 28 No space left on device
2021-09-20T01:00:34+00:00 RT-AX88U-27B8-464C3A0-C conn_diag[2069]: Error locking /var/lock/wlcevent.lock: 28 No space left on device
2021-09-20T01:00:34+00:00 RT-AX88U-27B8-464C3A0-C conn_diag[2069]: Error unlocking -1: 9 Bad file descriptor
[...]
2021-09-23T06:53:08+00:00 RT-AX88U-27B8-464C3A0-C conn_diag[2069]: Error locking /var/lock/conn_diag.lock: 28 No space left on device
2021-09-23T06:53:08+00:00 RT-AX88U-27B8-464C3A0-C conn_diag[2069]: Error unlocking -1: 9 Bad file descriptor
2021-09-23T06:53:08+00:00 RT-AX88U-27B8-464C3A0-C conn_diag[2069]: Error locking /var/lock/tg_roaming.lock: 28 No space left on device
2021-09-23T06:53:08+00:00 RT-AX88U-27B8-464C3A0-C conn_diag[2069]: Error unlocking -1: 9 Bad file descriptor
2021-09-23T06:53:08+00:00 RT-AX88U-27B8-464C3A0-C conn_diag[2069]: Error locking /var/lock/roaming.lock: 28 No space left on device
2021-09-23T06:53:08+00:00 RT-AX88U-27B8-464C3A0-C conn_diag[2069]: Error unlocking -1: 9 Bad file descriptor
2021-09-23T06:53:08+00:00 RT-AX88U-27B8-464C3A0-C conn_diag[2069]: Error locking /var/lock/tg_bsd_diag.lock: 28 No space left on device
2021-09-23T06:53:08+00:00 RT-AX88U-27B8-464C3A0-C conn_diag[2069]: Error unlocking -1: 9 Bad file descriptor
2021-09-23T06:53:08+00:00 RT-AX88U-27B8-464C3A0-C conn_diag[2069]: Error locking /var/lock/conn_diag.lock: 28 No space left on device
2021-09-23T06:53:08+00:00 RT-AX88U-27B8-464C3A0-C conn_diag[2069]: Error locking /var/lock/conn_diag.lock: 28 No space left on device
2021-09-23T06:53:09+00:00 RT-AX88U-27B8-464C3A0-C cfg_server: Error locking /var/lock/networkmap.lock: 28 No space left on device
2021-09-23T06:53:09+00:00 RT-AX88U-27B8-464C3A0-C cfg_server: Error unlocking -1: 9 Bad file descriptor
2021-09-23T06:53:09+00:00 RT-AX88U-27B8-464C3A0-C cfg_server: Error locking /var/lock/clientlist.lock: 28 No space left on device
2021-09-23T06:53:09+00:00 RT-AX88U-27B8-464C3A0-C cfg_server: Error unlocking -1: 9 Bad file descriptor
 
Last edited:
Asus already fixed the strongswan log issue upstream a few months ago, the fix will be implemented downstream once they resume releasing GPL updates.
 
ccleeland said:
That script will fail at some point when the content of however many lines overruns the maximum command buffer size for `/bin/sh` as part of the `echo $(tail ...)`

Safest is to do something like:

Bash: 
tmpfilename="/tmp/mktemp.$$"
tail -n $LINES $LOGFILE > $tmpfilename
cat $tmpfilename > $LOGFILE
rm -f $tmpfilename

The important thing is to NOT remove $LOGFILE, because the logger is holding that file open. You have to ensure that you overwrite the existing file (which the original script does) with the content you wish to persist.
Click to expand...

Thanks for the suggestion. I expanded on it a bit for a crontab and placed the script in the /jffs folder. Unfortunately, this did *not* shrink the log file as expected. I tried 100 lines, but it's still nearly 20MB.

Bash: 
#!/bin/sh

LINES="$1"
LOGFILE="$2"

if [[ $# -lt 2 ]]  # if the script did not receive the two required arguments...
then
        echo "Error: you must provide two arguments to run this script.  Example: logtrimmer.sh 999 /path/to/file.log"
        exit 0
fi

tmpfilename="/tmp/mktemp.$$"
tail -n $LINES $LOGFILE > $tmpfilename
cat $tmpfilename > $LOGFILE
rm -f $tmpfilename
echo "Log file $LOGFILE has been trimmed to the last $LINES lines."

And the actual cron job:
Code: 
0 0 * * * /jffs/scripts/logtrimmer.sh 1000 /var/log/strongswan.charon.log
 
Last edited:
WTF...

The strongswan charon process appears to be holding the ENTIRE log file in memory and writing it back again to disk after it's been overwritten.

Observe...

Code: 
admin@RT-AX88U-27B8:/jffs/scripts# ./logtrimmer.sh 100 "/var/log/strongswan.charon.log" ; ls -la /var/log/
+ LINES=100
+ LOGFILE=/var/log/strongswan.charon.log
+ [[ 2 -lt 2 ]]
+ tmpfilename=/tmp/mktemp.17559
+ tail -n 100 /var/log/strongswan.charon.log
+ cat /tmp/mktemp.17559
+ echo Log file /var/log/strongswan.charon.log has been trimmed to the last 100 lines.
Log file /var/log/strongswan.charon.log has been trimmed to the last 100 lines.
drwxr-xr-x    3 admin    root           180 Sep 27 17:13 .
drwxrwxrwt   19 admin    root           520 Sep 28 16:52 ..
-rw-rw-rw-    1 admin    root            11 Sep 27 18:40 commit_ret
drwx------    4 admin    root            80 May  4  2018 cores
srw-rw-rw-    1 admin    root             0 May  4  2018 log
-rw-r--r--    1 admin    root         76963 Sep 28 16:53 log.nmbd
-rw-r--r--    1 admin    root           288 Sep 27 17:13 log.smbd
-rw-r--r--    1 admin    root             0 Sep 27 17:13 samba.log
-rw-rw-rw-    1 admin    root          7724 Sep 28 16:55 strongswan.charon.log

...Spongebob Card: "a few moments later..."

admin@RT-AX88U-27B8:/jffs/scripts# ls -la /var/log
drwxr-xr-x    3 admin    root           180 Sep 27 17:13 .
drwxrwxrwt   19 admin    root           520 Sep 28 16:52 ..
-rw-rw-rw-    1 admin    root            11 Sep 27 18:40 commit_ret
drwx------    4 admin    root            80 May  4  2018 cores
srw-rw-rw-    1 admin    root             0 May  4  2018 log
-rw-r--r--    1 admin    root         76963 Sep 28 16:53 log.nmbd
-rw-r--r--    1 admin    root           288 Sep 27 17:13 log.smbd
-rw-r--r--    1 admin    root             0 Sep 27 17:13 samba.log
-rw-rw-rw-    1 admin    root      18795658 Sep 28 16:55 strongswan.charon.log

I just can't win...
 
Last edited:
You must log in or register to reply here.

Similar threads

I
RT-AX88U "ISP's DHCP does not function properly" but manual udhcpc call works
Replies
4
Views
3K
Lynx
Lynx
Similar threads
Thread starter Title Forum Replies Date
N FW Update Issue from 386.3_2 to latest on ASUS RT-AX88U Asuswrt-Merlin 10
W Latest firmware 388.7 for AX88U and AX11000 and 386.13 for AC88U Asuswrt-Merlin 3
Z Asus AX88u slow on 2.4ghz wifi on certain devices Asuswrt-Merlin 9
N Nightmare for AI mesh pairing with RT-AX88U Asuswrt-Merlin 12
D L2TP client on ax88U pro running latest merlin firmware how to add device Asuswrt-Merlin 0
StephanK RT-AX88U Pro Wireguard Asuswrt-Merlin 5
H RT-AX88U Merlin Parental Control not working Asuswrt-Merlin 3
N WiFi issues with latest version 388.8_2 on ASUS RT-AX88U Asuswrt-Merlin 7
A RT-AX88U - 3004.388.8_2 - SSH connection closed after few minutes from reboot Asuswrt-Merlin 10
S Merlin firmware and 2.4GHz band drops on AX88u and BE88u Asuswrt-Merlin 4

Similar threads

Latest threads

Support SNBForums w/ Amazon

If you'd like to support SNBForums, just use this link and buy anything on Amazon. Thanks!

Sign Up For SNBForums Daily Digest

Get an update of what's new every day delivered to your mailbox. Sign up here!

Staff online

Members online

Total: 971 (members: 31, guests: 940)
Top