Out of Memory Errors 388.1 / No space left on device [SOLUTIONS]

[northumberland]

Merlin is right about the random mac stuff, absolutely useless and likely to cause problems. Android users can turn it off per connection or overall in developer settings (no need to root).
AX88U here and only 1 disconnection that I know of in the past month, but I did have to cold reboot to get the connection back.

 

[Donga120]

Is this the same issue that completely locks me out of my GT AX 6000 UI? I can’t even access the router via web browser on 388.1 sometimes and all internet goes down..

 

[jtp10181]

Is this the same issue that completely locks me out of my GT AX 6000 UI? I can’t even access the router via web browser on 388.1 sometimes and all internet goes down..

That is a symptom of running out of free RAM, so its possible. Check the logs after rebooting and look back from right before it rebooted to look for errors about memory or free space, etc...

 

[mmacedo]

I'll let them alone for a few days to see if ASUS set a size limit on them and they stop increasing.

@router:/tmp/home/root# sqlite3 /tmp/.diag/wifi_detect.db
SQLite version 3.25.3 2018-11-05 20:37:38
Enter ".help" for usage hints.
sqlite> select count () from DATA_INFO;
14496

14,496 / 6 (records/minute) = 2,416 minutes = 1 day, 16 hours, 16 minutes. It's pretty close to router's uptime:

Hopefully, in a few days it'll start dropping older records as it adds new ones to the table.

Files didn't stop increasing. After 4 days 6 hour(s) 35 minute(s) of Uptime this is what I got under /tmp/.diag:

???@router:/jffs/scripts# ls -1lSA /tmp/.diag
-rw-rw-rw-    1   4792320 Jan 27 22:11 wifi_detect.db
-rw-rw-rw-    1   2785280 Jan 27 22:11 eth_detect.db
-rw-rw-rw-    1    720896 Jan 27 22:11 net_detect.db
-rw-rw-rw-    1    487424 Jan 27 22:11 sys_detect.db
-rw-rw-rw-    1     20480 Jan 23 15:42 channel_change.db
-rw-rw-rw-    1     20480 Jan 25 20:29 port_status_change.db
-rw-rw-rw-    1     20480 Jan 23 15:42 port_status_usb_change.db
-rw-rw-rw-    1     20480 Jan 23 16:19 sys_setting.db
-rw-rw-rw-    1     20480 Jan 23 16:19 wifi_setting.db
???@router:/jffs/scripts# sqlite3 /tmp/.diag/wifi_detect.db 'select count () from DATA_INFO;'
36798

So, I've just been brave and done the same as @TF1470:

???@router:/jffs/scripts# service stop_conn_diag

Done.
???@router:/jffs/scripts# rm -r /tmp/.diag

I'll also add the service stop_conn_diag command to my services-start.

I'll open a ticket with ASUS and refer them to this thread,

 

[Donga120]

That is a symptom of running out of free RAM, so its possible. Check the logs after rebooting and look back from right before it rebooted to look for errors about memory or free space, etc...

Currently have this... will monitor

 

[lamentary]

Really outside of my area of expertise, and I'm not sure if this model uses a different version of conn_diag or not, but it appears that there have been some efforts at correcting a memory leak for conn_diag in the DSL-AX82U. Whether that can/will find its way to other models, I have no idea.

DSL-AX82U｜Modem Routers｜ASUS Global

www.asus.com

 

[lamentary]

I ended up going a slightly different route.

Without specifically knowing what conn_diag is used for, instead of killing it altogether, I wrote a shell script (in my rudimentary fashion) that deletes entries older than 5 mins from the files that mmacedo mentioned previously, and set a cron job to run it every minute (I wanted to keep them separate in case I wanted to run it manually).

I had noted that my various db tables are growing at different rates than mmacedo's were, particularly the stainfo.db, but the cron job/sql combos seem to have abated a lot of the nastiness of growth. I'll post more here after some time has passed.

I added this to my /jffs/scripts/init-start:

cru a cleanWifiDetect "* * * * * /bin/sh /jffs/scripts/clean-dbs.sh"

Then I created the following shell script (in the same folder) named clean-dbs.sh:

#!/bin/sh

### WIFI detect
echo "wifi_detect.db count before delete"
/usr/sbin/sqlite3 /tmp/.diag/wifi_detect.db 'select count() FROM DATA_INFO;'

/usr/sbin/sqlite3 /tmp/.diag/wifi_detect.db 'delete from DATA_INFO WHERE strftime("%s","now") - data_time > 300;'

echo "wifi_detect.db count after delete"
/usr/sbin/sqlite3 /tmp/.diag/wifi_detect.db 'select count() FROM DATA_INFO;'

### STA info
echo "stainfo.db count before delete"
/usr/sbin/sqlite3 /tmp/.diag/stainfo.db 'select count() FROM DATA_INFO;'

/usr/sbin/sqlite3 /tmp/.diag/stainfo.db 'delete from DATA_INFO WHERE strftime("%s","now") - data_time > 300;'

echo "stainfo.db count after delete"
/usr/sbin/sqlite3 /tmp/.diag/stainfo.db 'select count() FROM DATA_INFO;'

### ETH detect
echo "eth_detect.db count before delete"
/usr/sbin/sqlite3 /tmp/.diag/eth_detect.db 'select count() FROM DATA_INFO;'

/usr/sbin/sqlite3 /tmp/.diag/eth_detect.db 'delete from DATA_INFO WHERE strftime("%s","now") - data_time > 300;'

echo "eth_detect.db count after delete"
/usr/sbin/sqlite3 /tmp/.diag/eth_detect.db 'select count() FROM DATA_INFO;'

### NET detect
echo "net_detect.db count before delete"
/usr/sbin/sqlite3 /tmp/.diag/net_detect.db 'select count() FROM DATA_INFO;'

/usr/sbin/sqlite3 /tmp/.diag/net_detect.db 'delete from DATA_INFO WHERE strftime("%s","now") - data_time > 300;'

echo "net_detect.db count after delete"
/usr/sbin/sqlite3 /tmp/.diag/net_detect.db 'select count() FROM DATA_INFO;'

### SYS detect
echo "sys_detect.db count before delete"
/usr/sbin/sqlite3 /tmp/.diag/sys_detect.db 'select count() FROM DATA_INFO;'

/usr/sbin/sqlite3 /tmp/.diag/sys_detect.db 'delete from DATA_INFO WHERE strftime("%s","now") - data_time > 300;'

echo "sys_detect.db count after delete"
/usr/sbin/sqlite3 /tmp/.diag/sys_detect.db 'select count() FROM DATA_INFO;'

I made both executable and have been monitoring my file sizes:

# ls -lah /tmp/.diag/
drw-rw-rw-    2 administ root         260 Jan 28 14:46 .
drwxrwxrwx   21 administ root        1.6K Jan 28 14:46 ..
-rw-rw-rw-    1 administ root       20.0K Jan 28 14:25 channel_change.db
-rw-rw-rw-    1 administ root       20.0K Jan 28 14:46 eth_detect.db
-rw-rw-rw-    1 administ root       20.0K Jan 28 14:46 net_detect.db
-rw-rw-rw-    1 administ root       20.0K Jan 28 14:25 port_status_change.db
-rw-rw-rw-    1 administ root       20.0K Jan 28 14:25 port_status_usb_change.db
-rw-rw-rw-    1 administ root       48.0K Jan 28 14:46 stainfo.db
-rw-rw-rw-    1 administ root       20.0K Jan 28 14:34 stainfo_stable.db
-rw-rw-rw-    1 administ root       20.0K Jan 28 14:46 sys_detect.db
-rw-rw-rw-    1 administ root       20.0K Jan 28 14:25 sys_setting.db
-rw-rw-rw-    1 administ root       28.0K Jan 28 14:46 wifi_detect.db
-rw-rw-rw-    1 administ root       20.0K Jan 28 14:25 wifi_setting.db

Thus far, my file sizes have been remaining steady. If this holds true for a few days, I will likely extend the time for records in their respective files prior to deleting them.

 

[mmacedo]

I ended up going a slightly different route.

Without specifically knowing what conn_diag is used for, instead of killing it altogether, I wrote a shell script (in my rudimentary fashion) that deletes entries older than 5 mins

Nice approach, I'll do the same. Thxs!

 

[lamentary]

Nice approach, I'll do the same. Thxs!

Thanks, I expect the files will grow to a point because of the way SQL works (delete won't free up space, bc growing the file size is an expensive process), but they will hopefully become "right-sized" at some point to the largest size over a 5 minute period and hold steady there. How big that will be is yet to be seen.

 

[lamentary]

For the sake of knowing what it's doing and comparison, I pushed the echos to a file in my jffs folder (not a good idea long term, bc it's going to eventually cause its own issues, but I'll sed the logging out after. Here's the first output:

********************************************************************************
*****                      Running Clean Dbs                               *****
********************************************************************************
wifi_detect.db count before delete
36
wifi_detect.db count after delete
30
stainfo.db count before delete
216
stainfo.db count after delete
180
eth_detect.db count before delete
36
eth_detect.db count after delete
30
net_detect.db count before delete
6
net_detect.db count after delete
5
sys_detect.db count before delete
6
sys_detect.db count after delete
5
********************************************************************************
********************************************************************************

 

[Donga120]

So after further investigation I have also found other issues with networkmap. When restarting it, it gets errors trying to create lock files due to using an invalid path. Also gets errors when you sigterm it and it attempts to unlock itself. I decided easiest thing was to just create that path, which I did and then it populated lock files there. Not hopeful but going to test this out and see if it fixes it alone without killing the process every night.

View attachment 46840

This is what I am adding at boot, in the init-start script file
Going to test it without the networkmap cron job first to see if the lock files alone makes a difference.

# Fix for networkmap and Restart processes daily
mkdir -p /var/lock/usr/networkmap
killall -v networkmap
cru a kill_sysState "14 2 * * * killall -v sysstate"
cru a kill_NetworkMap "18 2 * * * killall -v networkmap"

I have the same issue, reading through the rest of the thread now. Seems Adguard may be the issue for me...?

 

[ColinTaylor]

I have the same issue, reading through the rest of the thread now. Seems Adguard may be the issue for me...?

Your problem seems completely different than that discussed in this thread.

 

[jtp10181]

How large are your all wifi_detect.db files getting? Currently mine is at around 2Mb after 4 days of uptime with no pruning, I see a screenshot of one above that is 4Mb. This really is nothing...
My /tmp is still only at 3% capacity.

I also do not have a stainfo.db file which I see someone posted was at 38Mb, that is a little more a concern but still not enough to run /tmp out of space.

Is any of this possibly coming from the "Wifi Radar" data collection?

 

[ColinTaylor]

Is any of this possibly coming from the "Wifi Radar" data collection?

View attachment 47565

This function seems to only use the following file, which is capped at 2MB by default on my RT-AX86U:

# ls -lh /tmp/db/visdata.db
-rw-r--r--    1 admin    root        2.0M Jan 29 16:29 /tmp/db/visdata.db

 

[mmacedo]

How large are your all wifi_detect.db files getting? Currently mine is at around 2Mb after 4 days of uptime with no pruning, I see a screenshot of one above that is 4Mb. This really is nothing...
My /tmp is still only at 3% capacity.

I don't know exactly, because I've stopped conn_diag when wifi_detect.db got to 4MB (the screenshot you mentioned).

But, before I had found out about /tmp/.diag, my /tmp reached 26.4M, as you can see here. Unfortunatelly, I rebooted the router before checking the contents of /tmp/.diag, as I wasn't aware it was the culprit at the time.

Now that I've killed conn_diag it stays at:
tmpfs 206.0M 2.0M 204.0M 1% /tmp

So I believe wifi_detect.db will just keep growing indefinitely.

BTW, I don't use Wifi Radar.

 

[lamentary]

FWIW, here is a snapshot of my scripts output last night at around 11:15 last night on my AX68U (I left my scripts running to keep the dbs trimmed):

********************************************************************************
*****                      Running Clean Dbs                               *****
********************************************************************************
wifi_detect.db count before delete
36
wifi_detect.db count after delete
30
stainfo.db count before delete
216
stainfo.db count after delete
180
eth_detect.db count before delete
36
eth_detect.db count after delete
30
net_detect.db count before delete
6
net_detect.db count after delete
5
sys_detect.db count before delete
6
sys_detect.db count after delete
5
File Sizes:
-rw-rw-rw- 1 administ root 20.0K Jan 28 14:25 channel_change.db
-rw-rw-rw- 1 administ root 20.0K Jan 28 23:18 eth_detect.db
-rw-rw-rw- 1 administ root 20.0K Jan 28 23:18 net_detect.db
-rw-rw-rw- 1 administ root 20.0K Jan 28 21:56 port_status_change.db
-rw-rw-rw- 1 administ root 20.0K Jan 28 14:25 port_status_usb_change.db
-rw-rw-rw- 1 administ root 52.0K Jan 28 23:18 stainfo.db
-rw-rw-rw- 1 administ root 28.0K Jan 28 22:16 stainfo_stable.db
-rw-rw-rw- 1 administ root 20.0K Jan 28 23:18 sys_detect.db
-rw-rw-rw- 1 administ root 20.0K Jan 28 20:49 sys_setting.db
-rw-rw-rw- 1 administ root 28.0K Jan 28 23:18 wifi_detect.db
-rw-rw-rw- 1 administ root 20.0K Jan 28 20:49 wifi_setting.db

Free:
total used free shared buffers cached
Mem: 421864 288196 133668 1876 0 31304
-/+ buffers/cache: 256892 164972
Swap: 0 0 0

********************************************************************************
********************************************************************************

And from this morning's most recent output:

********************************************************************************
*****                      Running Clean Dbs                               *****
********************************************************************************
wifi_detect.db count before delete
36
wifi_detect.db count after delete
30
stainfo.db count before delete
204
stainfo.db count after delete
170
eth_detect.db count before delete
36
eth_detect.db count after delete
30
net_detect.db count before delete
6
net_detect.db count after delete
5
sys_detect.db count before delete
6
sys_detect.db count after delete
5
File Sizes:
-rw-rw-rw- 1 administ root 20.0K Jan 28 14:25 channel_change.db
-rw-rw-rw- 1 administ root 20.0K Jan 29 08:38 eth_detect.db
-rw-rw-rw- 1 administ root 20.0K Jan 29 08:38 net_detect.db
-rw-rw-rw- 1 administ root 20.0K Jan 29 02:42 port_status_change.db
-rw-rw-rw- 1 administ root 20.0K Jan 28 14:25 port_status_usb_change.db
-rw-rw-rw- 1 administ root 52.0K Jan 29 08:38 stainfo.db
-rw-rw-rw- 1 administ root 28.0K Jan 28 22:16 stainfo_stable.db
-rw-rw-rw- 1 administ root 20.0K Jan 29 08:38 sys_detect.db
-rw-rw-rw- 1 administ root 20.0K Jan 28 20:49 sys_setting.db
-rw-rw-rw- 1 administ root 28.0K Jan 29 08:38 wifi_detect.db
-rw-rw-rw- 1 administ root 20.0K Jan 28 20:49 wifi_setting.db

Free:
total used free shared buffers cached
Mem: 421864 296708 125156 2076 0 31664
-/+ buffers/cache: 265044 156820
Swap: 0 0 0

********************************************************************************
********************************************************************************

There has been no increase overnight in my /tmp/.diag folder db sizes. There seems to be a slight increase (~7MB) increase in used RAM utilization. I'll keep my eye on RAM through the day.

As of yesterday at around 11:20 AM local when I did a reboot on the router, based on the Tools/SysInfo page, my RAM usage was:
Total: 411.98 MB
Free: 173.38 MB
Buffers: 0.00 MB
Cache: 21.68 MB

Last night, at 8:47 PM:
Total: 411.98 MB
Free: 126.24 MB
Buffers: 0.00 MB
Cache: 28.39 MB

Right now:

​

Total	411.98 MB
Free	122.34 MB
Buffers	0.00 MB
Cache	30.92 MB

 

[mmacedo]

stainfo.db count before delete
204
stainfo.db count after delete
170

What's in stainfo? Could you run these and post the results, please?

sqlite3 /tmp/.diag/stainfo.db .schema
sqlite3 /tmp/.diag/stainfo.db 'select * from DATA_INFO'

 

[mmacedo]

There has been no increase overnight in my /tmp/.diag folder db sizes. There seems to be a slight increase (~7MB) increase in used RAM utilization. I'll keep my eye on RAM through the day.

Run top and then press the "S" key twice to sort by RSS. Check the memory usage of networkmap and then press the "Q" key to quit.

 

[lamentary]

I suspect stainfo.db it might have something to do with AiMesh?

Schema:

CREATE TABLE DATA_INFO (data_id INTEGER PRIMARY KEY AUTOINCREMENT, node_type TEXT DEFAULT '' NOT NULL,node_ip TEXT DEFAULT '' NOT NULL,node_mac TEXT DEFAULT '' NOT NULL,sta_mac TEXT DEFAULT '' NOT NULL,sta_band TEXT DEFAULT '' NOT NULL,
sta_rssi INT DEFAULT 0 NOT NULL,sta_active INT DEFAULT 0 NOT NULL,sta_tx DOUBLE DEFAULT 0 NOT NULL,sta_rx DOUBLE DEFAULT 0 NOT NULL,sta_tbyte UNSIGNED BIGINT DEFAULT 0 NOT NULL,sta_rbyte UNSIGNED BIGINT DEFAULT 0 NOT NULL,sta_tx_nrate T
EXT DEFAULT '' NOT NULL,sta_rx_nrate TEXT DEFAULT '' NOT NULL,conn_time INT DEFAULT 0 NOT NULL,tx_mcs INT DEFAULT 0 NOT NULL,rx_mcs INT DEFAULT 0 NOT NULL,tx_nss INT DEFAULT 0 NOT NULL,rx_nss INT DEFAULT 0 NOT NULL,bw INT DEFAULT 0 NOT
NULL,txtps UNSIGNED BIGINT DEFAULT 0 NOT NULL,txpr UNSIGNED BIGINT DEFAULT 0 NOT NULL,txpre UNSIGNED BIGINT DEFAULT 0 NOT NULL,conn_if TEXT DEFAULT '' NOT NULL,conn_if_idx INT DEFAULT 0 NOT NULL,conn_if_vidx INT DEFAULT 0 NOT NULL, data
_time TIMESTAMP);
CREATE TABLE sqlite_sequence(name,seq);
CREATE TABLE DB_INFO (info_name TEXT, info_value TEXT);
CREATE INDEX idx_DATA_INFO ON DATA_INFO (data_id);

Contents (abridged - also worth noting are the sequential IDs):

Top (I don't run entware/htop):

 

[lamentary]

Also worth noting: I've been up since yesterday morning and do not have any cron jobs aside from the one that runs the scripts to delete the .db file contents. I did manually restart the httpd and conn_diag at one point last night (no reboot) due to some issues that ended up having nothing to do with them