Router lockup: "can't fork" on command line

[theirongiant]

Hey @RMerlin, I wanted you to know that I added a cronjob to clear the log file, but the memory usage is still climbing steadily. Anything else I should look for?

Here's five minutes worth of overall memory usage... I see that 'cache' has increased 176 KB over a span of 5 minutes. This translates to 2,112KB/minute or 50,688 KB / day. At 50MB/day, it would exhaust the RAM in a week or so.

1. Is there any way for me to monitor or track what is consuming this 'cache' space?
2. Is there a way to purge it without rebooting?

0:00
Mem total:903560 anon:45164 map:18712 free:252256
slab:280484 buf:0 cache:232444 dirty:0 write:0

1:00
Mem total:903560 anon:45288 map:18712 free:252200
slab:280516 buf:0 cache:232476 dirty:0 write:0

2:00
Mem total:903560 anon:45160 map:18712 free:252236
slab:280536 buf:0 cache:232516 dirty:0 write:0

3:00
Mem total:903560 anon:45160 map:18712 free:252040
slab:280608 buf:0 cache:232548 dirty:0 write:0

4:00
Mem total:903560 anon:45164 map:18712 free:252336
slab:280468 buf:0 cache:232588 dirty:0 write:0

5:00
Mem total:903560 anon:45160 map:18712 free:251852
slab:280612 buf:0 cache:232620 dirty:0 write:0

 

[theirongiant]

Also, df -h shows one instance of tmpfs filesystem consuming 156.4 MB mounted at /var. Checking actual disk space with du -c -hd 1 /var, it thinks that /var is only 1.1 MB total. Are there virtual devices representing a RAM disk in /var, even if they're not taking disk space?

admin@RT-AX88U-27B8:/# df -h
Filesystem                Size      Used Available Use% Mounted on
ubi:rootfs_ubifs         69.4M     68.2M      1.2M  98% /
devtmpfs                441.1M         0    441.1M   0% /dev
tmpfs                   441.2M    156.4M    284.8M  35% /var
tmpfs                   441.2M      2.1M    439.1M   0% /tmp/mnt
mtd:bootfs                5.1M      4.2M    992.0K  81% /bootfs
mtd:data                  8.0M      3.3M      4.7M  42% /data
tmpfs                   441.2M      2.1M    439.1M   0% /tmp/mnt
tmpfs                   441.2M      2.1M    439.1M   0% /tmp
/dev/mtdblock9           63.0M     14.0M     49.0M  22% /jffs
admin@RT-AX88U-27B8:/# du -c -hd 1 /var
0    /var/nmbd
0    /var/empty
4.0K    /var/notice
244.0K    /var/lock
4.0K    /var/spool
120.0K    /var/lib
0    /var/webmon
0    /var/samba
0    /var/tmp
12.0K    /var/cache
0    /var/siproxd
0    /var/zebra
4.0K    /var/udhcpd
0    /var/ppp
0    /var/state
188.0K    /var/run
536.0K    /var/log
1.1M    /var
1.1M    total

 

[ColinTaylor]

Hey @RMerlin, I wanted you to know that I added a cronjob to clear the log file, but the memory usage is still climbing steadily.

What commands are you using to clear the log file?

Here's five minutes worth of overall memory usage... I see that 'cache' has increased 176 KB over a span of 5 minutes. This translates to 2,112KB/minute or 50,688 KB / day. At 50MB/day, it would exhaust the RAM in a week or so.

The 'cache' is what is used to cache disk writes. So it is perfectly normal to see it increase in this way if you are constantly appending to log files, for example. It doesn't mean the RAM will be exhausted though because once RAM usage reaches about 97% it will start discarding and reusing the oldest blocks.

 

[theirongiant]

What commands are you using to clear the log file?


The 'cache' is what is used to cache disk writes. So it is perfectly normal to see it increase in this way if you are constantly appending to log files, for example. It doesn't mean the RAM will be exhausted though because once RAM usage reaches about 97% it will start discarding and reusing the oldest blocks.

I had initially used this:

0 0 * * * rm /var/log/strongswan.charon.log

However, strongswan did not start writing a new log file.

Adding this resulted in a zero byte file created at the time of the cron job:

0 0 * * * rm /var/log/strongswan.charon.log; touch /var/log/strongswan.charon.log

Which is odd, considering the file is owned by admin:root and the process is running as root.

Then for laughs, I tried restarting the IPSec daemon:

/usr/sbin/ipsec restart

This had three immediate effects:

1. The cache had a significant flush (it's down to 75 M from 232 M);
2. The router GUI also reflects a significant increase in free RAM;
3. The log file started filling up again, as normal.

It would seem that there are still memory management issues with the ipsec daemon.

I'm going to amend the crontab as follows:

0 0 * * * rm /var/log/strongswan.charon.log; /usr/sbin/ipsec restart

This will delete the log file and restart the ipsec daemon at midnight. I'm hoping this will result in long term stability!

 

[ColinTaylor]

I had initially used this:

0 0 * * * rm /var/log/strongswan.charon.log

This is a common mistake. If the process has opened the file in "write mode" rather than "append mode" then rm'ing the file doesn't release the disk space. The file is still in use and being written to, only the directory entry has been removed.

The properly remove the file and reclaim the disk space the process needs to be terminated first, the file can then be deleted and the process restarted.

My guess for the best way to clear down the log file would be:

# service ipsec_stop
# rm /var/log/strongswan.charon.log
# service ipsec_start

Actually it looks like the log file is overwritten every time the service starts up anyway. So all that needs to be done is:

service ipsec_restart

 

[theirongiant]

Thank you. I think I edited my comment after posting the reply, so you might have missed the change I made.

The cronjob now reads as follows:

0 0 * * * rm /var/log/strongswan.charon.log; /usr/sbin/ipsec restart

 

[ColinTaylor]

Thank you. I think I edited my comment after posting the reply, so you might have missed the change I made.

The cronjob now reads as follows:

0 0 * * * rm /var/log/strongswan.charon.log; /usr/sbin/ipsec restart

As I said above the rm command is ineffective at releasing disk space because the process is still running (you're just altering the directory entry, which is why your df output didn't match your du output). Your cronjob will work because the "/usr/sbin/ipsec restart" is overwriting the file. So the rm command is redundant in this case.

P.S. It is generally advised to use the router's service command to stop/start/restart services rather than invoking them directly, although in this case it makes little difference (the service command additionally restarts the firewall and dnsmasq).

 

[theirongiant]

Thanks. I tried the “service” command first, but the internal --help info is very sparse, and it doesn’t have an obvious “list” or “--status-all” argument like Ubuntu or RedHat. Is the list of services documented somewhere?

 

[L&LD]

No, they are not.

 

[theirongiant]

No, they are not.

Could an enterprising sysadmin peek around in /etc/init.d or /etc/rc3.d and figure them out?

 

[ColinTaylor]

Thanks. I tried the “service” command first, but the internal --help info is very sparse, and it doesn’t have an obvious “list” or “--status-all” argument like Ubuntu or RedHat. Is the list of services documented somewhere?

Alas, no. You have to dig through the services.c source code.

 

[theirongiant]

Alas, no. You have to dig through the services.c source code.

It looks as though someone else asked Mr. Wizard this question seven years ago. And some new guy revived it from the dead with a list of some services he dug out of the source code.