What's new

Router lockup: "can't fork" on command line

  • SNBForums Code of Conduct

    SNBForums is a community for everyone, no matter what their level of experience.

    Please be tolerant and patient of others, especially newcomers. We are all here to share and learn!

    The rules are simple: Be patient, be nice, be helpful or be gone!

Hey @RMerlin, I wanted you to know that I added a cronjob to clear the log file, but the memory usage is still climbing steadily. Anything else I should look for?

Here's five minutes worth of overall memory usage... I see that 'cache' has increased 176 KB over a span of 5 minutes. This translates to 2,112KB/minute or 50,688 KB / day. At 50MB/day, it would exhaust the RAM in a week or so.

1. Is there any way for me to monitor or track what is consuming this 'cache' space?
2. Is there a way to purge it without rebooting?

Code:
0:00
Mem total:903560 anon:45164 map:18712 free:252256
slab:280484 buf:0 cache:232444 dirty:0 write:0

1:00
Mem total:903560 anon:45288 map:18712 free:252200
slab:280516 buf:0 cache:232476 dirty:0 write:0

2:00
Mem total:903560 anon:45160 map:18712 free:252236
slab:280536 buf:0 cache:232516 dirty:0 write:0

3:00
Mem total:903560 anon:45160 map:18712 free:252040
slab:280608 buf:0 cache:232548 dirty:0 write:0

4:00
Mem total:903560 anon:45164 map:18712 free:252336
slab:280468 buf:0 cache:232588 dirty:0 write:0

5:00
Mem total:903560 anon:45160 map:18712 free:251852
slab:280612 buf:0 cache:232620 dirty:0 write:0
 
Last edited:
Also, df -h shows one instance of tmpfs filesystem consuming 156.4 MB mounted at /var. Checking actual disk space with du -c -hd 1 /var, it thinks that /var is only 1.1 MB total. Are there virtual devices representing a RAM disk in /var, even if they're not taking disk space?

Code:
admin@RT-AX88U-27B8:/# df -h
Filesystem                Size      Used Available Use% Mounted on
ubi:rootfs_ubifs         69.4M     68.2M      1.2M  98% /
devtmpfs                441.1M         0    441.1M   0% /dev
tmpfs                   441.2M    156.4M    284.8M  35% /var
tmpfs                   441.2M      2.1M    439.1M   0% /tmp/mnt
mtd:bootfs                5.1M      4.2M    992.0K  81% /bootfs
mtd:data                  8.0M      3.3M      4.7M  42% /data
tmpfs                   441.2M      2.1M    439.1M   0% /tmp/mnt
tmpfs                   441.2M      2.1M    439.1M   0% /tmp
/dev/mtdblock9           63.0M     14.0M     49.0M  22% /jffs
admin@RT-AX88U-27B8:/# du -c -hd 1 /var
0    /var/nmbd
0    /var/empty
4.0K    /var/notice
244.0K    /var/lock
4.0K    /var/spool
120.0K    /var/lib
0    /var/webmon
0    /var/samba
0    /var/tmp
12.0K    /var/cache
0    /var/siproxd
0    /var/zebra
4.0K    /var/udhcpd
0    /var/ppp
0    /var/state
188.0K    /var/run
536.0K    /var/log
1.1M    /var
1.1M    total
 
Hey @RMerlin, I wanted you to know that I added a cronjob to clear the log file, but the memory usage is still climbing steadily.
What commands are you using to clear the log file?

Here's five minutes worth of overall memory usage... I see that 'cache' has increased 176 KB over a span of 5 minutes. This translates to 2,112KB/minute or 50,688 KB / day. At 50MB/day, it would exhaust the RAM in a week or so.
The 'cache' is what is used to cache disk writes. So it is perfectly normal to see it increase in this way if you are constantly appending to log files, for example. It doesn't mean the RAM will be exhausted though because once RAM usage reaches about 97% it will start discarding and reusing the oldest blocks.
 
What commands are you using to clear the log file?


The 'cache' is what is used to cache disk writes. So it is perfectly normal to see it increase in this way if you are constantly appending to log files, for example. It doesn't mean the RAM will be exhausted though because once RAM usage reaches about 97% it will start discarding and reusing the oldest blocks.
I had initially used this:

Code:
0 0 * * * rm /var/log/strongswan.charon.log

However, strongswan did not start writing a new log file.

Adding this resulted in a zero byte file created at the time of the cron job:

Code:
0 0 * * * rm /var/log/strongswan.charon.log; touch /var/log/strongswan.charon.log

Which is odd, considering the file is owned by admin:root and the process is running as root.

Then for laughs, I tried restarting the IPSec daemon:

Code:
/usr/sbin/ipsec restart

This had three immediate effects:

1. The cache had a significant flush (it's down to 75 M from 232 M);
2. The router GUI also reflects a significant increase in free RAM;
3. The log file started filling up again, as normal.

It would seem that there are still memory management issues with the ipsec daemon.

I'm going to amend the crontab as follows:

Code:
0 0 * * * rm /var/log/strongswan.charon.log; /usr/sbin/ipsec restart

This will delete the log file and restart the ipsec daemon at midnight. I'm hoping this will result in long term stability!
 
I had initially used this:
Code:
0 0 * * * rm /var/log/strongswan.charon.log
This is a common mistake. If the process has opened the file in "write mode" rather than "append mode" then rm'ing the file doesn't release the disk space. The file is still in use and being written to, only the directory entry has been removed.

The properly remove the file and reclaim the disk space the process needs to be terminated first, the file can then be deleted and the process restarted.

My guess for the best way to clear down the log file would be:
Code:
# service ipsec_stop
# rm /var/log/strongswan.charon.log
# service ipsec_start
Actually it looks like the log file is overwritten every time the service starts up anyway. So all that needs to be done is:
Code:
service ipsec_restart
 
Last edited:
Thank you. I think I edited my comment after posting the reply, so you might have missed the change I made.

The cronjob now reads as follows:

Code:
0 0 * * * rm /var/log/strongswan.charon.log; /usr/sbin/ipsec restart
 
Thank you. I think I edited my comment after posting the reply, so you might have missed the change I made.

The cronjob now reads as follows:

Code:
0 0 * * * rm /var/log/strongswan.charon.log; /usr/sbin/ipsec restart
As I said above the rm command is ineffective at releasing disk space because the process is still running (you're just altering the directory entry, which is why your df output didn't match your du output). Your cronjob will work because the "/usr/sbin/ipsec restart" is overwriting the file. So the rm command is redundant in this case.

P.S. It is generally advised to use the router's service command to stop/start/restart services rather than invoking them directly, although in this case it makes little difference (the service command additionally restarts the firewall and dnsmasq).
 
Last edited:
Thanks. I tried the “service” command first, but the internal --help info is very sparse, and it doesn’t have an obvious “list” or “--status-all” argument like Ubuntu or RedHat. Is the list of services documented somewhere?
 
No, they are not.
 
Thanks. I tried the “service” command first, but the internal --help info is very sparse, and it doesn’t have an obvious “list” or “--status-all” argument like Ubuntu or RedHat. Is the list of services documented somewhere?
Alas, no. You have to dig through the services.c source code.
 

Similar threads

Support SNBForums w/ Amazon

If you'd like to support SNBForums, just use this link and buy anything on Amazon. Thanks!

Sign Up For SNBForums Daily Digest

Get an update of what's new every day delivered to your mailbox. Sign up here!
Top