/sbin/watchdog - yet another memory leak

[kvic]

Asus ARM routers. 380 series FW. Memory leak in /sbin/watchdog process. Can inspect and monitor in htop (installable from Entware-ng) or by command line:

$ cat /proc/$(pidof watchdog)/status|grep VmRSS

VmRSS: 5348 kB

This process occupies 1500 - 2000 KB of RAM on reboot (I think...can't recall exactly ATM). My RT-AC56U being up for 11 days accumulates over 5 MB.

The leak is slower than the one in /sbin/preinit. From my observation, the process leaks about 4 KB every 15 minutes.

Didn't pay attention when I was on 378.55. So not sure if it's 380 series FW specific..

Anyone else see the leak on ARM routers with a 380 series FW?

 

[L&LD]

Can you give at least one example of an ARM router (thanks)?

If left to itself, will this make the router reboot or otherwise malfunction?

 

[kvic]

Can you give at least one example of an ARM router (thanks)?

If left to itself, will this make the router reboot or otherwise malfunction?

I only verified on RT-AC56U. I suspect the issue exists on RT-AC68 and RT-AC87 as well as newer arm routers.

The leak is slow at a rate of ~4 KB per 15 minutes. Most 256 MB RAM routers will have 150 MB free after reboot. That will take little more than a full year to leak all 150 MB. Assume it's the only leak.

But usually complications show up way before that..leading to spontaenous reboot itself.

 

[L&LD]

Okay, (I could have done the math myself!), so this is basically a non-issue when the firmware will probably be updated half a dozen times before the ram is used?

Are there any symptoms (yet) of this memory leak affecting day to day performance?

I mean actual issues, not anticipated ones that low memory would be known to induce?

 

[kvic]

For people set up to auto reboot routers weekly, they will hardly notice any issue I think.

Any memory leak could end up nasty to system performance and stability...

 

[swetoast]

Can comfirm on RT-AC56U using firmware 380.59_beta2 merlin

leaks 4kb ~15 minutes.

 

[RMerlin]

After half an hour, mine hasn't moved from 6328 kB.

I reviewed the watchdog.c code, and nothing comes up as being potentially leaking anything - very few allocations are done in it. A few fopen(), one single malloc()... If it's really leaking, then the leak would be from outside of the core watchdog code.

Unfortunately, valgrind isn't available on Entware's ARM repo.

 

[RMerlin]

It finally took a bit over an hour before my VmSize increased by a single 4 KB segment.

Without valgrind there's no way to track down the source - watchdog is just an instanced copy of rc, which means a few thousand lines of code related to it.

 

[kvic]

It finally took a bit over an hour before my VmSize increased by a single 4 KB segment.

Without valgrind there's no way to track down the source - watchdog is just an instanced copy of rc, which means a few thousand lines of code related to it.

Shall be looking at VmRSS not VmSize..

VmSize is the process address space. Only a subset is mapped to RAM which is accounted as VmRSS.

I reviewed the watchdog.c code, and nothing comes up as being potentially leaking anything - very few allocations are done in it. A few fopen(), one single malloc()... If it's really leaking, then the leak would be from outside of the core watchdog code.

The leak could have happened in this signal handler here: https://github.com/RMerl/asuswrt-merlin/blob/master/release/src/router/rc/watchdog.c#L3359

One or more of the functions called inside... I didn't look further.

 

[swetoast]

maybe @ryzhov_al would be nice enough to add valgrind to the armv7 repo since its possible to crosscompile it

worst case scenario @RMerlin cross compile it and dump it in the router

export CROSS_COMPILE=arm-fsl-linux-gnueabi-
export CC=${CROSS_COMPILE}gcc
export CPP=${CROSS_COMPILE}cpp
export CXX=${CROSS_COMPILE}g++
export LD=${CROSS_COMPILE}ld
export AR=${CROSS_COMPILE}ar

 ./configure --target=arm-fsl-linux-gnueabi \
              --host=armv7-fsl-linux-gnueabi \
              --prefix=/opt/valgrind \
              CFLAGS=-static
sed -i -e "s#armv7#arm#g" configure

 

[zyxmon]

@swetoast - thanks for the hint (sed -i -e "s#armv7#arm#g"). Now valgrind can be built for armv7.
Test packages are here - http://pkg.entware.net/binaries/armv7/test/

 

[swetoast]

Thank you makes it easier for @kvic and @RMerlin to hunt bugs

 

[kvic]

Very nice collaboration. But I think valgrind doesn't work on softfloat ARM.

@sfx2000 may know more about it..

 

[swetoast]

are you sure about that ? cause looking at their mailing list that only applied to < 3.7.0

it might be worth too try the packages and see if it works

 

[sfx2000]

Very nice collaboration. But I think valgrind doesn't work on softfloat ARM.

@sfx2000 may know more about it..

Should work as long as it's built for the same abi as the kernel/userland... and need to be fairly new release

I'm doing some tinkering at the moment on an armv7, but it's armhf, not armel..

 

[sfx2000]

After half an hour, mine hasn't moved from 6328 kB.

I reviewed the watchdog.c code, and nothing comes up as being potentially leaking anything - very few allocations are done in it. A few fopen(), one single malloc()... If it's really leaking, then the leak would be from outside of the core watchdog code.

Unfortunately, valgrind isn't available on Entware's ARM repo.

might not be the dog, could very well be something that the dog is monitoring...

 

[swetoast]

well on mine and kvic RT-AC56U its steadily rising 4kb every ~15 min

 

[L&LD]

well on mine and kvic RT-AC56U its steadily rising 4kb every ~15 min

And has been calculated to run out of memory in about a year.

 

[swetoast]

yes we know still doesnt make it ok mem leak is a mem leak is a mem leak even if it takes a year to fill up

 

[L&LD]

yes we know still doesnt make it ok mem leak is a mem leak is a mem leak even if it takes a year to fill up

Agree 100% (on mem leak).

On the 'year' part? Power outages, firmware upgrades (with the possibility of the devs fixing this issue) and the sheer time span needed for the issue to normal users to become 'very' concerned about it makes it seem like a make work project to me at this time.