Memory leak in Asuswrt-Merlin

[Adrian Tundrea]

Daily updates are much too frequent, we'll just get swamped with pages and pages of stats. Try once or twice a week.

If you want to collect daily stats I'd suggest you put them into a single spreadsheet and give us a link to it.

You are right. I have created a spreadsheet with them, each information on a sheet. I will keep it updated. It can be viewed here https://docs.google.com/spreadsheets/d/1iJzWRuJCkbn8qqtEcYYALlw1KWVi8PqBY08isNi5pkY/edit?usp=sharing

 

[Adrian Tundrea]

I'm at 79 days uptime with 380.57 on a 66U and 87U. No memory issues, but I'm not using QoS or other "advanced" features.

How much traffic do you do daily? I only have OpenVPN server on it... nothing else extra.

 

[highwire]

How much traffic do you do daily? I only have OpenVPN server on it... nothing else extra.

It's all over the place. Average is about ~4GB/day (streaming), but some days I'll do tens or hundreds of GB in a day.

RT-AC66U hovers at 22% of 234MB. This one is in AP mode.
RT-AC87U hovers at 15% of 250MB. This one is in router mode.

I just reflashed both with 380.58 beta 1 and memory use is the same as it was in 380.57.

 

[Adrian Tundrea]

It's all over the place. Average is about ~4GB/day (streaming), but some days I'll do tens or hundreds of GB in a day.

RT-AC66U hovers at 22% of 234MB. This one is in AP mode.
RT-AC87U hovers at 15% of 250MB. This one is in router mode.

I just reflashed both with 380.58 beta 1 and memory use is the same as it was in 380.57.

My traffic is more liniar in the tens of GB. The used RAM is over 29% right now of 250 RAM. One thing I will try it is disable the OpenVPN server. That seems to be the only special thing I have on my router.

 

[Adrian Tundrea]

Not at this stage. Having memory usage increase over time is expected, as the router is running a number of services. dnsmasq has its own cache, the DPI engine will also generate internal databases, the networkmap will increase its internal list of known clients, etc... Any further debugging would require a lot of work, something not worth doing at this stage unless routers actually DO run out of memory, confirming that there is really a leak worth investigating, and not just some normal increased memory usage.

What makes me doubt that there is really a memory leak is the fact that you experienced the same thing with numerous past firmware releases, some of which have shown to be stable enough for people to reach multiple weeks of uptime. I myself reached three weeks uptime during the Holidays, and my router's free memory was still normal.

I think I have found the leak/cache that is using up the memory. After the discussion with highwire I thought about disabling the OpenVPN server since it was the only difference I had with his configuration and behold for 5h the memory usage has not increased at all. As soon as I disabled OpenVPN it froze at 68-72 MB.

Then I re-enabled OpenVPN and did the ps command. I noticed the client was refreshed every 1h so I check the memory without the client connected, after the first connection and 1h later. Noticed the memory is growing... Here are the results:

After OpenVPN server enabled, but with no client connected:
7768 42 4112 S /etc/openvpn/vpnserver1 --cd /etc/openvpn/server1 --
7770 42 4016 S /etc/openvpn/vpnserver1 --cd /etc/openvpn/server1 --

OpenVPN server enabled, but just after client connected:
7768 42 4396 S /etc/openvpn/vpnserver1 --cd /etc/openvpn/server1 --
7770 42 4016 S /etc/openvpn/vpnserver1 --cd /etc/openvpn/server1 --

OpenVPN server enabled, after the first client refresh (basically one hour later):
7768 42 4476 S /etc/openvpn/vpnserver1 --cd /etc/openvpn/server1 --
7770 42 4016 S /etc/openvpn/vpnserver1 --cd /etc/openvpn/server1 --

After a disconnect/connect of the OpenVPN client:
7768 42 4716 S /etc/openvpn/vpnserver1 --cd /etc/openvpn/server1 --
7770 42 4016 S /etc/openvpn/vpnserver1 --cd /etc/openvpn/server1 --

I have noticed that in version 2.3.9 of OpenVPN multiple memory leaks have been fixed (https://community.openvpn.net/openvpn/wiki/ChangesInOpenvpn23). The firmware (378.56_2) I am having has OpenVPN version 2.3.8. I will update to version 380.58 which includes version OpenVPN 2.3.10 and see how that goes.

 

[Adrian Tundrea]

It is still too early to give any definite answers, but 380.58 seems to fix the issue (or at least improve it to the point where it is not so noticeable). After 10h uptime the free memory hovers around 200 MB. On prior versions it would already be at 190 MB. I will come back later, but for now I would suggest to all those using the OpenVPN server to update to the latest merlin version (380.58) which seems to fix the leak.

 

[L&LD]

There is no leak as stated many times by others above.

The spreadsheet is also less than optimal, btw.

 

[RMerlin]

It is still too early to give any definite answers, but 380.58 seems to fix the issue (or at least improve it to the point where it is not so noticeable). After 10h uptime the free memory hovers around 200 MB. On prior versions it would already be at 190 MB. I will come back later, but for now I would suggest to all those using the OpenVPN server to update to the latest merlin version (380.58) which seems to fix the leak.

I find it surprising that OpenVPN would have such a major (in the sense that it's leaking megabytes, not just a few kilobytes), but it's entirely possible, as they did mention fixing a few memory leaks in 2.3.9.

 

[Adrian Tundrea]

There is no leak as stated many times by others above.

The spreadsheet is also less than optimal, btw.

The leak is there in OpenVPN for sure. Still it may be connected to my exact OpenVPN settings. For example I use a TAP interface which I think is less common. As proof I have 4days 10h of uptime right now and 184 MB free. Right now on the old version I would be in at around 140 MB.

 

[L&LD]

The leak is there in OpenVPN for sure. Still it may be connected to my exact OpenVPN settings. For example I use a TAP interface which I think is less common. As proof I have 4days 10h of uptime right now and 184 MB free. Right now on the old version I would be in at around 140 MB.

Yes, I spoke too soon.

I have read RMerlin's response and now know better!

But it did seem very much like an earlier thread about memory 'leaking' though, in my (weak) defense.

 

[Adrian Tundrea]

A small memory leakage is still present... still coming from OpenVPN I think. But overall this is a trickle compared to the flood it used to be. My stats after almoust 28 days uptime: 148 MB free. So it lost almoust 50 MB from where it starts. I have continually been connected with an OpenVPN client to the router.

 

[kvic]

Personally I think your evidence of memory leak is not convincing..

You have to provide better evidence on the claim that 1) there is memory leak in the firmware, or 2) specially OpenVPN server that you point to.

 

[CiscoX]

No memory leak here.

 

[swetoast]

Cant report the same findings either, been running both client and server at the same time.

 

[Bamsefar]

I can only say that I have something that might be in the same area. Like OP I have a router that only works as router, no WEB server or anything is exposed against internet - the ONLY thing i have is a OpenVPN server 1, when enabled my router will at some point hang 100%. Now

I have only had this twice so far. So I am still trying to figure out what happens, and why. Currently I hav a log script running each hour, that collects some stats - when the next hang occurs I might be able to give some info of what happens.

I'm not ready to say it is a memory leak - I simply have no data that says what happens. But I'll never give up this ;-)

 

[swetoast]

using Openvpn server 1 too on Firmware:380.59_beta2 not getting those issues so it might be a config issue or something not saying that there isnt a bug there but there might be a difference on triggering that bug.

i run my openvpn server with tun instead of tap not sure what the rest is running

 

[eddiez]

Nice to see how a futile occurrence with no practical impact whatsoever can keep grown ups busy for days and weeks ;-)

 

[Bamsefar]

I still have no clu of why this is happening in 380.58 - there is no apparent memory leak at all. Yes I do see some memory numbers going up - but nothing that big to be even a concern for the moment, and I have been trashing my OpenVPN Server 1 during 24h. I'll keep testing a bit more, and if I can't find the issue, I might go up to 380.59 just to test it...

 

[kvic]

I still have no clu of why this is happening in 380.58 - there is no apparent memory leak at all. Yes I do see some memory numbers going up - but nothing that big to be even a concern for the moment, and I have been trashing my OpenVPN Server 1 during 24h. I'll keep testing a bit more, and if I can't find the issue, I might go up to 380.59 just to test it...

How did you stress it?

I think would be fun to see what happens on the router if your mobile client connects & disconnects to the vpn server at a rate of maybe 40 times per minute..

 

[Bamsefar]

Mmm that is indeed a good question ;-)

Mainly I am trying to replicate what I did the two times before I had this problem - but guess what, I can't replicate the issue. Although my script has only ran for 45h now (and will run until reboot) there is NO evidence what so ever for any memory leak with heavy (my definition ;-) ) OpenVPN usage. Now of course one could easily make it harder for the OpenVPN server part - no question about that - but this is for me to understand why and what happens during MY kind of load. Synthetic tests comes later ;-)

You will not be able to do reconnect 40 times a minute since the connection time seems to be around 5 seconds = 20 times - but I see no use for that currently. Why? Well my connection is not always that good, so I have frequent disconnect/connections from time to time - so in a way I do test that real world scenario