Memory leak issue

[hyelton]

I've had this issue with several different models over the years and it still occurs today all running merlin. It seems mostly related to the DPI inpsection/Statstics being enabled.

After 15-25 days the ram will fill up to the point where nothing is stable and the GUI is no longer accessable.

Currently happening on latest merlin and on my AC86U also occured last with my 66U-B1

No QoS or anything, just most default, wifi disabled, statstics turned on. Its all I want, but its not stable.

 

[FTC]

Hi, do you have swapping enabled ? could be a bypass to at least prolong that period much longer. Other option would be to program a periodic router reboot every 2 weeks or so.. Note that merlin 384.12 fixed a leak for european models which could cause your sympthoms, but as you say you are already running the latest merlin this should not be your case..

 

[L&LD]

After flashing to the latest firmware, did you happen to do a full reset to factory defaults and then do a minimal and manual configuration to secure the router and connect to your ISP?

Or, have you been doing dirty upgrades to each new firmware?

I have customers with similar usage and 384.12_0 is running stable with no issues for them for almost 4 weeks now (26 days 16 hours).

As FTC suggests, I suggest trying using the amtm Step-by-Step guide to setup a swap file for your router on a spare USB drive.

Please see the link in my signature for both the M&M Config to fully reset your router and also the amtm guide too.

 

[hyelton]

After flashing to the latest firmware, did you happen to do a full reset to factory defaults and then do a minimal and manual configuration to secure the router and connect to your ISP?

Or, have you been doing dirty upgrades to each new firmware?

I have customers with similar usage and 384.12_0 is running stable with no issues for them for almost 4 weeks now (26 days 16 hours).

As FTC suggests, I suggest trying using the amtm Step-by-Step guide to setup a swap file for your router on a spare USB drive.

Please see the link in my signature for both the M&M Config to fully reset your router and also the amtm guide too.

I have an OCD when it comes to the dirty flash/clean install so yeah 2 factory resets usually before setup.

Yeah I've tried the swap file thing before and it still happens.

 

[FTC]

If the problem (unstability) happened with swapping enabled, then probably the leak is caused by something that causes unstability instead of unstability being the consequence of the leak.

Several versions ago and in my previous router I had a similar case where the connections table was leaking.. the number of connections was increasing with time until a moment where it caused unstability besides using much memory.. are you looking at how many connections do you have opened? dose it increase ? If this is the case you can also bypass this by reserving more space for the connections table.. i.e :

To set the connections table to 8192 :
nvram get ct_hashsize < to see current value (default used to be just under 4000)
nvram set ct_hashsize=8192
nvram commit
service restart_conntrack

or low level (without persistance) :

cat /proc/sys/net/ipv4/netfilter/ip_conntrack_buckets < to see current value
echo "8192" > /sys/module/nf_conntrack/parameters/hashsize

.. but of course this is just a guess.. To see where the leak is you may have to compare the output of a 'top' command right after reboot with the same several days after.

 

[brummygit]

Are you using any of the Tend Micro functionality. I have had the same for many releases and routinely reboot my router at least once per week to resolve.

There was a thread recently that implicate the closed source Trend Micro components and I believe it's been reported but no fix yet. Merlin can't resolve as it's in the closed source code.

[Edit] Found the thread https://www.snbforums.com/threads/s...ks-aiprotection-dcd-crashing-is-guilty.55416/

 

[hyelton]

How does one make the statistics use the swap file? As since doing the swap file, it never gets used at all

 

[Vexira]

I just set the reboot schedule to reboot every day, you could set it to once a week.

 

[PeterV]

How does one make the statistics use the swap file? As since doing the swap file, it never gets used at all

Do you use a Diversion with Pixelserv-tls ?
Currently it is an interoperability issue between Trendmicro engine(dcd) and Diversion/Pixelserv-tls. It's due to diversion creating a sub-interface for pixelserv-tls. And Trenmicro can't work correctly with this sub-interface, and crashed every 15-30 minutes, without memory releasing.
This behavior currently identiied only on ac86u model ...
So, you have three option:

• disable trendmicro
• not use pixelserv-tls
• schedule a periodical reboot

 

[FTC]

Do you use a Diversion with Pixelserv-tls ?
Currently it is an interoperability issue between Trendmicro engine(dcd) and Diversion/Pixelserv-tls. It's due to diversion creating a sub-interface for pixelserv-tls. And Trenmicro can't work correctly with this sub-interface, and crashed every 15-30 minutes, without memory releasing.
This behavior currently identiied only on ac86u model ...
So, you have three option:

• disable trendmicro
• not use pixelserv-tls
• schedule a periodical reboot

Hi, how to check for this ? My router is an ax88 but I can see some cases of unexplainable memory consumption, and I am using Diversion with Pixelserv-tls.. Any specific periodic message to look for in the logs or.. processes in the 'TOP' command listing ? Thanks in advance...

 

[PeterV]

Hi, how to check for this ? My router is an ax88 but I can see some cases of unexplainable memory consumption, and I am using Diversion with Pixelserv-tls.. Any specific periodic message to look for in the logs or.. processes in the 'TOP' command listing ? Thanks in advance...

Do you have in the syslog a lot of dcd process crashing messages?
In parallel you can check

cat /proc/meminfo

If Slab, and especially SUnreclaim is growing - there is a result of memory leaks

 

[FTC]

Do you have in the syslog a lot of dcd process crashing messages?
In parallel you can check

cat /proc/meminfo

If Slab, and especially SUnreclaim is growing - there is a result of memory leaks

Thanks for this.
I don't see dcd process crashing messages in my syslog, although slab and sureclaim are quite big.. will keep an eye on those.

 

[hyelton]

Do you use a Diversion with Pixelserv-tls ?
Currently it is an interoperability issue between Trendmicro engine(dcd) and Diversion/Pixelserv-tls. It's due to diversion creating a sub-interface for pixelserv-tls. And Trenmicro can't work correctly with this sub-interface, and crashed every 15-30 minutes, without memory releasing.
This behavior currently identiied only on ac86u model ...
So, you have three option:

• disable trendmicro
• not use pixelserv-tls
• schedule a periodical reboot

Nope.

And trendmicro is only enabled for statistics to work.




Sent from my iPhone using Tapatalk

 

[Elmer]

Recently upgraded from an AC88 to and AX88, and it works wonderfully - except - memory is gobbled up over the course of a day. It typically starts out at 600M and climbs slowly to max over a 30 hour cycle, I, too, run diversion and skynet. If this is a leak, it was certainly unsustainable on the AC88, I had constant crashes. With the AX88 a once a day reboot is handling things so far. I do have a huge number of "smart home" devices but it's not like they are bandwidth bandits. Eventually someone will figure this out. For now the AX88 and a daily reboot are working, and that was not the case with the AC88.

 

[Elmer]

Would also be nice if someone would write a script to monitor memory and do a reboot at some percentage. Be a nice bandaid till things get sorted.

 

[RMerlin]

Would also be nice if someone would write a script to monitor memory and do a reboot at some percentage. Be a nice bandaid till things get sorted.

No, because the way Linux manages memory, "free memory" is meaningless. Memory gets used by buffers until it is required by processes, at which point it will release those buffers.

 

[hyelton]

No, because the way Linux manages memory, "free memory" is meaningless. Memory gets used by buffers until it is required by processes, at which point it will release those buffers.

Well it doesn’t lol. When it fills up the routers gui is no longer accessible and packets randomly start dropping


Sent from my iPhone using Tapatalk

 

[RMerlin]

Well it doesn’t lol. When it fills up the routers gui is no longer accessible and packets randomly start dropping

There's a difference between memory used by processes, and memory used by buffers. What I'm telling you is you cannot just monitor the free memory amount and reboot because it gets low - just copying a large file from a shared USB disk would cause your router to reboot then, as that free memory would get close to zero due to buffering.

 

[Elmer]

There's a difference between memory used by processes, and memory used by buffers. What I'm telling you is you cannot just monitor the free memory amount and reboot because it gets low - just copying a large file from a shared USB disk would cause your router to reboot then, as that free memory would get close to zero due to buffering.

Thanks for the feedback.

My AX88 runs throughout the day at about 970 MB, that is, 30 MB or less free. It's working (my AC88 was constantly crashing). It's odd. I reboot once a day.

There is one thing I'm doing that might be out of the ordinary, and that's link aggregation. I suppose for the sake of science I should run without and see if there's a difference. Haven't seen any posts about link aggregation problems, but maybe that's because hardly anyone is using it.

 

[hyelton]

Thanks for the feedback.

My AX88 runs throughout the day at about 970 MB, that is, 30 MB or less free. It's working (my AC88 was constantly crashing). It's odd. I reboot once a day.

There is one thing I'm doing that might be out of the ordinary, and that's link aggregation. I suppose for the sake of science I should run without and see if there's a difference. Haven't seen any posts about link aggregation problems, but maybe that's because hardly anyone is using it.

It’s the traffic statistics specifically that’s causing my ram usage.


Sent from my iPhone using Tapatalk