Merlin 380.57 Potential Memory Issue

[jegesq]

I don't know if this really is a problem or not, but I have some concerns and want to see if anyone else has observed this same issue.

I am running an AC3200, with Merlin FW 380.57. The router is not using any of the other services (e.g., Samba, FTP, VPN, etc). Just a wireless tri-band router with about 20-25 client devices. Not using Smart Connect, and instead have three separate SSID's.

I upgraded to 380.57 about four days ago. Performed a factory reset after installing the FW and then manually input all of my settings (not many, just SSID's, passwords, etc.). As I said, this is just acting as a wifi router. No USB drives (I have separate servers so no need for USB2 or 3 drives), no media servers, no AI protect, not using Trend Micro, etc. Simply put, other than as a router, all other services turned off (including Upnp off).

So when I first installed 380.57, used memory was showing at about 65 MB (out of 250MB). I noticed this because with prior Merlin FW versions, used memory never exceeded 35-40MB. So right away, the higher used memory was notable. But I just figured that there was a bunch of new code, so it used more memory natively.

A day later, used memory showed at 70MB. Another day later, 80MB. Now today it's at 90MB.

So used memory seems to be creeping upward slowly, but pretty steadily.

Other than this memory creep, my AC3200 is rock solid, and I have no issues at all with it. Range and bandwidth are excellent, and I have no trouble with both wired and wireless devices reaching and exceeding my ISP's max up and download speeds.

I seem to recall reading other messages in a couple of threads discussing memory use that referred to some code ASUS had included in an earlier recent iteration of its FW that caused memory to be consumed until a router would just "run off the cliff" running out of memory after two or three days of continuous use to the point that it would become completely non-responsive, needing to be rebooted. Obviously, that's not happened with me, but this upward creep does suggest that something like that could be occurring.

Anyone else noticed anything like this with this firmware or other versions of fw?

 

[L&LD]

Nothing to worry about. This is memory being managed by the Linux kernel as it should be. And will be released when and as needed.

Search to find more info on this exact topic.

 

[jegesq]

Thanks L&LD, that's sort of what I thought. I guess the new SDK has more "stuff" in it and just consumes more memory. I am familiar with how Linux manages memory, but what I was referring to when I mentioned "other threads", there were some cryptic references in some prior threads about different earlier versions of firmware (official ASUS code?) that I cannot seem to locate again (even though I've searched extensively) where there were reference to ASUS developers including (inadvertently?) some odd debug codes in a release. In other words, the assertion was that ASUS developers had forgotten to take out these debug code routines, and the effect was that the debug codes were consuming all available memory of the devices onto which that version of firmware was loaded. Other posters in some other threads I was able to locate (searching for "memory debug code") refer to this as an issue which causes RAM to be continually consumed until the point where the router simply freezes up because there is not enough available memory, or which were related to issues on some models with 2.4ghz just no longer working. See for example: http://www.snbforums.com/threads/asuswrt-merlin-380-57-is-now-available.29308/page-9#post-226763 (which refers to ASUS's "faceplant" of having inserted "runaway debug code" which causes all memory to be eventually consumed), and this one from the same user: http://www.snbforums.com/threads/asuswrt-merlin-380-57-is-now-available.29308/page-13#post-227323.

The odd thing is that if it isn't some debug code which is causing my AC3200's to incrementally increase memory usage, could it possibly then be the effect and/or impact of the new Network Map routines. My system log file on the AC3200 shows the router repeatedly calling each device on the network about once every hour (to confirm that its still there?). I had also read something about this in an earlier iteration of Merlin (perhaps 378.56_2?), which someone else was saying had some debugging code that caused debug logs to literally consume all available memory, which in the case of AC68's was causing a problem with the router freezing). I also recall Merlin responding to that or a similar thread by saying that this was simply part of the some errant debug codes that was found in ASUS new SDK code and that he had reverted his version so that 378.56_2 no longer had that problem (but then another problem ensued with 2.4ghz band losing connection for seemingly no reason). I may be confusing issues, but the debug scripts filling up memory until the router freezes is not something I dreamed up. See, e.g., http://www.snbforums.com/threads/asuswrt-merlin-378-56_2-is-now-available.27952/page-25#post-227293.

I am fully aware that Linux manages memory differently than Windows. But that doesn't exclude the possibility that some stray bit of errant code or a debug routine designed for testing may have inadvertently been left in a release version of a FW by mistake, nor does it exclude the possibility that such code might have the effect of eventually consuming all available memory, causing a device to freeze. Or to use the phrase someone else came up with in a different thread, "the router uses all available memory until it just runs off a cliff".

I will continue to monitor my GUI to see if memory usage continues to mount steadily and incrementally. I will let you all know what I see.

 

[L&LD]

Thanks L&LD, that's sort of what I thought. I guess the new SDK has more "stuff" in it and just consumes more memory. I am familiar with how Linux manages memory, but what I was referring to when I mentioned "other threads", there were some cryptic references in some prior threads about different earlier versions of firmware (official ASUS code?) that I cannot seem to locate again (even though I've searched extensively) where there were reference to ASUS developers including (inadvertently?) some odd debug codes in a release. In other words, the assertion was that ASUS developers had forgotten to take out these debug code routines, and the effect was that the debug codes were consuming all available memory of the devices onto which that version of firmware was loaded. Other posters in some other threads I was able to locate (searching for "memory debug code") refer to this as an issue which causes RAM to be continually consumed until the point where the router simply freezes up because there is not enough available memory, or which were related to issues on some models with 2.4ghz just no longer working. See for example: http://www.snbforums.com/threads/asuswrt-merlin-380-57-is-now-available.29308/page-9#post-226763 (which refers to ASUS's "faceplant" of having inserted "runaway debug code" which causes all memory to be eventually consumed), and this one from the same user: http://www.snbforums.com/threads/asuswrt-merlin-380-57-is-now-available.29308/page-13#post-227323.

The odd thing is that if it isn't some debug code which is causing my AC3200's to incrementally increase memory usage, could it possibly then be the effect and/or impact of the new Network Map routines. My system log file on the AC3200 shows the router repeatedly calling each device on the network about once every hour (to confirm that its still there?). I had also read something about this in an earlier iteration of Merlin (perhaps 378.56_2?), which someone else was saying had some debugging code that caused debug logs to literally consume all available memory, which in the case of AC68's was causing a problem with the router freezing). I also recall Merlin responding to that or a similar thread by saying that this was simply part of the some errant debug codes that was found in ASUS new SDK code and that he had reverted his version so that 378.56_2 no longer had that problem (but then another problem ensued with 2.4ghz band losing connection for seemingly no reason). I may be confusing issues, but the debug scripts filling up memory until the router freezes is not something I dreamed up. See, e.g., http://www.snbforums.com/threads/asuswrt-merlin-378-56_2-is-now-available.27952/page-25#post-227293.

I am fully aware that Linux manages memory differently than Windows. But that doesn't exclude the possibility that some stray bit of errant code or a debug routine designed for testing may have inadvertently been left in a release version of a FW by mistake, nor does it exclude the possibility that such code might have the effect of eventually consuming all available memory, causing a device to freeze. Or to use the phrase someone else came up with in a different thread, "the router uses all available memory until it just runs off a cliff".

I will continue to monitor my GUI to see if memory usage continues to mount steadily and incrementally. I will let you all know what I see.

The specific process may be different between Linux and Windows, but they are effectively the same.

Ram that is not being utilized by the os as fully as possible is just a resource that is being wasted.

There was a thread where someone else also thought their ram usage was too high, but after a few days or weeks monitoring it, the percentage used simply changed as needed.

I think you are worrying about this too much. If it was a real issue, others would have found and reported it by now.

 

[jegesq]

. . . I think you are worrying about this too much. If it was a real issue, others would have found and reported it by now.

I guess we shall see. And I'm not "worried" about anything. I can always revert back to an earlier version without difficulty, so no worries here. Also, I didn't mean to suggest that memory usage was "too high", just that it was interesting to me to observe that memory usage seemed to incrementally increase every day, with no sign of any increased services being activated, nor any other changes in settings or usage from one day to the next.

I also found it interesting (in a dispassionate sort of way...again, no "worries"), that from one firmware version to the next there would be such a noticeable increase in ram usage, along with an increase in the system log entries showing repeated calls to attached client devices, seemingly for every device with an assigned IP, occurring all day and every hour of the day. (I'm not at home now so can't post a sample from the system log, but the log entries are pretty voluminous, and repetitive).

I'm sure you're correct, and that this will turn out to be nothing..... But if any other AC3200 users have observed any similar behavior with their routers using Merlin 380.57 (or any other Asus official firmware), I would be interested in learning of the same.

Thanks.

 

[sfx2000]

Nothing to worry about. This is memory being managed by the Linux kernel as it should be. And will be released when and as needed.

Search to find more info on this exact topic.

Concur... not much to worry about - the kernel will mange memory within it's parameters...

What would be a concern is if memory usage continues to grow to to the limit - e.g. memory leak somewhere...

 

[jegesq]

One day later: 95MB of 250MB used.

Upward creep continues.

 

[Puppa]

Did you try pressing the clear button on the system log page? That will consume RAM, but at some point the system will limit the size. At the current rate you will run out of memory in another month...

Sent from my SM-N920V using Tapatalk

 

[Nullity]

Can log log into the router via SSH and issue the "cat /proc/meminfo" command?

See if the "Active:" section increases. Cached things are purely things that are stored in memory but serve no vital purpose, like a webpage that was read from disk.

"Used" memory seems to fool Windows expats, who were sold on the idea that free memory is important for performce (I had way too many POS memory cleaners in my Windows days). When I first started with linux, I saw the high ram usage and asked about in a linux IRC chatroom and they told me "free RAM is wasted RAM", which was sort of a linux epiphany for me.

The important distinction is between active & cached (and maybe buffered?) memory, which are all technically using memory, but in very different ways (vital vs non-vital).

 

[jegesq]

I'll try and log in later via SSH (I'm at work now). But I'm not going to reset anything, since performance doesn't seem to be at all impacted in any way (yet).

I want to let this play out to see if the upward creep continues or, whether as Puppa predicts will occur "at some point the system will limit the size."

We shall see.

 

[Nullity]

I would not even begin to worry until you have no free ram. Honestly though, unless there is a problem, I would not try to "fix" anything.

Are you having some network problems?

 

[jegesq]

Nullity:

At the risk of repeating myself, I'm not going to reset anything, since performance doesn't seem to be at all impacted in any way (yet).

So, no, I am not experiencing any network problems (again...yet).

 

[octopus]

http://www.snbforums.com/threads/memory-buffering-optimization-for-arm-devices.27953/#post-214798

 

[Deleted member 22229]

Unless this was a random issue i disagree with some posts here. When i had my 68U running firmware that shows Ram increasing every day when it hit near 100% it crashed a reboot seem to reset the Ram back to about 19% and the cycle started over. I also noticed just surfing around the UI back and forth would chow up Ram. I now own the AC3100 and this is no longer a issue. There has to be more here then meets the eye. Older firmware did not behave this way.

 

[L&LD]

Unless this was a random issue i disagree with some posts here. When i had my 68U running firmware that shows Ram increasing every day when it hit near 100% it crashed a reboot seem to reset the Ram back to about 19% and the cycle started over. I also noticed just surfing around the UI back and forth would chow up Ram. I now own the AC3100 and this is no longer a issue. There has to be more here then meets the eye. Older firmware did not behave this way.

Which firmware version?

The issue you had is not what is happening here (at least, not yet). You seem to have been experiencing a memory leak.

Normal usage of any device running Vista or later uses the ram as much as possible and releases as needed.

Nothing more here (yet).

If there is a memory leak, that will be interesting that only one person found it after almost a month since it was released.

 

[got_milk]

Normal usage of any device running Vista or later uses the ram as much as possible and releases as needed.

As the old saying goes, "unused RAM is wasted RAM".

This is expected behaviour of any modern operating system - as RAM is much faster than flash memory, it is ideal to store as much (potentially) relevant data in there as possible. Modern Windows installations (and OS X, I believe) will make educated guesses on what programs are run most often and will partially load them into RAM to help with start times.

 

[kvic]

So when I first installed 580.57, used memory was showing at about 65 MB (out of 250MB). I noticed this because with prior Merlin FW versions, used memory never exceeded 35-40MB. So right away, the higher used memory was notable. But I just figured that there was a bunch of new code, so it used more memory natively.

A day later, used memory showed at 70MB. Another day later, 80MB. Now today it's at 90MB.

So used memory seems to be creeping upward slowly, but pretty steadily.

It's very likely memory leak! Somehow for unknown reason in your particular setup, the system is leaking memory.

Keep your router running as long as possible. If it's indeed memory leak, it will eventually run out of memory and crash.

 

[ColinTaylor]

Also bear in mind that the router uses tmpfs for its writable filesystems. So if you've got a log file somewhere that is increasing in size quickly, that might be the cause. IIRC there was a bug in a recent ASUS release where they had left debug logging on for something and it was filling up /tmp.

You can check with:

# df -h /tmp
Filesystem                Size      Used Available Use% Mounted on
tmpfs                   117.0M      2.5M    114.6M   2% /tmp

 

[jegesq]

Another day, and up we continue. Now up to 100MB used. So the slow, but steady upward increase continues.

I tried Colin's suggestion (above): SSH'd into the router, "df -h /tmp". Shows the following:

Filesystem Size Used Available Use% Mounted on
tmpfs 124.8M 976.0K 123.8M 1% /tmp

If I am reading that correctly, it's only 1% usage of the log file stored in /tmp

Looking at Nullity's suggestion of reviewing "cat /proc/meminfo" to see what's active in memory, returned the following:

MemTotal: 255572 kB
MemFree: 159312 kB
Buffers: 352 kB
Cached: 4536 kB
SwapCached: 0 kB
Active: 23504 kB
Inactive: 2172 kB
Active(anon): 21184 kB
Inactive(anon): 640 kB
Active(file): 2320 kB
Inactive(file): 1532 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 0 kB
Writeback: 0 kB
AnonPages: 20836 kB
Mapped: 3192 kB
Shmem: 988 kB
Slab: 57008 kB
SReclaimable: 2176 kB
SUnreclaim: 54832 kB
KernelStack: 528 kB
PageTables: 616 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 127784 kB
Committed_AS: 39072 kB
VmallocTotal: 516096 kB
VmallocUsed: 30468 kB
VmallocChunk: 405372 kB

Any thoughts from those who might recognize something I'm not seeing/understanding here?

 

[L&LD]

http://superuser.com/questions/521551/cat-proc-meminfo-what-do-all-those-numbers-mean


Nothing to see still. I think you have too much time on your hands.