Merlin 380.57 Potential Memory Issue

[eddiez]

Another day, and up we continue. Now up to 100MB used. So the slow, but steady upward increase continues.

I tried Colin's suggestion (above): SSH'd into the router, "df -h /tmp". Shows the following:

Filesystem Size Used Available Use% Mounted on
tmpfs 124.8M 976.0K 123.8M 1% /tmp

If I am reading that correctly, it's only 1% usage of the log file stored in /tmp

Looking at Nullity's suggestion of reviewing "cat /proc/meminfo" to see what's active in memory, returned the following:

MemTotal: 255572 kB
MemFree: 159312 kB
Buffers: 352 kB
Cached: 4536 kB
SwapCached: 0 kB
Active: 23504 kB
Inactive: 2172 kB
Active(anon): 21184 kB
Inactive(anon): 640 kB
Active(file): 2320 kB
Inactive(file): 1532 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 0 kB
Writeback: 0 kB
AnonPages: 20836 kB
Mapped: 3192 kB
Shmem: 988 kB
Slab: 57008 kB
SReclaimable: 2176 kB
SUnreclaim: 54832 kB
KernelStack: 528 kB
PageTables: 616 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 127784 kB
Committed_AS: 39072 kB
VmallocTotal: 516096 kB
VmallocUsed: 30468 kB
VmallocChunk: 405372 kB

Any thoughts from those who might recognize something I'm not seeing/understanding here?

Why do you think the memory usage is not normal? I mean, you're using the router and it is serving it's tasks. Like with a car driving means it uses petrol. So doing it's job for a router means using the cpu and storing data in memory, both active RAM, temporary data and cached items and data.

 

[kvic]

Looking at Nullity's suggestion of reviewing "cat /proc/meminfo" to see what's active in memory, returned the following:

He might have put you on the right track. Give us a snapshot around similar time tomorrow

 

[ColinTaylor]

He might have put you on the right track. Give us a snapshot around similar time tomorrow

I'd suggest that you also do a "cat /proc/slabinfo" and compare the outputs a day apart.

I don't have the same hardware/firmware as you so can't do a valid comparison, but FWIW whilst the overall free memory is about the same as mine, proportionally you're using it a lot more for userspace stuff (slab/anon).

It might also be worth running "top" and hitting "s" twice to see what it thinks is the most memory hungry process.

Mem total:239712 anon:6964 map:4128 free:155504
 slab:12664 buf:22716 cache:32932 dirty:0 write:0
Swap total:0 free:0
  PID   VSZ VSZRW   RSS^(SHR) DIRTY (SHR) STACK COMMAND
 5200  2872  2124  2424   188  2036     0    84 dnsmasq --log-async
  923  4244   712  1584   460   552     0    84 httpd
 2106  3240   872  1448   504   604     0    84 smbd -D -s /etc/smb.conf
 2090  2380   672  1180   508   500     0    84 nmbd -D -s /etc/smb.conf
    1  2604   384  1144   620   128     0    84 /sbin/init noinitrd
 5270  1528   440   676   252   288     0    84 nas
  364  2596   376   632   508    96     0    84 /sbin/wanduck

 

[jegesq]

http://superuser.com/questions/521551/cat-proc-meminfo-what-do-all-those-numbers-mean

Nothing to see still. I think you have too much time on your hands.

Yes, I agree, nothing to see still.

But then I didn't say I saw anything unusual either (other than memory increasing in a seemingly stepwise fashion, about 5MB per day since I installed this version of the FW). I wanted to let Nullity and Colin know what their suggestions re: memory queries showed. Next time I'll just PM, so as not to waste your valuable time.... or mine.

 

[jegesq]

Why do you think the memory usage is not normal? I mean, you're using the router and it is serving it's tasks. Like with a car driving means it uses petrol. So doing it's job for a router means using the cpu and storing data in memory, both active RAM, temporary data and cached items and data.

I have never asserted that "memory usage is not normal". What I've said is that memory usage creeping upward incrementally day after day has never previously been observed with any other Merlin firmware I've used with this router (AC3200), or any of my other Asus routers over the years. I've been using Merlin's firmware on various Asus routers for at least two years now, and I've never observed this before. In previous iterations, using the same or similar settings (e.g., no DLNA server, no FTP, none of the AI utilities, no SmartConnect, no USB drives attached...just a pretty bare-bones three SSID wifi set up with mostly stock settings other than using Google DNS servers, and always running between 20-25 attached AC client devices), this has not been seen. Used memory always pretty much uniformly hovered around 30% used, and never higher. But with 380.57, used memory has continued to inch upward every day. That's all.

You and others may well be right, and the kernel will manage this with no adverse effects every manifesting. But only time will tell, and the only way to see if this continues is to....well, to see if it continues.

 

[Nullity]

Yes, I agree, nothing to see still.

But then I didn't say I saw anything unusual either. I wanted to let Nullity and Colin know what their suggestions re: memory queries showed. Next time I'll just PM, so as not to waste your valuable time.... or mine.

I understand your frustration, but my criticism comes from years of my time being wasted on emotional reasonings for uselessly tweaking configs. If there is something to be fixed, fix it. Otherwise... leave it be, regardless of what you want to do. (A current mantra of mine.)


Forgetting the above, let me ask, if you were to discover that there was a memory leak, what would you do? I honestly would not know where to even begin solving the problem. Recently I have learned that from a admnistrative perspective, if there is no reason to upgrade, then don't. Software upgrades rarely bring performance increases (from my limited experience following the pinux kernel). Anecdotally, my PCs all have better wireless speeds with older (LTS) linux kernels.

Sorry for the rant-like post. I assume the other seemingly critical posts mirror my perspective.


PS - Regarding computer programming/systems Donald Knuth coined the phrase "premature optimization is the root of all evil".


Edit: "pinux" = "linux".

 

[jegesq]

Nullity,

No worries, and no "frustration" on my part. I hope my posts aren't too frustrating for you. Your message did seem to me be expressing a level of angst about what I posted, but that was certainly not my intent.

Honestly, to answer your question about "what would you do", the answer to that is "nothing"....I wouldn't even begin to know how to fix it if it was indeed a memory leak or some other related issue (unless it just involved changing some setting in the GUI). I realize Eric (RMerlin) is having some personal health issues at the moment, and that any further updates to his firmware are "on hold indefinitely" (at least according to a message he posted a day or two ago), but I'd have to leave any fixes to those who know better what they are doing in that area. I'm just a guy using the firmware provided by others, not a coder or tweaker.

What I would do if this was a confirmed problem would be to revert back to another version of Merlin's FW that hasn't exhibited the same phenomenon, a version that I've previously used that I know to be without any issues (e.g., 378.53, 54 or 55).

But to be clear, I have no emotion invested in my router or the firmware. It is, after all, just a router.

Oh, and btw, I enjoyed that link to Wikipedia. Thanks for that.

 

[L&LD]

Yes, I agree, nothing to see still.

But then I didn't say I saw anything unusual either (other than memory increasing in a seemingly stepwise fashion, about 5MB per day since I installed this version of the FW). I wanted to let Nullity and Colin know what their suggestions re: memory queries showed. Next time I'll just PM, so as not to waste your valuable time.... or mine.

No need to PM, we're here to share our knowledge.

No time wasted if someone learns something.

 

[jegesq]

http://www.snbforums.com/threads/memory-buffering-optimization-for-arm-devices.27953/#post-214798

Didn't mean to overlook what you posted, but I'm not running Samba, nor do I have any USB devices attached to the AC3200, so I am not sure that would do anything here.

 

[octopus]

Didn't mean to overlook what you posted, but I'm not running Samba, nor do I have any USB devices attached to the AC3200, so I am not sure that would do anything here.

Reverted the memory buffering optimization
for ARM devices, as people keep panicking
over the lower amount of free RAM. You can
manually re-enable the optimization by setting

nvram set drop_caches=0
nvram commit
reboot

 

[kvic]

I'd suggest that you also do a "cat /proc/slabinfo" and compare the outputs a day apart.

It might also be worth running "top" and hitting "s" twice to see what it thinks is the most memory hungry process.

Any snapshots for today..? Please share here

 

[jegesq]

Sorry, didn't have time yesterday to post. Good news though. RAM usage appears to be stable for the past two days, hovering around 90MB or so. Never climbed back up over 100, and in fact yesterday was reporting in the mid-80's.

So the kernel is doing its thing, and it appears no mem leak issues with this version of firmware.

Here's the latest cat /proc/meminfo:

admin@RT-AC3200-2010:/tmp/home/root# cat /proc/meminfo
MemTotal: 255572 kB
MemFree: 162620 kB
Buffers: 1452 kB
Cached: 6272 kB
SwapCached: 0 kB
Active: 20036 kB
Inactive: 4840 kB
Active(anon): 17504 kB
Inactive(anon): 644 kB
Active(file): 2532 kB
Inactive(file): 4196 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 0 kB
Writeback: 0 kB
AnonPages: 17152 kB
Mapped: 3196 kB
Shmem: 996 kB
Slab: 55000 kB
SReclaimable: 1976 kB
SUnreclaim: 53024 kB
KernelStack: 520 kB
PageTables: 596 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 127784 kB
Committed_AS: 30520 kB
VmallocTotal: 516096 kB
VmallocUsed: 30468 kB
VmallocChunk: 405372 kB

So nothing appears at all unusual as far as I can see.

I do note that the System Log is filled with repetitive dnsmasq-dhcp query calls (looking to confirm network devices?), and perhaps the log entries and querying may account for what seems to be higher memory usage with this firmware version than with other prior versions of firmware (where I did not previously see such querying activity). Here's a sample from the System Log of what the log is filled with:

Jan 21 06:58:50 dnsmasq-dhcp[2689]: DHCPSOLICIT(br0) 00:01:00:01:1d:4b:5e:be:00:1a:4d:93:ef:c8
Jan 21 06:59:05 dnsmasq-dhcp[2689]: DHCPSOLICIT(br0) 00:03:00:01:d4:85:64:34:01:5d
Jan 21 07:01:02 dnsmasq-dhcp[2689]: DHCPSOLICIT(br0) 00:01:00:01:1a:93:28:95:00:14:d1:2b:df:db
Jan 21 07:01:03 dnsmasq-dhcp[2689]: DHCPSOLICIT(br0) 00:01:00:01:1a:93:28:95:00:14:d1:2b:df:db
Jan 21 07:01:04 dnsmasq-dhcp[2689]: DHCPSOLICIT(br0) 00:01:00:01:1a:93:28:95:00:14:d1:2b:df:db
Jan 21 07:01:05 dnsmasq-dhcp[2689]: DHCPSOLICIT(br0) 00:03:00:01:d4:85:64:34:01:5d
Jan 21 07:01:06 dnsmasq-dhcp[2689]: DHCPSOLICIT(br0) 00:01:00:01:1a:93:28:95:00:14:d1:2b:df:db
Jan 21 07:01:10 dnsmasq-dhcp[2689]: DHCPSOLICIT(br0) 00:01:00:01:1a:93:28:95:00:14:d1:2b:df:db
Jan 21 07:01:19 dnsmasq-dhcp[2689]: DHCPSOLICIT(br0) 00:01:00:01:1a:93:28:95:00:14:d1:2b:df:db
Jan 21 07:01:35 dnsmasq-dhcp[2689]: DHCPSOLICIT(br0) 00:01:00:01:1a:93:28:95:00:14:d1:2b:df:db
Jan 21 07:02:06 dnsmasq-dhcp[2689]: DHCPSOLICIT(br0) 00:01:00:01:1a:93:28:95:00:14:d1:2b:df:db
Jan 21 07:03:05 dnsmasq-dhcp[2689]: DHCPSOLICIT(br0) 00:03:00:01:d4:85:64:34:01:5d
Jan 21 07:05:05 dnsmasq-dhcp[2689]: DHCPSOLICIT(br0) 00:03:00:01:d4:85:64:34:01:5d
Jan 21 07:07:05 dnsmasq-dhcp[2689]: DHCPSOLICIT(br0) 00:03:00:01:d4:85:64:34:01:5d
Jan 21 07:08:10 dnsmasq-dhcp[2689]: DHCPSOLICIT(br0) 00:01:00:01:1a:93:28:95:00:14:d1:2b:df:db
Jan 21 07:08:11 dnsmasq-dhcp[2689]: DHCPSOLICIT(br0) 00:01:00:01:1a:93:28:95:00:14:d1:2b:df:db
Jan 21 07:08:12 dnsmasq-dhcp[2689]: DHCPSOLICIT(br0) 00:01:00:01:1a:93:28:95:00:14:d1:2b:df:db

And yes, I have IPv6 enabled, but have not observed the issue noted by Readon in this thread (http://www.snbforums.com/threads/380-57-ipv6-misconfigures-dnsmasq.29961/), i.e., no loss of IPv6 connectivity at all. I assume if I wasn't using IPv6, the router would be making the same queries but using IPv4 addresses to search for network devices. I do wonder though why the router is constantly polling for certain devices only and not others (typically those that are turned on and connected, but not being actively used at the time the DHCPSolicit is sent). I do have 23 devices currently on the network, most connected either directly via wifi, or connected to two AC66U's, one acting as a Media Bridge, and the other as a Repeater, so perhaps that also accounts for the querying, i.e., to keep the Network Client list up-to-date.

Other than flooding the System Log with these DHCPsolicit queries there doesn't seem to be any adverse impact on my network. I may consider changing the level of events that are reported in the log, just to see if this also has an effect on memory usage, but for now, I'm not going to touch a thing since my network seems to be functioning great with the settings as they are.

So other than observing greater memory usage, in all others respects with regard to network device connections (both wired and wifi on both 2.4ghz and 5ghz) and internet speed, quality and connectivity, 380.57 appears to be otherwise rock-solid. Connections at the same distance appear to be slightly stronger than they were under versions 378.53, 34 and 55, which as I understand it is to be expected with newer drivers.

So those who said "nothing to see here" were right. Yes, you all told me so, and I thank you for that.

 

[L&LD]

Sorry, didn't have time yesterday to post. Good news though. RAM usage appears to be stable for the past two days, hovering around 90MB or so. Never climbed back up over 100, and in fact yesterday was reporting in the mid-80's.

So the kernel is doing its thing, and it appears no mem leak issues with this version of firmware.

Here's the latest cat /proc/meminfo:

admin@RT-AC3200-2010:/tmp/home/root# cat /proc/meminfo
MemTotal: 255572 kB
MemFree: 162620 kB
Buffers: 1452 kB
Cached: 6272 kB
SwapCached: 0 kB
Active: 20036 kB
Inactive: 4840 kB
Active(anon): 17504 kB
Inactive(anon): 644 kB
Active(file): 2532 kB
Inactive(file): 4196 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 0 kB
Writeback: 0 kB
AnonPages: 17152 kB
Mapped: 3196 kB
Shmem: 996 kB
Slab: 55000 kB
SReclaimable: 1976 kB
SUnreclaim: 53024 kB
KernelStack: 520 kB
PageTables: 596 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 127784 kB
Committed_AS: 30520 kB
VmallocTotal: 516096 kB
VmallocUsed: 30468 kB
VmallocChunk: 405372 kB

That's a fair amount of cached data, but must be normal with the number of client devices showing.

I do note that the System Log is filled with repetitive dnsmasq-dhcp query calls (looking to confirm network devices?), and perhaps the log entries and querying may account for what seems to be higher memory usage with this firmware version than with other prior versions of firmware (where I did not previously see such querying activity). Here's a sample from the System Log of what the log is filled with:

Jan 21 06:58:50 dnsmasq-dhcp[2689]: DHCPSOLICIT(br0) 00:01:00:01:1d:4b:5e:be:00:1a:4d:93:ef:c8
Jan 21 06:59:05 dnsmasq-dhcp[2689]: DHCPSOLICIT(br0) 00:03:00:01:d4:85:64:34:01:5d
Jan 21 07:01:02 dnsmasq-dhcp[2689]: DHCPSOLICIT(br0) 00:01:00:01:1a:93:28:95:00:14:d1:2b:df:db
Jan 21 07:01:03 dnsmasq-dhcp[2689]: DHCPSOLICIT(br0) 00:01:00:01:1a:93:28:95:00:14:d1:2b:df:db
Jan 21 07:01:04 dnsmasq-dhcp[2689]: DHCPSOLICIT(br0) 00:01:00:01:1a:93:28:95:00:14:d1:2b:df:db
Jan 21 07:01:05 dnsmasq-dhcp[2689]: DHCPSOLICIT(br0) 00:03:00:01:d4:85:64:34:01:5d
Jan 21 07:01:06 dnsmasq-dhcp[2689]: DHCPSOLICIT(br0) 00:01:00:01:1a:93:28:95:00:14:d1:2b:df:db
Jan 21 07:01:10 dnsmasq-dhcp[2689]: DHCPSOLICIT(br0) 00:01:00:01:1a:93:28:95:00:14:d1:2b:df:db
Jan 21 07:01:19 dnsmasq-dhcp[2689]: DHCPSOLICIT(br0) 00:01:00:01:1a:93:28:95:00:14:d1:2b:df:db
Jan 21 07:01:35 dnsmasq-dhcp[2689]: DHCPSOLICIT(br0) 00:01:00:01:1a:93:28:95:00:14:d1:2b:df:db
Jan 21 07:02:06 dnsmasq-dhcp[2689]: DHCPSOLICIT(br0) 00:01:00:01:1a:93:28:95:00:14:d1:2b:df:db
Jan 21 07:03:05 dnsmasq-dhcp[2689]: DHCPSOLICIT(br0) 00:03:00:01:d4:85:64:34:01:5d
Jan 21 07:05:05 dnsmasq-dhcp[2689]: DHCPSOLICIT(br0) 00:03:00:01:d4:85:64:34:01:5d
Jan 21 07:07:05 dnsmasq-dhcp[2689]: DHCPSOLICIT(br0) 00:03:00:01:d4:85:64:34:01:5d
Jan 21 07:08:10 dnsmasq-dhcp[2689]: DHCPSOLICIT(br0) 00:01:00:01:1a:93:28:95:00:14:d1:2b:df:db
Jan 21 07:08:11 dnsmasq-dhcp[2689]: DHCPSOLICIT(br0) 00:01:00:01:1a:93:28:95:00:14:d1:2b:df:db
Jan 21 07:08:12 dnsmasq-dhcp[2689]: DHCPSOLICIT(br0) 00:01:00:01:1a:93:28:95:00:14:d1:2b:df:db

And yes, I have IPv6 enabled, but have not observed the issue noted by Readon in this thread, and assume if it wasn't the same calls would be made using IPv4 network device addresses. I do wonder though why the router is constantly polling for certain devices (typically those that are on and connected, but otherwise unused at the time the DHCPSolicit goes out).

Other than flooding the System Log with these DHCPsolicit queries there doesn't seem to be any adverse impact on my network. I may consider changing the level of events that are reported in the log, just to see if this also has an effect on memory usage, but for now, I'm not going to touch a thing since my network seems to be functioning great with the settings as they are.

So other than observing greater memory usage, in all others respects with regard to network device connections (both wired and wifi on both 2.4ghz and 5ghz) and internet speed, quality and connectivity, 380.57 appears to be otherwise rock-solid. Connections at the same distance appear to be slightly stronger than they were under versions 378.53, 34 and 55, which as I understand it is to be expected with newer drivers.

So those who said "nothing to see here" were right. Yes, you all told me so, and I thank you for that.

Still, glad to see that it really was nothing and your network is stable.

 

[kvic]

Sorry, didn't have time yesterday to post. Good news though. RAM usage appears to be stable for the past two days, hovering around 90MB or so. Never climbed back up over 100, and in fact yesterday was reporting in the mid-80's.

So the kernel is doing its thing, and it appears no mem leak issues with this version of firmware.

That's quite an anti-climax! But very glad to hear your router is running well.

Looking at your snapshots, mem alloc is rock solid. Some of us were thinking a user app or two could be leaking memory. Apparently not so.

I would also invite people, especially opinion leaders on this forum not to pose strong presumptions when troubleshooting issues. There are many ways to leak memory...not saying merlin's firmware isn't rock solid.

 

[L&LD]

That's quite an anti-climax! But very glad to hear your router is running well.

Looking at your snapshots, mem alloc is rock solid. Some of us were thinking a user app or two could be leaking memory. Apparently not so.

I would also invite people, especially opinion leaders on this forum not to pose strong presumptions when troubleshooting issues. There are many ways to leak memory...not saying merlin's firmware isn't rock solid.

No presumptions were necessary. I have already seen this with a few threads already.

Also, the presumption that it was a memory leak was just as strong too.

 

[GreGHM]

Why do you think the memory usage is not normal? I mean, you're using the router and it is serving it's tasks. Like with a car driving means it uses petrol. So doing it's job for a router means using the cpu and storing data in memory, both active RAM, temporary data and cached items and data.

Not correct comparison with a car.
If we are talking about routers (cars) - my old Asus RT N56U with FW from Padavan was never increasing RAM usage running 24x7. And if we are talking about Linux based systems - my Qnap TS-251 has also Linux based operation system, and it is not increasing consumption of RAM same running 24x7.
So we are talking about specific firmware for specific Broadcom hardware.
btw, my AC68U is running 380-56_2 and has same RAM usage increasing from 49-50 MB after reboot and untill 80-100 MB in few days. I am just rebooting it once per week or two and that's all.
I already got an explanations from forum members, same as here, that it is normal for this router.
But when you have a possibility to compare with other devices - then you will always have this question...

With RT N56U i was never worrying about how it is running - uptime was 2-3-4 month.
With RT AC-68U i do not have this feeling - it should be rebooted from time to time ))).

 

[MarkyPancake]

Now this thread has got me checking my AC68U's memory usage.

 

[kvic]

Just if someone is interested in hearing..

For my personal use, I recompiled 378.55 with this toolchain.

On startup, it uses about 43MiB RAM. At steady idle state, it uses ~50MiB. Currently registers 51MiB after running for 23 days and 1,090,136 jobs/processes. I've never seen gradual increase to 100MiB and top off..

I shall clarify...I kill off quite a few processes launched by default in ASUSWRT. I also don't use ASUS applications nor TrendMicro. But I run additional applications from Entware. My current count of processes stay at 37.

 

[jegesq]

A small update.....as noted above, used RAM hit about 100 MB and then gradually backed down to the 80-90MB range and stayed there for several days. But in going through my settings one more time last night, I discovered that when I upgraded the firmware and then did a factory reset, I had inadvertently missed turning off WPS (a discovery that I made through testing the Asus Router App for Android, which, when doing a "Security Scan" will reports specifically whether WPS is on or off). So I turned WPS off, and because everything else had been stable (in terms of connectivity, bandwidth, speed, etc) both internally in my network and vis-a-vis internet, I decided to reboot the router (first time since doing the upgrade) just to see if a reboot would have any impact one way or the other on used memory.

So I did a reboot of my AC3200 last night. When it came back up, memory usage had immediately dropped to 32MB, which is a level that was entirely consistent with what I'd been used to seeking with literally every other version of Merlin FW that I'd used over the course of the past two years (first year with AC66U, and since last year, with my AC3200). I checked used memory again this morning (about 12 hours later), and there had been no change at all, and it was still at 32MB (or about 13% of total RAM). Other than turning off WPS, no other changes to settings were made, and I don't think there were any other services that were terminated or started (i.e., I am still not running any servers, no FTP, no SAMBA, no VPN, no AI Protect, no Trend Micro, no QoS, no Smart Connect, etc.). Just a bare-bones tri-band router with three separate SSID's, and the same number of client devices (i.e., about 23).

I will again watch and wait to see if memory usage again increases to the 90-100MB range previously observed. Whether it does or not doesn't seem to be an issue though that I need be concerned about, since everything on my network seems to be functioning just fine. Thanks again for some of the insights above into memory allocation, usage, etc. I learned a couple of things and hopefully the discussion may have been helpful to others as well.

 

[MarkyPancake]

I'm currently on 378.56_2 and observed yesterday after work that the RAM used was at 60 MB, then by bedtime it was 67 MB and right now it's up to 75 MB.