AdGuardHome AdGuardHome Memory Usage Analysis

[ERamseth2]

Analyzing memory usage of AGH to determine if there is a memory leak.

AGH parameters:

• using oisd dbl blocklist
• mmotti filters imported into custom filters area (I see no need to have these pulled in regularly from github)
• a few extra filter entries to allow a few apps to work
• Upstream DoT and DoQ servers
• I think everything else is basically stock
• Service restarted 7 minutes before first data point in this set

System:

• Asus RT-AX86U
• Running Asuswrt-merlin 386.5_2
• Skynet, FlexQoS, YazFi, scribe, scMerlin, uiScribe, YazDHCP, vnStat, AdGuardHome, entware installed
• 32GB USB storage
• 2GB swap
• swappiness set to 45 (echo 45 > /proc/sys/vm/swappiness)
• UDP receive buffer set to 2.5M (echo 2500000 > /proc/sys/net/core/rmem_max)

Here is a 24 hour chart of memory usage of AdGuardHome

(Note: I opened the web interface for maybe 20 minutes around where the red circle is. I did not duplicate the red circle on the orange data line, but it still applies)

Raw data is attached in a text file (comma separated)

Thoughts/notes:

• Going to use this as the baseline to compare changes against
• Odd to me that the memory usage jumps, then never comes back down. No/bad memory cleanup?
• Need to track down why the big jump. Doesn't seem connected to usage. Nothing in syslog. Maybe filter refreshes? Can check that by increasing filter refresh frequency.
• I guess I'll need to open the web interface during each 24-hour run and document when that happened.
• "Opened web interface" I think corresponds to the really small bump inside the red oval, not the larger jump that comes later

 

[ERamseth2]

Next up: using "--no-mem-optimization" as recommended by @SomeWhereOverTheRainBow

 

[SomeWhereOverTheRainBow]

Analyzing memory usage of AGH to determine if there is a memory leak.

AGH parameters:

• using oisd dbl blocklist
• mmotti filters imported into custom filters area (I see no need to have these pulled in regularly from github)
• a few extra filter entries to allow a few apps to work
• Upstream DoT and DoQ servers
• I think everything else is basically stock
• Service restarted 7 minutes before first data point in this set

System:

• Asus RT-AX86U
• Running Asuswrt-merlin 386.5_2
• Skynet, FlexQoS, YazFi, scribe, scMerlin, uiScribe, YazDHCP, vnStat, AdGuardHome, entware installed
• 32GB USB storage
• 2GB swap
• swappiness set to 45 (echo 45 > /proc/sys/vm/swappiness)
• UDP receive buffer set to 2.5M (echo 2500000 > /proc/sys/net/core/rmem_max)

Here is a 24 hour chart of memory usage of AdGuardHome

(Note: I opened the web interface for maybe 20 minutes around where the red circle is. I did not duplicate the red circle on the orange data line, but it still applies)

View attachment 40684

Raw data is attached in a text file (comma separated)

Thoughts/notes:

• Going to use this as the baseline to compare changes against
• Odd to me that the memory usage jumps, then never comes back down. No/bad memory cleanup?
• Need to track down why the big jump. Doesn't seem connected to usage. Nothing in syslog. Maybe filter refreshes? Can check that by increasing filter refresh frequency.
• I guess I'll need to open the web interface during each 24-hour run and document when that happened.

The big jump might be when the filter list update happens. I notice a huge increase in memory when ever one loads a new list which leads me to think it is whenever the 24 hour list update happens. To test this theory, maybe change the list update interval to 3 days. See if after 3 days, the memory usage goes up, or if it still happens after 24 hours. If this is the list update, then it sounds like they have a bad memory release garbage cleanup policy upon list updates and loading.

My theory is that whenever it updates the list, that it makes a huge jump in memory usage upon download, and another jump once it loads the new entries. From there, the memory usage doesn't get freed. Keep in mind this is all conjecture, but the only thing that actually happens 24 hours after starting adguardhome is the list update process.

 

[ERamseth2]

The big jump might be when the filter list update happens. I notice a huge increase in memory when ever one loads a new list which leads me to think it is whenever the 24 hour list update happens. To test this theory, maybe change the list update interval to 3 days. See if after 3 days, the memory usage goes up, or if it still happens after 24 hours. If this is the list update, then it sounds like they have a bad memory release garbage cleanup policy upon list updates and loading.

My theory is that whenever it updates the list, that it makes a huge jump in memory usage upon download, and another jump once it loads the new entries. From there, the memory usage doesn't get freed. Keep in mind this is all conjecture, but the only thing that actually happens 24 hours after starting adguardhome is the list update process.

I had a similar thought. After getting data for --no-mem-optimization I will start trying to get data for filter list reloads by tweaking the reload interval.

 

[ERamseth2]

Here is a similar graph using --no-mem-optimization

Some notes:

• "Opened web interface" I think corresponds to the really small bump inside the red oval, not the larger jump that comes later
• "checkpoints" where the mem usage plateaus are about the same, but come at different times

 

[ERamseth2]

@SomeWhereOverTheRainBow See above. Not sure --no-mem-optimization is helping (at least on a 24hr view)

 

[gspannu]

I think @SomeWhereOverTheRainBow is probably correct that it is the downloading of updates and applying the updates that is not releasing memory.
Lets see if the AGH devs on GitHub get this fixed…

 

[SomeWhereOverTheRainBow]

I think @SomeWhereOverTheRainBow is probably correct that it is the downloading of updates and applying the updates that is not releasing memory.
Lets see if the AGH devs on GitHub get this fixed…

I agree, I feel like we could supply them with graphs and charts for days, but it is really up to them to investigate in their own code to find where the issue is.

 

[SomeWhereOverTheRainBow]

@SomeWhereOverTheRainBow See above. Not sure --no-mem-optimization is helping (at least on a 24hr view)

I think the only thing it shows is that at some point there is a consistency again in the amount of memory usage for a period of time before the new entries get loaded, the memory optimization built in seemed to hinder us seeing that process alittle. By using the --no-mem-optimization flag, you get a more accurate understanding of what is going on with the memory.

See here W/o --no-mem-optimization

see here with --no-mem-optimization

At least now we get a somewhat better understanding of what we are seeing.

 

[SomeWhereOverTheRainBow]

Analyzing memory usage of AGH to determine if there is a memory leak.

AGH parameters:

• using oisd dbl blocklist
• mmotti filters imported into custom filters area (I see no need to have these pulled in regularly from github)
• a few extra filter entries to allow a few apps to work
• Upstream DoT and DoQ servers
• I think everything else is basically stock
• Service restarted 7 minutes before first data point in this set

System:

• Asus RT-AX86U
• Running Asuswrt-merlin 386.5_2
• Skynet, FlexQoS, YazFi, scribe, scMerlin, uiScribe, YazDHCP, vnStat, AdGuardHome, entware installed
• 32GB USB storage
• 2GB swap
• swappiness set to 45 (echo 45 > /proc/sys/vm/swappiness)
• UDP receive buffer set to 2.5M (echo 2500000 > /proc/sys/net/core/rmem_max)

Here is a 24 hour chart of memory usage of AdGuardHome

(Note: I opened the web interface for maybe 20 minutes around where the red circle is. I did not duplicate the red circle on the orange data line, but it still applies)

View attachment 40684

Raw data is attached in a text file (comma separated)

Thoughts/notes:

• Going to use this as the baseline to compare changes against
• Odd to me that the memory usage jumps, then never comes back down. No/bad memory cleanup?
• Need to track down why the big jump. Doesn't seem connected to usage. Nothing in syslog. Maybe filter refreshes? Can check that by increasing filter refresh frequency.
• I guess I'll need to open the web interface during each 24-hour run and document when that happened.
• "Opened web interface" I think corresponds to the really small bump inside the red oval, not the larger jump that comes later

@ERamseth2 wouldn't using

swappiness set to 45 (echo 45 > /proc/sys/vm/swappiness)

tell your router to use your swap less than the router default of 60? Which means your router would move less of the stale process's that it deemed swappable to swap in preference of leaving it stored in the ram. (a.k.a. your router would use more ram than intended under normal use than if you were to leave it at the default 60.)

So would setting your swappiness to 45 assure that enough stale process's have cleared out of regular ram so there is enough ram necessary to accommodate the regularly active use of AdGuardHome?

 

[ERamseth2]

@ERamseth2 wouldn't using

swappiness set to 45 (echo 45 > /proc/sys/vm/swappiness)

tell your router to use your swap less than the router default of 60? Which means your router would move less of the stale process's that it deemed swappable to swap in preference of leaving it stored in the ram. (a.k.a. your router would use more ram than intended under normal use than if you were to leave it at the default 60.)

View attachment 40773

So would setting your swappiness to 45 assure that enough stale process's have cleared out of regular ram so there is enough ram necessary to accommodate the regularly active use of AdGuardHome?

When I checked the swappiness it was defaulted to zero which I believe means use swap as an absolute last resort when we're about to completely run out of memory.

Maybe this makes sense with swap on something like a usb stick or flash memory where in theory there are limited writes. for what its worth I had a full default ubuntu desktop (installed, not live) running on the same brand of usb flash memory I am using in my router now (Samsung FIT USB or something... the little tiny samsung one) for 2 years, swap and default swap settings included, and had no problems whatsoever.

If you have a swap set up through amtm maybe you can check it too.

 

[SomeWhereOverTheRainBow]

When I checked the swappiness it was defaulted to zero which I believe means use swap as an absolute last resort when we're about to completely run out of memory.

Maybe this makes sense with swap on something like a usb stick or flash memory where in theory there are limited writes. for what its worth I had a full default ubuntu desktop (installed, not live) running on the same brand of usb flash memory I am using in my router now (Samsung FIT USB or something... the little tiny samsung one) for 2 years, swap and default swap settings included, and had no problems whatsoever.

If you have a swap set up through amtm maybe you can check it too.

There may be an issue with your kernel settings ( i.e. how they are setup at build time, obviously not your fault though), but my router (the RT-AX88U) appears to default to the standard linux default for swappiness, which is 60. I have no clue why yours is zero (without your intervention), unless that: it is the new asuswrt standard, it is not correctly set for the firmware build time for your model router maybe (@RMerlin can check), or somehow you have a script running that defaults it to zero. But setting it to zero would mean it is pointless to even have a swap.

this is without my own intervention

RT-AX88U-C7C0:/tmp/home/root# cat /proc/sys/vm/swappiness
60

At either rate, having this set to a really low number is a rather bad idea if you are expecting your swap to at somepoint alleviate you ram resources of stale pages which would hopefully free up extra ram resources for your thriving AdGuardHome and other active resources.

 

[ERamseth2]

Yeah I mean I didnt think 45 vs. 60 was terribly low. I guess I can change back to 60. Probably won't hurt anything.

 

[SomeWhereOverTheRainBow]

Yeah I mean I didnt think 45 vs. 60 was terribly low. I guess I can change back to 60. Probably won't hurt anything.

I am curious to see what your ram usage looks like with adguardhome after two to three days of running with not restart, obviously if you have to, you can pop in and check what the logging looks like between each day. The reason why I bring this up is because some users had reported adguardhome needed to be restarted after a couple of days of straight running. I am wondering if it is linked to this memory chokehold. I have yet to experience this on my RTAX88U, but it could be only happening on certain models.

 

[ERamseth2]

I have some logs to analyze for the 1hr filter refresh with non mem optimization. Just havn't gotten to it yet. After that I will extend my logging and restart period to 2 days and see what happens.

 

[ERamseth2]

This is with --no-mem-optimization and hourly filter updates

The big jumps in memory usage match up with the timing of SOME filter updates in the log, but not all filter updates logged have a jump associated with it.

Oddly enough, the log (talking about messages.log here) doesn't show hourly updates. Maybe it only logs when something changes and doesn't log when the pulled file is the same?

 

[SomeWhereOverTheRainBow]

This is with --no-mem-optimization and hourly filter updates

View attachment 40820

Well at least there is some indication of a consistent behavior, the real question is, does some of the memory usage drop after a certain point , or does it continue to rise. I think a two or a three day run, should show us more about what happens in the longer run.

 

[ERamseth2]

Yep, I agree. Setting that up now. Might have some unhappy folks around if the internet goes out.

 

[null_]

Yep, I agree. Setting that up now. Might have some unhappy folks around if the internet goes out.

Did you arrive at any conclusions with this? Reason I ask is that I'm seeing similar behaviour on a GT-AX6000. I'm not sure what my "swappiness" is set to (as the file shows blank in WinSCP and I have zero confidence in my CLI skills to access that info in another way) but if there's any insight into how to avoid AGH from consuming all of the available system memory I'd be glad to hear it. Thanks!

 

[SomeWhereOverTheRainBow]

Did you arrive at any conclusions with this? Reason I ask is that I'm seeing similar behaviour on a GT-AX6000. I'm not sure what my "swappiness" is set to (as the file shows blank in WinSCP and I have zero confidence in my CLI skills to access that info in another way) but if there's any insight into how to avoid AGH from consuming all of the available system memory I'd be glad to hear it. Thanks!

Have a good swap. that should be it. adguardhome should relinquish memory and use it as it needs it, but on the routers it operates in a tight amount of space. Having a good swap will atleast cushin that space.