[Bug] GT-AX6000 - swappiness=0 causing issues when swap is needed

[SomeWhereOverTheRainBow]

No, I meant Linux as that's what we're talking about with this router. I've spent a lot of time in my career fine tuning virtual memory and swap on Solaris, Unixware, AIX, etc. (even IBM mainframes) and they each have their own subtle differences. But regardless of that the OP's router shouldn't be failing to swap when required.

But doesn't setting swappiness to zero means the OS ignores the kernel algorithm for swapping? Wouldn't that effectively mean that swap never happens unless the kernel parameters are told not to use the algorithm?

 

[ColinTaylor]

But doesn't setting swappiness to zero means the OS ignores the kernel algorithm for swapping?

I'm reluctant to make generalisations as the mechanism has changed over time (I believe it changed again in Linux 5.8, which is why I asked about the kernel version) - and I'm certainly not an expert in the minute detail of all (or any) of those possibilities. It's even different between the same version of Linux but running on different architectures (e.g. ARM vs. x86).

Wouldn't that effectively mean that swap never happens unless the kernel parameters are told not to use the algorithm?

I can't parse that question as it's a triple negative... But as I understand it (and from my own testing) setting swappiness to zero means that it will always choose to drop pages from the system page cache instead of swapping out runtime memory. However, that doesn't mean that swapping won't occur. It simply means that it will instead be triggered by hitting the high water mark as described in post #11. Maybe something has changed in kernel 4.19.183 and swappiness=0 now means something different.

 

[SomeWhereOverTheRainBow]

I'm reluctant to make generalisations as the mechanism has changed over time (I believe it changed again in Linux 5.8, which is why I asked about the kernel version) - and I'm certainly not an expert in the minute detail of all (or any) of those possibilities. It's even different between the same version of Linux but running on different architectures (e.g. ARM vs. x86).


I can't parse that question as it's a triple negative... But as I understand it (and from my own testing) setting swappiness to zero means that it will always choose to drop pages from the system page cache instead of swapping out runtime memory. However, that doesn't mean that swapping won't occur. It simply means that it will instead be triggered by hitting the high water mark as described in post #11. Maybe something has changed in kernel 4.19.183 and swappiness=0 now means something different.

Sorry for the "triple-negative", for clarification I thought “1” is the minimum possible “active swapping” setting while “0” means disable swapping completely and only revert to when RAM is completely filled. That would be counter productive to having a swap, if the school of thought for having such is to avoid an out of memory scenario.

 

[ColinTaylor]

Sorry for the "triple-negative", for clarification I thought “1” is the minimum possible “active swapping” setting while “0” means disable swapping completely and only revert to when RAM is completely filled.

I think we have to be careful with the phrasing here. I've frequently read that "0 disables swapping", but that's clearly not the case - or at least it doesn't mean it never swaps.

From https://docs.kernel.org/admin-guide/sysctl/vm.html#swappiness

At 0, the kernel will not initiate swap until the amount of free and file-backed pages is less than the high watermark in a zone.

 

[SomeWhereOverTheRainBow]

I think we have to be careful with the phrasing here. I've frequently read that "0 disables swapping", but that's clearly not the case - or at least it doesn't mean it never swaps.

From https://docs.kernel.org/admin-guide/sysctl/vm.html#swappiness

That is more for the reason to ask for clarification. It is onething to mention high-water mark in a zone, but then you have to ask yourself what defines this zone. For example if you are already at an out of memory situation when this high water mark zone is surpassed, then it might be prudent to consider using a swappiness higher than zero.

 

[ColinTaylor]

That is more for the reason to ask for clarification. It is onething to mention high-water mark in a zone, but then you have to ask yourself what defines this zone.

For the router it's simple, there's only one zone - DMA. But perhaps you meant "what defines the thresholds set within a zone". That would be the kernel developers AFAIK.

For example if you are already at an out of memory situation when this high water mark zone is surpassed, then it might be prudent to consider using a swappiness higher than zero.

You can't already be out of memory when you reach the high water mark because the high water mark is set (in our case) 7680 x 4k (30MB) before the end of memory. But yes, you could try to preempt hitting the high water mark by setting swappiness to 1, 40, 60, etc. That would work very effectively if you have lots of memory and lots of inactive anonymous pages that aren't being used (like a desktop PC).

 

[SomeWhereOverTheRainBow]

That would work very effectively if you have lots of memory and lots of inactive anonymous pages that aren't being used (like a desktop PC).

Or perhaps when applications don't always play nicely with available RAM.

 

[Martinski]

Maybe something has changed in kernel 4.19.183 and swappiness=0 now means something different.

Yes, that's a possibility, and if so the semantics & behavior would not match any previously established understanding of how the swap mechanism is supposed to work under such conditions.

 

[Martinski]

That is more for the reason to ask for clarification. It is onething to mention high-water mark in a zone, but then you have to ask yourself what defines this zone.

The OS doesn't view physical RAM as one huge pool of homogeneous memory. RAM is actually divided into distinct regions called "zones" and the number of zones varies depending on the system architecture (32-bit vs 64-bit) and the total RAM size.

To see the memory zone(s) and the node attached to each, type the following command:

cat /proc/buddyinfo

As @ColinTaylor correctly stated, in ASUS routers there is only one memory zone called DMA (Direct Memory Access) which is attached to Node 0 which, in turn, is associated with the existing CPU cores. The top line of /proc/zoneinfo shows the Node & Zone association as well.

Here is a screenshot from the RT-AC86U router:

 

[Martinski]

Or perhaps when applications don't always play nicely with available RAM.

A classic example would be a memory leak from a program that fails to free up some runtime memory allocations upon exiting or termination. Over time (hours or days, depending on the size of the leak or the frequency of code execution) such memory leaks would slowly consume RAM, leading to a gradual performance degradation where even writing the inactive anonymous memory pages to swap might not be enough to prevent an eventual system freeze or failure.

 

[Martinski]

The OS doesn't view physical RAM as one huge pool of homogeneous memory. RAM is actually divided into distinct regions called "zones" and the number of zones varies depending on the system architecture (32-bit vs 64-bit) and the total RAM size.

To see the memory zone(s) and the node attached to each, type the following command:

cat /proc/buddyinfo

Here is a screenshot from my VM running Ubuntu that shows more memory zones:

Just FYI.

 

[Weblee2407]

I'm just going to tag onto this discussion as it is related - but if a new thread is needed, I know someone will let me know.

On my very first Merlin install (386.7), I manually set the swappiness value to 72 in the "/proc/sys/vm/swappiness" file. One afternoon, lo and behold, the free -w command indicated swapping was active! My scheduled reboot fired off the next morning and memory usage fell below 72 and the swappiness file was back to 0. After resetting swappiness to 72 and memory usage going all the way to 83 - no swapping. This is with both 386.7 and 386.7_2.

So I have three questions:

1 - is there any way, simply for testing, to get swapping to start just to make sure it works and see what pops up in the logs etc.?

2 - Is there an add-on that is so hungry it makes swapping mandatory?

3 - how do you edit system-config.sh

 

[Martinski]

After resetting swappiness to 72 and memory usage going all the way to 83 - no swapping.

This is normal & as expected. In ASUS routers with 1GB RAM, swapping will not be usually triggered until RAM utilization is roughly ~97% (~95% in routers with 512MB RAM). This is due to the value set for the "low water mark" threshold (6400 pages = 25MB) in the DMA memory zone. You can see the current threshold values (the "low" & "high" lines) with the following command:

head /proc/zoneinfo

Now, to be abundantly clear, note that the swappiness value itself does *not* set a specific threshold for RAM utilization so that swapping is triggered when the amount of used RAM hits that threshold. Instead, the swappiness value is used to set a relative ratio for the preference of the swapping algorithm to mark for reclamation & free up two types of memory pages: file-backed pages or anonymous pages (see post #14 in this same thread for definitions).

IOW, when RAM pages need to be freed, the swappiness value is taken into account by the memory reclamation & swap algorithms to determine which memory page types (file-backed or anonymous) are preferentially considered to be freed. See post #11 where I briefly described this, and the two threshold values used (low water mark & high water mark) to trigger the swapping mechanism.

1 - is there any way, simply for testing, to get swapping to start just to make sure it works and see what pops up in the logs etc.?

Off the top of my head, I don't know of a simple way to trigger swapping in ASUS routers. Many years ago (more than a decade by now) when I needed to test/verify a memory management & garbage collection algorithm in other Linux-based devices (not routers), I wrote a program in C++ that had a recursive function which dynamically allocated memory every 60 seconds in fixed chunks (1MB, 5MB, 10MB or 20MB, depending on RAM size & other factors) until RAM was very near exhaustion. In the meantime, other utility programs were gathering stats regarding RAM & swap usage. You could try setting *temporarily* the swappiness value to 100 to increase the chance of swapping, but you still need something else to gradually & dynamically consume RAM until almost close to full depletion.

2 - Is there an add-on that is so hungry it makes swapping mandatory?

The only add-on I have installed & currently use is YazFi. I've read about the Diversion & Transmission add-ons requiring at least a 1GB swap and actually using swap quite frequently. There are probably other add-ons that use swap as well, but I have no actual personal experience with any of those scripts to know for sure.

3 - how do you edit system-config.sh

If your goal is to simply change the swappiness value every time upon reboot, you can add the following command to the "/jffs/scripts/services-start" script (you can use the nano or vi text editor to do this).

echo 60 > /proc/sys/vm/swappiness

If you don't already have the "/jffs/scripts/services-start" script file, you can create it on an SSH terminal window using the following commands:

cat << "EOF" > /jffs/scripts/services-start
#!/bin/sh
echo 60 > /proc/sys/vm/swappiness
EOF
chmod a+x /jffs/scripts/services-start

NOTE:
The above assumes that you have the "Enable JFFS custom scripts and configs" option already enabled (i.e. set to "Yes" in the "Administration -> System" web GUI tab).

My 2 cents.

 

[Weblee2407]

Thanks for the detailed description of the swap mechanism.

yes I ended up doing an echo in post-mount right before the swap on command.

 

[bluepoint]

Off the top of my head, I don't know of a simple way to trigger swapping in ASUS routers. Many years ago (more than a decade by now) when I needed to test/verify a memory management & garbage collection algorithm in other Linux-based devices (not routers), I wrote a program in C++ that had a recursive function which dynamically allocated memory every 60 seconds in fixed chunks (1MB, 5MB, 10MB or 20MB, depending on RAM size & other factors) until RAM was very near exhaustion. In the meantime, other utility programs were gathering stats regarding RAM & swap usage. You could try setting *temporarily* the swappiness value to 100 to increase the chance of swapping, but you still need something else to gradually & dynamically consume RAM until almost close to full depletion.


The only add-on I have installed & currently use is YazFi. I've read about the Diversion & Transmission add-ons requiring at least a 1GB swap and actually using swap quite frequently. There are probably other add-ons that use swap as well, but I have no actual personal experience with any of those scripts to know for sure.

Here's an actual swapping first moment from my network to illustrate how it works @ swappiness=60. Didn't know what triggers it @95% as it reaches the threshold.

Other add ons that can be added that uses the swap files that I know of are Unbound and Skynet.

Note: Network is stable up to this point.

 

[Martinski]

Didn't know what triggers it @95% as it reaches the threshold.

I don't think the swap mechanism is being triggered at ~95% of RAM usage on your router. I'm assuming here that your snapshots were taken *after* some RAM pages were already written to swap and then freed. IOW, a higher threshold (probably closer to ~97% based on the "low water mark" value) was reached at some point before, and then after the RAM pages were freed, RAM utilization went down to 95%, which is when you captured the screenshots.

Other add ons that can be added that uses the swap files that I know of are Unbound and Skynet.

Note: Network is stable up to this point.

Do you mind telling us which specific add-ons you've installed on your router which is consuming RAM up to the threshold that triggers swapping?

In any event, thanks for the additional info.

 

[bluepoint]

I don't think the swap mechanism is being triggered at ~95% of RAM usage on your router. I'm assuming here that your snapshots were taken *after* some RAM pages were already written to swap and then freed. IOW, a higher threshold (probably closer to ~97% based on the "low water mark" value) was reached at some point before, and then after the RAM pages were freed, RAM utilization went down to 95%, which is when you captured the screenshots.


Do you mind telling us which specific add-ons you've installed on your router which is consuming RAM up to the threshold that triggers swapping?

In any event, thanks for the additional info.

That's correct the snapshots were taken after some pages were freed and I believe it did reach 97% when swapping happened. In fact, today this morning, there were more freed RAM space as now it's at 81% from 95% possibly because it needeed more space to accommodate Diversion's Large blocking list update @4 AM(diversion's drop caches is disabled) and it did use small amount of additional swapping file for the deed.

And to answer your request which add0ns I use now, I only have Diversion lite using Large blocking list. In the past I've used Unbound and Skynet that's why I know they add up to memory usage probably more than diversion.

 

[Martinski]

That's correct the snapshots were taken after some pages were freed and I believe it did reach 97% when swapping happened. In fact, today this morning, there were more freed RAM space as now it's at 81% from 95% possibly because it needeed more space to accommodate Diversion's Large blocking list update @4 AM(diversion's drop caches is disabled) and it did use small amount of additional swapping file for the deed.

OK, that's very good; the swap mechanism is working as intended. I assume that the list files used by Diversion are fairly big (multiple MBs?), and so while the add-on continues to do its work, eventually many of the inactive file-backed memory pages in RAM were marked for reclamation & then freed, which significantly increased the amount of free RAM as you noted (~150MB more, in this particular instance, based on your two sample snapshots taken hours apart).

And to answer your request which add0ns I use now, I only have Diversion lite using Large blocking list. In the past I've used Unbound and Skynet that's why I know they add up to memory usage probably more than diversion.

Thanks for the additional info & details.

 

[bluepoint]

OK, that's very good; the swap mechanism is working as intended. I assume that the list files used by Diversion are fairly big (multiple MBs?), and so while the add-on continues to do its work, eventually many of the inactive file-backed memory pages in RAM were marked for reclamation & then freed, which significantly increased the amount of free RAM as you noted (~150MB more, in this particular instance, based on your two sample snapshots taken hours apart).


Thanks for the additional info & details.

Yes, definitely the swapping mechanism works at least for RT-AX88U and most likely to all models. The blocking list is about 22 MB in size more or less of which there's about 1 million domains to block, updates are scheduled twice a week.

 

[Weblee2407]

Tested - Found True