[Bug] GT-AX6000 - swappiness=0 causing issues when swap is needed

[Maloon]

Hi,

GT-AX6000 has this script: /etc/init.d/system-config.sh, that sets /proc/sys/vm/swappiness to 0, making use of swap only when no RAM is available. Problem is that a value of 0 doesn't work and system doesn't use swap when there is not RAM available, causing several crashes.

Other routers keep this value in 60, and reverting this value to 60 makes all things to work right again, because swap is used when is needed.

Best regards.

 

[SomeWhereOverTheRainBow]

Hi,

GT-AX6000 has this script: /etc/init.d/system-config.sh, that sets /proc/sys/vm/swappiness to 0, making use of swap only when no RAM is available. Problem is that a value of 0 doesn't work and system doesn't use swap when there is not RAM available, causing several crashes.

Other routers keep this value in 60, and reverting this value to 60 makes all things to work right again, because swap is used when is needed.

Best regards.

Yea that is a misconception. Setting swappiness to zero means that it will almost never swap if it does at all. I would not mess with the router defaults. If you are worried about it swapping too much then start with lower the number in small increments. For example, a reasonable adjustment would have been to lower the value to 50. It is weird it is 0 with that particular model.

 

[Maloon]

Yea that is a misconception. Setting swappiness to zero means that it will almost never swap if it does at all. I would not mess with the router defaults. If you are worried about it swapping too much then start with lower the number in small increments. For example, a reasonable adjustment would have been to lower the value to 50. It is weird it is 0 with that particular model.

Yea that is a misconception. Setting swappiness to zero means that it will almost never swap if it does at all. I would not mess with the router defaults. If you are worried about it swapping too much then start with lower the number in small increments. For example, a reasonable adjustment would have been to lower the value to 50. It is weird it is 0 with that particular model.

Sorry, but I don't understand you. Default value for GT-AX6000 is 0. I am not talking about using swap too much, I am talking about a bug because swap is never used, and cause is swappiness value.

 

[ColinTaylor]

It is weird it is 0 with that particular model.

AFAIK all Asus routers have the swappiness set to zero. This would be "correct" for a router performing normal duties. But regardless of the value that's set swapping should always occur when the router approaches an out of memory situation. So the bug is that this doesn't appear to be happening. Which is very strange as this is the basic design of memory management and Asus shouldn't be altering it.

I checked this on my RT-AX86U (with swappiness=0) and it works as expected.

@Maloon What kernel version does your router have?

 

[Martinski]

AFAIK all Asus routers have the swappiness set to zero.

That would be incorrect. All AC-type routers that I have access to right now (RT-AC86U, RT-AC68U, GT-AC5300) have the default value for swappiness set to 60.

Perhaps the default value was changed for some AX routers (e.g. routers with 1GB RAM or more).

 

[Maloon]

AFAIK all Asus routers have the swappiness set to zero. This would be "correct" for a router performing normal duties. But regardless of the value that's set swapping should always occur when the router approaches an out of memory situation. So the bug is that this doesn't appear to be happening. Which is very strange as this is the basic design of memory management and Asus shouldn't be altering it.

I checked this on my RT-AX86U (with swappiness=0) and it works as expected.

@Maloon What kernel version does your router have?

Hi,

My kernel version is 4.19.183, from AsusWRT-Merlin 386.7_2. And yes, bug is that swappiness=0 doesn't work as expected.

I have checked a RT-AX88U, and swappiness is set to 60.

 

[Maloon]

I think that swappiness is set to 0 in the SDK used by GT-AX6000 and XT12. Other models, even with 1GB of RAM, have a value of 60.

 

[Martinski]

AFAIK all Asus routers have the swappiness set to zero.

FYI,

 

[SomeWhereOverTheRainBow]

That would be incorrect. All AC-type routers that I have access to right now (RT-AC86U, RT-AC68U, GT-AC5300) have the default value for swappiness set to 60.

Perhaps the default value was changed for some AX routers (e.g. routers with 1GB RAM or more).

I concur, and most linux devices I have ever used it is not unusual for the typical default to be no lower than 45. Typically you wouldn't want to wait for an out of memory situation before stale pages are allowed to swap.

 

[ColinTaylor]

That would be incorrect. All AC-type routers that I have access to right now (RT-AC86U, RT-AC68U, GT-AC5300) have the default value for swappiness set to 60.

That's very strange because my old RT-AC68U also had swappiness = 0 (I haven't used it for a year so maybe it changed in later firmware versions).

Are those routers running standard firmware without addons? I remember there were discussions with certain addons about changing the default swappiness.

 

[Martinski]

I think that swappiness is set to 0 in the SDK used by GT-AX6000 and XT12. Other models, even with 1GB of RAM, have a value of 60.

In a typical system where the swappiness value > 0, when the free memory in the memory zone (DMA) falls below the "low water mark" the swap algorithm starts scanning RAM pages looking for memory that can be freed using the swappiness value to determine the ratio of file-backed memory pages (nr_file_pages) & anonymous memory pages (nr_anon_pages) that can be marked for reclamation and decide whether the kernel runs with a preference for freeing up file-backed memory pages or anonymous memory pages.

Normally the preferred mode (i.e. higher priority) is to free the file-backed memory pages first. If the swappiness value == 100 it means that both file-backed memory pages & anonymous memory pages have the same priority WRT memory reclamation.

When swappiness value == 0, swapping occurs only when the combined value of file-backed pages (nr_file_pages) and free (nr_free_pages) pages is less than the "high water mark."

The values for low water mark, high water mark, nr_file_pages, nr_free_pages & nr_anon_pages are shown with the following command:

cat /proc/zoneinfo

Example:

For some reason, it looks like ASUS decided that for some AX routers the swappiness value should be zero.

 

[ColinTaylor]

I concur, and most linux devices I have ever used it is not unusual for the typical default to be no lower than 45.

A router isn't a typical Linux server or PC therefore the memory usage is different.

For some reason, it looks like ASUS decided that for some AX routers the swappiness value should be zero.

I would say that is the correct value to set for a router. Swap on a USB device is massively slow and should be avoided at all costs. Unlike a PC or server there are almost no anonymous memory pages that aren't being used all the time - not enough to have a noticeable affect if they were preemptively paged out. In my test there was only ~40MB of anonymous pages that could be swapped.

 

[Martinski]

That's very strange because my old RT-AC68U also had swappiness = 0 (I haven't used it for a year so maybe it changed in later firmware versions).

Are those router's running standards firmware without addons? I remember there were discussions with certain addons about changing the default swappiness.

Here is a screenshot of an RT-AC68U model:

The GT-AC5300 is running the latest stock OEM f/w with no add-ons. The RT-AC68U & the RT-AC86U are running RMerlin's f/w. No add-ons for the RT-AC68U. Only the YazFi add-on is installed on my RT-AC86U. I just checked another RT-AC86U router running RMerlin f/w with no add-ons and the swappiness default value is 60 as well.

 

[Martinski]

In a typical system where the swappiness value > 0, when the free memory in the memory zone (DMA) falls below the "low water mark" the swap algorithm starts scanning RAM pages looking for memory that can be freed using the swappiness value to determine the ratio of file-backed memory pages (nr_file_pages) & anonymous memory pages (nr_anon_pages) that can be marked for reclamation and decide whether the kernel runs with a preference for freeing up file-backed memory pages or anonymous memory pages.

Normally the preferred mode (i.e. higher priority) is to free the file-backed memory pages first. If the swappiness value == 100 it means that both file-backed memory pages & anonymous memory pages have the same priority WRT memory reclamation.

When swappiness value == 0, swapping occurs only when the combined value of file-backed pages (nr_file_pages) and free (nr_free_pages) pages is less than the "high water mark."

The values for low water mark, high water mark, nr_file_pages, nr_free_pages & nr_anon_pages are shown with the following command:

cat /proc/zoneinfo

Example:
View attachment 43550

For some reason, it looks like ASUS decided that for some AX routers the swappiness value should be zero.

Some technical context for those interested:

RAM is allocated in pages, where each memory page is typically 4KBytes, but the size can also be dynamically determined by the kernel at boot time based on the system architecture & filesystem type.

File-backed memory pages contain data that has been read from a file, so if the system freed this type of memory and needed to obtain that data again, it can be retrieved from the source file again. This kind of memory pages is not stored in swap.

Anonymous memory pages contain data that has no underlying file (including block devices) associated with it. These pages typically consist of memory allocations requested on the fly by programs to store runtime data, critical function call stacks & heap memory. Because there are no actual files behind this type of data, special care must be taken to ensure that this "anonymous" data is stored & maintained in RAM for as long as possible. Only when absolutely necessary the anonymous pages are written to swap (if it exists) before they can be freed.

 

[Martinski]

A router isn't a typical Linux server or PC therefore the memory usage is different.


I would say that is the correct value to set for a router. Swap on a USB device is massively slow and should be avoided at all costs. Unlike a PC or server there are almost no anonymous memory pages that aren't being used all the time - not enough to have a noticeable affect if they were preemptively paged out. In my test there was only ~40MB of anonymous pages that could be swapped.

Yeah, in a typical router being used only for networking duties a swap file is really not needed, especially when RAM size is 512MB or more. In my RT-AC86U there is a 1GB swap file, and I have never seen it used. I think the need for a swap file comes up only when using some add-ons like diversion, or when using a USB drive as a file or media server, where a lot of file-backed memory pages are allocated.

Also, for ASUS routers it looks like the typical values set for "low water mark" & "high water mark" are fairly low (6400 & 7680 pages respectively, equivalent to 25MB & 30MB in my RT-AC86U) so the swap algorithm is not triggered until RAM has almost been depleted. So this explains why a swap file is very rarely needed under "normal" operating conditions.

 

[bluepoint]

Just for reference my RT-Ax88u's swappiness value in 386.8 is

ASUSWRT-Merlin RT-AX88U 386.8_0 Sat Aug 13 20:54:16 UTC 2022
xxxxx@RT-AX88U/tmp/home/root# cat /proc/sys/vm/swappiness
60

It uses the swap file when the memory is @ 95-97% capacity of 1G.

 

[RMerlin]

Since this comes from Broadcom's SDK, I'm not really confortable with blindly change this, and potentially face issues down the road.

If a user really needs a different value because they use a swap, then I recommend they change the value themselves in a startup script.

 

[SomeWhereOverTheRainBow]

A router isn't a typical Linux server or PC therefore the memory usage different.

Sorry let me rephrase that for you, Unix device. Linux was a mistyped. Even on embedded devices. I have yet to see them ever set a swappiness to zero. While I am not saying it is a bad choice to do such if swapping isn't warranted or wanted, I still have yet to see it as a default on any Unix system or embedded device that supports having a swap space.

 

[SomeWhereOverTheRainBow]

And it doesn't look like all AX models are treated equally in this regard.

RT-AX88U-C7C0:/tmp/home/root# cat /proc/sys/vm/swappiness
60

 

[ColinTaylor]

Sorry let me rephrase that for you, Unix device. Linux was a mistyped.

No, I meant Linux as that's what we're talking about with this router. I've spent a lot of time in my career fine tuning virtual memory and swap on Solaris, Unixware, AIX, etc. (even IBM mainframes) and they each have their own subtle differences. But regardless of that the OP's router shouldn't be failing to swap when required.