nf_conntrack: expectation table full and other log oddities

[Maverickcdn]

Hello all

Wondering if someone can shed some light on what these are and if there is any resolution to these appearing in the log. Its been mentioned on here a couple of times but I can't see a clear answer or resolution as to why this happens.

Besides my AC86U (384.9) consistently dropping/reconnecting 5Ghz clients (regardless of MIMO/Fairness/Beamforming settings) and the annoying WLAN Assoc/Disassoc messages flooding the syslog.

I run a webserver behind my Router via ethernet and when running a website test, like GTmetrix,Page Insights etc, the router syslog will post about 20-30 lines of nf_conntrack: Expectation table full.

This can also be induced 'sometimes' by simply running a download speedtest on a client on the network though fast.com/speedtest.net etc.

Ive tried increasing the nf_conntrack_max setting in /proc/sys/net which seems to be symlinked to the nf_conntrack_max in /proc/sys/net/netfilter but doesnt appear to help. It will save my changes but I have no idea if they actually get applied as upon reboot they reset to the default 300000

I could be plagued with a bad client somewhere on my network but just thought Id start with finding out why they appear in the first place and if I can possibly use a script at boot to maybe increase the table limit to avoid these messages.

 

[Maverickcdn]

Happens everytime... I make a post then find my answer later on... haha

For anyone else curious about this nf_conntrack: expectation table full showing up in their logs, I followed the suggestions in https://www.snbforums.com/threads/tcp-timeout-established-and-max.26580/page-2#post-200012 this post and increased my hash table size from 3XXX ( it was 3000 something, cant remember exactly) and changed it to 8192,

Not seeing any issues at the moment and it succeeded at removing the nf_conntrack messages I was getting in the logs... for the time being anyway, will update if it changes.

UPDATE: still got them, but after changing nf_conntrack_expect_max (duh) from 52 to 352 and restart conntrack so far so good.

 

[Maverickcdn]

Can anyone help enlighten me on how nf_conntrack is used on Asus firmware. Im just curious if by increasing my conntrack_expect_max Im introducing potential issues.

Updated to 384.10beta last night, still receive the nf_conntrack_expect_max-table full errors. After adjusting the value from 52 to 152 my issue disappears...

But according to https://www.kernel.org/doc/Documentation/networking/nf_conntrack-sysctl.txt... if I do the math right the AC86 should have a value set to 32 for the table size..... default is 52.... I've set it to 152, besides an obvious increase in Ram usage (about 10mb) I havent noticed any issues.

 

[RamGuy]

Did you get this resolved? I see these messages nonstop in the system logs.

I followed your linked discussion and I tried to limit my TCP Connection limit to 28244, it was 300000. My default hash size on the RT-AX88U is 7061 so 7061 * 4 is that's why I edited ct_max=28244. I did also do the echo 152 > /proc/sys/net/netfilter/nf_conntrack_expect_max.

But I'm still seeing these messages nonstop;

Apr 10 06:32:25 kernel: nf_conntrack: expectation table full
Apr 10 06:32:25 kernel: nf_conntrack: expectation table full
Apr 10 06:32:25 kernel: nf_conntrack: expectation table full
Apr 10 06:32:25 kernel: nf_conntrack: expectation table full
Apr 10 06:32:25 kernel: nf_conntrack: expectation table full
Apr 10 06:32:25 kernel: nf_conntrack: expectation table full
Apr 10 06:32:26 kernel: nf_conntrack: expectation table full
Apr 10 06:32:26 kernel: nf_conntrack: expectation table full
Apr 10 06:32:26 kernel: nf_conntrack: expectation table full
Apr 10 06:32:26 kernel: nf_conntrack: expectation table full
Apr 10 06:59:03 kernel: net_ratelimit: 7 callbacks suppressed
Apr 10 06:59:03 kernel: nf_conntrack: expectation table full
Apr 10 06:59:03 kernel: nf_conntrack: expectation table full
Apr 10 06:59:03 kernel: nf_conntrack: expectation table full
Apr 10 06:59:03 kernel: nf_conntrack: expectation table full
Apr 10 06:59:03 kernel: nf_conntrack: expectation table full
Apr 10 06:59:03 kernel: nf_conntrack: expectation table full
Apr 10 06:59:03 kernel: nf_conntrack: expectation table full
Apr 10 06:59:03 kernel: nf_conntrack: expectation table full
Apr 10 06:59:03 kernel: nf_conntrack: expectation table full
Apr 10 06:59:03 kernel: nf_conntrack: expectation table full

 

[RMerlin]

Fixing this is trickier, because the expectation table size is computed by the kernel. Some of the parameters used to compute it are not user-configurable through /proc, so this will require further digging.

 

[Maverickcdn]

I never got back to reading about this more after I was able to suppress them unfortunately.

What I can tell you is changing the hash table sizes/connection limits wont do anything and should likely be left alone.

The only thing worth changing to save from potentially not harming anything else and to resolve the messages is the value in /proc/sys/net/netfilter/nf_conntrack_expect_max.

My 86u had 52 as a default, I started with 352 but was able to set it down to 102 and I very rarely see the 'expectation table full' messages in the log anymore.

From what I can tell its a first line defense against a DOS attack limiting the amount of connections that are expected to be handled by Linux. Increasing the expect max shouldnt have any affect on the rest of the system from what I could tell granted you have the memory space.

Im curious though, do you run a website or anything behind your router? Is there anything you can do to force the log entry to show up?

 

[Maverickcdn]

And for what its worth I used nano to edit the value instead of echo'ing it to the file, but really shouldnt make a difference

and you have to 'service restart_conntrack' as well or the setting doesnt take

 

[RamGuy]

I have a ton of various clients on my network, and my torrentserver is most like the major source of the issue. I have a great VMware ESXi server, I have access to some stellar routers and firewalls so I can easily get around any TCP connection issues but the problem is that I have to rely on a solution that supports UPNP and OPEN NAT/Cone NAT filtration and currently newer Asus routers with Asus Merlin firmware is my only solution....

Do you have any idea on what would be the maximum value I can set for nf_conntrack_expect_max?

 

[RamGuy]

Fixing this is trickier, because the expectation table size is computed by the kernel. Some of the parameters used to compute it are not user-configurable through /proc, so this will require further digging.

I fully understand. I finally got myself a /54 IPv6 prefix and I had hoped all my needs for Cone NAT for gaming would vanish but unfortunately only 10% of the games out there actually offers IPv6 connectivity... So I still have to rely on the Asus for most of my networking and as it gets hit with a huge amount of connections I guess it's just not feasible to expect it to handle it all.

 

[Maverickcdn]

I have a ton of various clients on my network, and my torrentserver is most like the major source of the issue.

Do you have any idea on what would be the maximum value I can set for nf_conntrack_expect_max?

Id agree, if you seed files its likely those incoming connections as there is many connections involved with Torrent.

Personally, I know nothing about how conntrack works in Linux. Hopefully a resident Guru can chime in on the 'consequences' of changing that value.

From my experience, the only side effect I noticed was increased RAM usage as I increased the value. Havent noticed any issues with clients connections (stable pings/packets etc) or any disrupted/slow access to the webserver behind the router.

Its an expect table so as long as the entries are being moved to the active table and the TTL (if they have such a thing) is not high on those expected connections and doesnt plug up the table you should be able to keep the number fairly low but suppress the messages and not high enough to introduce a potential issue, just my guess. Its only a DOS front line protection from what I can tell and is just a throttle for incoming connections. For me it was a week or two of fadiddling with the setting and hammering the router till I was comfortable with the result. I settled on 102, the messages still appear only when there is multiple clients in use and the web server is being accessed from WAN.

 

[RMerlin]

My workload is simply too high at this time for me to have another look at this, so if someone else wants to, be my guest. The two key variables I had identified at the time were:

/proc/sys/net/netfilter/nf_conntrack_expect_max (SDK7 uses 128, HND is lower)
/proc/sys/net/nf_conntrack_buckets

Unfortunately I didn't note down the URL where I tracked the technical details on how these values are calulated by the kernel. I only remember that there's a read-only value set by the kernel, and the expect table size is derived from it.

 

[Maverickcdn]

Way out of my league to make any sense of this but if it helps anyone...

The only issue I saw was the math when you're calculating buckets vs expect max size
@16384 (max size) the expect max should only be 64 default. Not sure about going above default. It seems to me the value of expect_max would be tied to available memory and the only consequence would be insufficient memory.... my totally uneducated guess.

My 86U when running cat on /proc/sys/net/netfilter/nf_conntrack_buckets gives 3584 (this may not be stock Im not 100%) .... use the quoted math below and default would be 14 where Asus or something else set it to 52

https://www.kernel.org/doc/Documentation/networking/nf_conntrack-sysctl.txt

nf_conntrack_buckets - INTEGER
Size of hash table. If not specified as parameter during module
loading, the default size is calculated by dividing total memory
by 16384 to determine the number of buckets but the hash table will
never have fewer than 32 and limited to 16384 buckets. For systems
with more than 4GB of memory it will be 65536 buckets.
This sysctl is only writeable in the initial net namespace.

nf_conntrack_expect_max - INTEGER
Maximum size of expectation table. Default value is
nf_conntrack_buckets / 256. Minimum is 1.

 

[RMerlin]

The recommendations don't specify however if these values are suggested for a workstation, or a router - totally different network workload.

Personally, I'd use the values used by SDK6/SDK7 routers (RT-AC68U, RT-AC88U, etc...) as a reference. They use larger values than the HND models like the AC86U or AX88U, and so far these two models are the only ones to generate these error messages. But they are also the only ones using a 4.1.xx kernel...

 

[Maverickcdn]

So true, but lazy me that was the first page of Google

Thanks for the reply RMerlin, if you're comfortable with recommending the values from other models, I might just bump mine to 128 to dispel even the sparse messages I still get.

Just for a note even the few days testing at 352 I never had any noticeable issues. Looking at values from other manufacturers they vary wildly from 16 (cheapie low end) to 2048->4096 (ubiquiti edge)

Not to be a hater on the 86u but this thing has given me more grief than any other previous Asus router, but you cant argue with the awesome VPN performance. Connecting with 4096bit keys is lightning fast.

 

[RamGuy]

Setting the nf_conntrack_expect_max to 352 seems to fix this issue for me, but this does not survive reboot so I just created a .sh file with the following;


#!/bin/sh
# Purpose: Increase nf_conntrack_expect_max to maximum value
# Author: RamGuy
# -------------------------------------------------------------
echo 352 > /proc/sys/net/netfilter/nf_conntrack_expect_max
service restart_conntrack

And saved it into /jffs/scripts so it runs as boot and we are finally rolling.

EDIT:

Nevermind, doesn't seem like the scripts is being run at boot. How do I make sure that the scripts will run at boot?

 

[ColinTaylor]

Nevermind, doesn't seem like the scripts is being run at boot. How do I make sure that the scripts will run at boot?

You have to give it the appropriate name. Try nat-start.

 

[RamGuy]

Thank you! That did the trick!

 

[sinshiva]

have you noticed this happening when 5ghz drops clients?

 

[RamGuy]

352 is the highest value? I still start to get the "Apr 27 12:37:06 kernel: nf_conntrack: expectation table full" messages in the system log even with the nf_conntrack_expect_max set to 352 after about 12 hours+

I have tried to limit the maximum amount of connections allowed on my torrentserver down from unlimited to 1000 but it doesn't seem to help much. Are these messages something that one should really care / worry about? What effect does this actually have on network performance and stability?

 

[RamGuy]

have you noticed this happening when 5ghz drops clients?

I'm not using my Asus RT-AX88U for WLAN, it's strictly being used for routing and NAT due to the Open NAT / Full Cone NAT support.