TCP Timeout: Established and MAX

[Calisro]

I have my TCP Timeout: Established set to 1200 seconds but I noticed the "default" on the rt-ac68u is 5 days? That is a long "established" timeout and the longest i've seen. Its way more than defaults i've seen on other firmwares/routers.

What gives? Why so high? Seems like a great way to max the connections.

 

[ColinTaylor]

Strange. My N66U says the default is 1200.

 

[Calisro]

Yes, that is more inline with what i'm used to seeing

mine is "Default: 432000 (5 days)"

 

[Calisro]

What else is interesting is that on the ac68u the conntrack max is fixed (seems unchangable) at 300000.

I thought the ct_max was based on memory. This router has 250M of RAM. The formula I've always thought was applicable for an absolute max was RAMSIZE (in MegaBytes) * 64 = 16000. (which i think is still too high.... )

What am I missing?

 

[RMerlin]

Newer Linux kernels increased that value to 5 days. That's why it only appears on the newer ARM models, with the newer kernel.

 

[Calisro]

I meant 5 days is a long time for established connectiins. I cant see a use case for such a large number especially considering the forced 300000 tc_max. Can we not adjust tc_max? When I try and apply, it remains 300000.

 

[Calisro]

I am really perplexed by the 300000 max connections. The hash size for conntrack on this router is 4k. The netfilter.org recommendation is for 8 connections per bucket. That only leaves room to track 32k connections yet we have a max connections set to 300000.

What will happen is when we get above 32k, conntrack will drop packets. If it was sized correctly, conntrack would just overwrite old entries. With such a small hash in relation to the max connections it can't manage.

and its not adjustable!

 

[john9527]

Hmmm....maybe I shouldn't fix the bug in my fork . I had fixed the inability to change the max connections value, but when I went to pick up the default after a factory reset it picked up the system value (15K-16K depending on the router) instead of the 300000 value.

 

[kvic]

I don't see problem in practice with such huge numbers though. ESTABLISHED entries change to TIME_OUT pretty quickly and then disappear. nf_conntrack never blow up either.

Could it be that all socket apps are good citizens or something else in play...?

 

[Calisro]

I don't see problem in practice with such huge numbers though. ESTABLISHED entries change to TIME_OUT pretty quickly and then disappear. nf_conntrack never blow up either.

Could it be that all socket apps are good citizens or something else in play...?

Hash size and 300000 conn limit do not jive. If asus wants a 300000 limit then they need a 50-100k hash size and not 4k. and they didnt add the huge hash because they didnt want to eat the RAM.

A 5 day ESTABLISHED makes it so easy to DOS the system (intentionally or not) and since that limit exceeds the actual storage table (hash size) conntrack can't even self-correct by overwriting its entries.

In practice someone at asus can't do math.

Anyone using this router in an environment where they don't control the clients will run into problems (torrents and intentional/non-intentional DDOS).

@RMerlin, can we at least get the max value to be changable? I'd also like to see the defaults changed to a sane value but i'd settle for at least being able to change it.

 

[RMerlin]

Hash size and 300000 conn limit do not jive. If asus wants a 300000 limit then they need a 50-100k hash size and not 4k. and they didnt add the huge hash because they didnt want to eat the RAM.

A 5 day ESTABLISHED makes it so easy to DOS the system (intentionally or not) and since that limit exceeds the actual storage table (hash size) conntrack can't even self-correct by overwriting its entries.

In practice someone at asus can't do math.

Anyone using this router in an environment where they don't control the clients will run into problems (torrents and intentional/non-intentional DDOS).

@RMerlin, can we at least get the max value to be changable? I'd also like to see the defaults changed to a sane value but i'd settle for at least being able to change it.

They're all already editable under Tools -> Other Settings.

The 5 days default value on 2.6.36 has already been changed upstream by Asus.

diff --git a/release/src-rt-7.x.main/src/linux/linux-2.6.36/net/netfilter/nf_conntrack_proto_tcp.c b/release/src-rt-7.x.main/src/linux/linux-2.6.36/net/netfilter/nf_conntrack_proto_tcp.c
index 2193461..aa5df5e 100644
--- a/release/src-rt-7.x.main/src/linux/linux-2.6.36/net/netfilter/nf_conntrack_proto_tcp.c
+++ b/release/src-rt-7.x.main/src/linux/linux-2.6.36/net/netfilter/nf_conntrack_proto_tcp.c
@@ -78,7 +78,7 @@ static unsigned int nf_ct_tcp_timeout_unacknowledged __read_mostly =   5 MINS;
 static unsigned int tcp_timeouts[TCP_CONNTRACK_MAX] __read_mostly = {
        [TCP_CONNTRACK_SYN_SENT]        = 2 MINS,
        [TCP_CONNTRACK_SYN_RECV]        = 60 SECS,
-       [TCP_CONNTRACK_ESTABLISHED]     = 5 DAYS,
+       [TCP_CONNTRACK_ESTABLISHED]     = 40 MINS, /* was 5 DAYS, no less then tcp_keepalive_time + tcp_keepalive_probes * tcp_keepalive_intvl */
        [TCP_CONNTRACK_FIN_WAIT]        = 2 MINS,
        [TCP_CONNTRACK_CLOSE_WAIT]      = 60 SECS,
        [TCP_CONNTRACK_LAST_ACK]        = 30 SECS,

Once again, it's already user-configurable.

 

[Calisro]

They're all already editable under Tools -> Other Settings.
The 5 days default value on 2.6.36 has already been changed upstream by Asus.

The ESTABLISHED is configurable (which I changed). The MAX connections of 300,000 is NOT configurable. When you put a number other than 300,000 in, it still saves as 300,000. try it. It just reverts it back to 300000.

I see this in git.

nvram_set("ct_max", "300000"); // force

 

[RMerlin]

The ESTABLISHED is configurable (which I changed). The MAX connections of 300,000 is NOT configurable. When you put a number other than 300,000 in, it still saves as 300,000. try it. It just reverts it back to 300000.

I see this in git.

nvram_set("ct_max", "300000"); // force

Asus must have changed it at some point, it used to be properly configurable (at one point I had mine accidentally set to a lower value). I'll have to check where they are doing it, and figure out if there was any reason behind that design change.

 

[RMerlin]

BTW, I've forwarded this discussion to Asus (on the mismatch between hash size and maximum allowed connections).

 

[Calisro]

Thx Merlin. It'll be good to have a response.

 

[RMerlin]

They tell me that they actually do tests to validate that the router can handle 200,000-300,000 connections, so it does reach that many connections without crashing. However I assume this means that those connections aren't all filling up the hash table since they are never transferring data at the same time (I have a hard time seeing any real-life case where a home router could transfer data on 300,000 simultaneous connections anyway).

Sorry, but conntrack stuff is really not my area of expertise, so I don't understand the difference between their test scenario, and the one described in this thread. I assume it's a case of marketing vs engineering semantics.

 

[vanic]

The easy way to test is using BT, prepare 50+ PC and setup BT. And then, running that, 65535*50 is almost near 300,000 connections.

 

[RMerlin]

The ESTABLISHED is configurable (which I changed). The MAX connections of 300,000 is NOT configurable. When you put a number other than 300,000 in, it still saves as 300,000. try it. It just reverts it back to 300000.

I see this in git.

nvram_set("ct_max", "300000"); // force

Only certain models force it. Most only set it if the nvram isn't set.


Sent from my Nexus 9 using Tapatalk

 

[RMerlin]

The easy way to test is using BT, prepare 50+ PC and setup BT. And then, running that, 65535*50 is almost near 300,000 connections.

That would be a lot of virtual machines LOL. But it shows what type of network would be able to reach such a high number of connections.

Sent from my Nexus 9 using Tapatalk

 

[kvic]

Thanks guys, you really got me curious about this topic. I dig further. I think I would share what I found. So that it could be cross checked if the below understanding is right.

This page provides very good skeleton info on conntrack. Here are what we have on asuswrt:

# cat /proc/sys/net/ipv4/netfilter/ip_conntrack_max
300000
# cat /proc/sys/net/ipv4/netfilter/ip_conntrack_buckets
4096

Note that a bucket in the hash table is a pointer to a linked list. With the above numbers and in the worst scenario, we will have 4096 linked lists of each on average 74 entries. When that happens, performance will degrade possibly a lot since accessing linked list is sequential. However, it will still be functional.

I think Asus makes a sensible choice here between maximum capability and optimal performance on average scenarios. Let's say 4096 buckets with 8 entries each. It's 32k connections. That's a lot for a home even if people are crazy for BT..

One question remains. The "5 day" thing could easily drive conntrack into the worse scenario. In reality, seems we rarely see many entries in the hash table nor observe any performance degradation. This might explain why:

# cat /proc/sys/net/ipv4/tcp_tw_reuse
1

This enables the kernel to re-use connections in TIME_WAIT states as soon as feasible. Now seems all pieces fall in place and look like a carefully engineered craft.

Also thanks for the 50+ BT machine challenge...lol. On my back of the envelope calculation, you will only need about 7 hosts. Doesn't matter how you split between client and server sides (router's NAT in between). I'll leaved it as an exercise for interested folks. I wish people having time could put this into test and challenge the 300k connections. If you do, let us hear you in a future post. lol