Voxel Best tunnel for R7800 (HW acceleration, perfs…)

[HELLO_wORLD]

Hi dear community,

I am about to get a tunnel to a good secondary provider, as my main one is having routing issues lately and I need to use VPN's a lot from LAN devices.

Here are my choices:
OpenVPN, Up to 100 Mbits/s
L2TP, up to 2 Gbits/s
EoIP, up to 2 Gbits/s
IPIP, up to 1 Gbits/s
6in4, up to 1 Gbits/s
GRE, up to 2 Gbits/s
Wireguard, up to 1 Gbits/s

OpenVPN and Wireguard would impact too much the performances I believe (and OVPN is limited to 100 Mbits/s).
6in4 is specific so not considered here.

I am recommended L2TP.

What would be the solution impacting the less possible the performances?
Are any of these tunnels taken in charge by HW acceleration in our R7800?

 

[Voxel]

It is interesting... But is not quite clear for me (or I am missing something...). Your provider is VPN provider? I.e. VPN server. What is on your side? Single specific VPN client? Or multiply clients from your LAN?

For example, why only 100 Mbits/s for OpenVPN? Even R7800 could get more. I use it with ChaCha20-Poly1305 cipher and I have about 150 Mbit/s.
Why WG is limited? Not clear as well.

And what is goal? Just speed or security in the first turn? Pure L2TP is a VPN protocol that doesn’t offer any encryption. L2TP/IPsec - yes, secure...

Voxel.

 

[HELLO_wORLD]

Actual FTTH ISP provider is DHCP on WAN side (ONT). Routing is getting bad and unreliable to some destinations, and right now I am using VPN from individual LAN devices to circumvent that.

Ultimately, I will change ISP for another one (I know the owners of the new ISP). The one I will go to in the future offers tunneling services (the ones described), and the IPv4 & v6 I will get through the tunnel will be the ones I will keep once I switch ISP.
The route to the tunnel entry (through current ISP) is reliable, so no problem from current ISP getting to it.

So the goal is using tunnel for reliable internet backbone and connection to peering (PP, IXP) and PNI…

My idea is to put the entire LAN in the one tunnel, so tunnel on the router and routing all traffic through GRE. I don't need encryption, my goal is best performance.
I already enabled GRE on the firmware, and made a few tunnel tests on my LAN.
I also bought the tunnel service and will have it this coming week (it is cheap), and will test the router abilities for such tunneling.

If it is too slow, I will probably get something like a Mikrotik RB5009UPr+S+IN (ISP confirmed it works very well with tunnels and it is powerful), and still use R7800 as AP.

 

[sfx2000]

I am recommended L2TP

L2TP by itself doesn't have encryption, so use it with ipsec...

L2TP/IPsec is the gold standard if both ends support it...

Bit of a pain to configure, but worth the effort.

 

[HELLO_wORLD]

L2TP by itself doesn't have encryption, so use it with ipsec...

L2TP/IPsec is the gold standard if both ends support it...

Bit of a pain to configure, but worth the effort.

I don't need encryption (plus it would slow down).
I am starting with a simple GRE tunnel, and I will see how it goes.

If it it still too slow, I have several options:

• get a SBC/mini PC between the router and the WAN to just deal with the tunnel and let the R7800 do the routing as usual.
• replace the R7800 with a new router like:
  • the Mikrotik,
  • a MASHAbin,
  • or a mini PC like this https://www.aliexpress.com/item/325...st.0.0.2d395e5bVbIVLB&gatewayAdapt=4itemAdapt
• I can also just with the R7800 decide to only send some devices through the tunnel (mostly the tablets and PCs), using a different routing for them in the router.

 

[HELLO_wORLD]

Ok, so I got GRE the tunnel.

I did set it up that way on the router:

ip tunnel add mw0 mode gre remote TUN_ENDPOINT_IP local MY_PUBLIC_IP ttl 255
ip link set mw0 up
ip address add 10.0.0.26/30 dev mw0
ip rule add from LAN_IP_OF_TEST_DEVICE/32 table 100
ip route add default via 10.0.0.25 dev mw0 table 100

iptables -t raw -I PREROUTING -i mw0 -j ACCEPT
iptables -I INPUT -i mw0 -j ACCEPT
iptables -I FORWARD -i br0 -o mw0 -j ACCEPT
iptables -I FORWARD -i mw0 -o br0 -j ACCEPT
iptables -I OUTPUT -o mw0 -j ACCEPT
iptables -t nat -I POSTROUTING -o mw0 -j SNAT --to-source MY_TUNNEL_PUBLIC_IP

TUN_ENDPOINT_IP being the public IP where the tunnel is in internet.
MY_PUBLIC_IP being my public IP through my ISP (not tunnel).
10.0.0.26 being the IP on my side of the tunnel.
10.0.0.25 being the IP on the remote side of the tunnel.
MY_TUNNEL_PUBLIC_IP being the public IP I have through the tunnel.
LAN_IP_OF_TEST_DEVICE being the local IP of the device I want my R7800 to route through the tunnel.

Now, the TEST_DEVICE on my LAN is connected to internet through the tunnel, but the performance are awful:

root@hestia:~# speedtest -s 2231

   Speedtest by Ookla

     Server: LaFibre.info BBR - Massy (id = 2231)
        ISP: MilkyWan
    Latency:    93.61 ms   (399.74 ms jitter)
   Download:     0.33 Mbps (data used: 529.8 kB )                               
     Upload:    20.45 Mbps (data used: 21.1 MB )                               
Packet Loss:     0.0%

And one of the CPU on the R7800 is at 100% during all the test, so it is clearly the bottleneck.
This shows that my R7800 seems to very poorly be able to deal with a simple GRE tunnel (when it can handle OpenVPN at about 150 Mbps, that is more complex and has encryption).
I must be doing something wrong… The way I do it is clearly bypassing all NSS acceleration, and the performance is abnormally low ! GRE tunnel tests between the R7800 and one device on my LAN showed like 400 Mbps other ways with iPerf3…

Maybe it is MTU related… MTU for mw0 after it is set up is 1476, and LAN is 1500. Maybe it has to completely deconstruct/reconstruct all the packets?
I will look into this, but if someone already know all the trick, please share

For reference, same device being routed through usual WAN:

root@hestia:~# speedtest -s 2231

   Speedtest by Ookla

     Server: LaFibre.info BBR - Massy (id = 2231)
        ISP: K-NET SARL
    Latency:    31.69 ms   (0.01 ms jitter)
   Download:   961.46 Mbps (data used: 1.1 GB )                               
     Upload:   900.32 Mbps (data used: 1.0 GB )                               
Packet Loss:     0.0%

 

[sfx2000]

And one of the CPU on the R7800 is at 100% during all the test, so it is clearly the bottleneck.
This shows that my R7800 seems to very poorly be able to deal with a simple GRE tunnel (when it can handle OpenVPN at about 150 Mbps, that is more complex and has encryption).
I must be doing something wrong… The way I do it is clearly bypassing all NSS acceleration, and the performance is abnormally low ! GRE tunnel tests between the R7800 and one device on my LAN showed like 400 Mbps other ways with iPerf3…

Something is clearly wrong here -- r7800 on l2tp should saturate a gigabit link without Krait working hard...

Could be the GRE tunnel - try just l2tp - that should be fastpath, thus going thru NSS

 

[sfx2000]

Also - go back and review this > again <

https://people.netfilter.org/pablo/netdev0.1/slides/IPQ806x-Hardware-acceleration_v2.pdf

 

[HELLO_wORLD]

Ok, changed the MTU of the LAN device to 1400, and it is better, particularly upload:

root@hestia:~# speedtest

   Speedtest by Ookla

     Server: Netprotect - Paris (id = 37599)
        ISP: MilkyWan
    Latency:    21.17 ms   (0.13 ms jitter)
   Download:   197.76 Mbps (data used: 203.5 MB )                               
     Upload:   930.30 Mbps (data used: 1.1 GB )                               
Packet Loss: Not available.

For both, upload and download, CPU is maxed. Still using the GRE tunnel.
Now, why is download so slow in comparison to upload? GRE decapsulation is taking more CPU that GRE encapsulation?

That means that what I suspected was happening… The router had to re-fragment the packets 1500 to 1476.
I will try now with a MTU identical to the one of the tunnel (1476) because maybe it has to re-fragment 1476 to 1400…


I am quite surprised that according to the doc you sent, L2TP is accelerated, but not GRE…
To use L2TP, I will need to ask the tunnel provider to change it, so before I do that, I want to do my best with GRE first, and check how to make a L2TP tunnel on my side to practice before as well.

 

[HELLO_wORLD]

Update:
Changing MTU to 1476 did not make it better than 1400.

Since I am sending several devices through the tunnel, and setting manually MTU for each one would be a pain, I came with this solution:

iptables -t mangle -I FORWARD -o mw0 -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1436
iptables -t mangle -I FORWARD -i mw0 -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1436

This way, all TCP traffic is limited to 1476 (and situations where UDP or ICMP would use a huge frame is rare).

The tunnel provider cannot easily change GRE to L2TP (different infrastructure), so I need to make sure L2TP performances will be significantly better before asking a change).
So I will be setting up a L2TP tunnel to another location I have with another ISP and experiment/test and see how it performs.

In parallel to that, I am changing my rules, so my GRE tunnel can have the same firewall rules and protection than brwan.

 

[HELLO_wORLD]

Ok, I do have a strange problem with the GRE tunnel related to routing to LAN.

I use a specific routing table to put any LAN device in the tunnel. It is mostly working, except for a few sites like https://serverfault.com (and the affiliated sites like superuser, stackexchange, stackoverflow…).
For those, if a LAN device is using the tunnel, I get this error with curl:

~ % curl -sv https://serverfault.com >/dev/null
*   Trying 151.101.65.69:443...
* Connected to serverfault.com (151.101.65.69) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (OUT), TLS handshake, Client hello (1):
} [320 bytes data]
* error:02FFF036:system library:func(4095):Connection reset by peer
* Closing connection 0

Of course, I have no problem with these sites outside of the tunnel, or via un VPN (even if the VPN is using the tunnel).
More relevant here, is if I do the curl from the R7800 using the tunnel, it works (suggesting it is related to the routing/forwarding in the router):

~$ curl -sv --interface mw0 https://serverfault.com >/dev/null
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
} [5 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [108 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [4536 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [333 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
{ [1 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
> GET / HTTP/1.1
> Host: serverfault.com
> User-Agent: curl/7.84.0
> Accept: */*
>
{ [5 bytes data]
< HTTP/1.1 200 OK
< Connection: keep-alive
< cache-control: private
< content-type: text/html; charset=utf-8
< strict-transport-security: max-age=15552000
< x-frame-options: SAMEORIGIN
< x-request-guid: 7f171ec5-0d4e-4951-a0e5-25f912ca4104
< content-security-policy: upgrade-insecure-requests; frame-ancestors 'self' https://stackexchange.com
< Accept-Ranges: bytes
< Date: Mon, 08 Aug 2022 09:44:09 GMT
< Via: 1.1 varnish
< X-Served-By: cache-cdg20747-CDG
< X-Cache: MISS
< X-Cache-Hits: 0
< X-Timer: S1659951850.677056,VS0,VE81
< Vary: Fastly-SSL
< X-DNS-Prefetch-Control: off
< Set-Cookie: prov=bd80801c-deaf-6d8a-203d-6ae832f7b3b4; domain=.serverfault.com; expires=Fri, 01-Jan-2055 00:00:00 GMT; path=/; HttpOnly
< transfer-encoding: chunked
<
{ [5 bytes data]

The problem seems specific when the tunnel is routed to the LAN, but most sites are working fine, so why these?

It looks like a MSS problem, but I am not sure it is.
The router/WAN MTU is 1,500; the router/LAN MTU is also 1,500. The router/GRE MTU is 1,476.

Here is how I set the tunnel up:

ip t add mw0 mode gre remote IP_WAN_TUN_ENDPOINT local MY_ISP_PUBLIC_IP ttl 255
ip l s mw0 up
ip addr add MY_TUNNEL_IP/30 dev mw0
ip route add default via REMOTE_SIDE_TUNNEL_IP protocol static dev mw0 table 100
ip rule add preference 200 from 192.168.0.1 table main
ip rule add preference 300 from 192.168.0.1/26 table 100 # <- to have LAN using the tunnel

iptables rules on the router related to the tunnel:

iptables -t raw -w -I PREROUTING -i mw0 -j ACCEPT

iptables -t mangle -w -I FORWARD -o mw0 -j ACCEPT
iptables -t mangle -w -I FORWARD -o mw0 -p udp -j ACCEPT
iptables -t mangle -w -I FORWARD -o mw0 -p tcp -j ACCEPT
iptables -t mangle -w -I FORWARD -o mw0 -p tcp -m tcpmss --mss 1436 -j ACCEPT
iptables -t mangle -w -I FORWARD -o mw0 -p tcp -m tcpmss --mss 1437:1500 -j ACCEPT
iptables -t mangle -w -I FORWARD -o mw0 -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1436

iptables -t mangle -w -I FORWARD -i mw0 -j ACCEPT
iptables -t mangle -w -I FORWARD -i mw0 -p udp -j ACCEPT
iptables -t mangle -w -I FORWARD -i mw0 -p tcp -j ACCEPT
iptables -t mangle -w -I FORWARD -i mw0 -p tcp -m tcpmss --mss 1436 -j ACCEPT
iptables -t mangle -w -I FORWARD -i mw0 -p tcp -m tcpmss --mss 1437:1500 -j ACCEPT
iptables -t mangle -w -I FORWARD -i mw0 -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1436

iptables -w -I INPUT -i mw0 -j ACCEPT
iptables -w -I FORWARD -o mw0 -j ACCEPT
iptables -w -I FORWARD -i mw0 -j ACCEPT
iptables -w -I OUTPUT -o mw0 -j ACCEPT

iptables -t nat -w -I POSTROUTING -o mw0 -j SNAT --to-source MY_TUNNEL_PUBLIC_IP
iptables -t nat -w -I PREROUTING -i mw0 -j ACCEPT

I put TCPMSS to 1,500 - 26 (GRE) - 40 (TCP header) = 1,436.
Some rules are there simply to check the mss clamping is happening.

PART 1/3

 

[HELLO_wORLD]

serverfault.com and affiliated are in the subnet range 151.101.0.0/16

With a tcpdump (on the router, tunnel interface), here is what I get when I load the site from the LAN (and it fails):

root@HERMES:~$ tcpdump -tnn -i mw0 net 151.101.0.0/16
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on mw0, link-type LINUX_SLL (Linux cooked v1), snapshot length 262144 bytes
IP TUNNEL_PUBLIC_IP.57321 > 151.101.193.69.443: Flags [S], seq 1016197911, win 65535, options [mss 1436,nop,wscale 6,nop,nop,TS val 1435897636 ecr 0,sackOK,eol], length 0
IP 151.101.193.69.443 > TUNNEL_PUBLIC_IP.57321: Flags [S.], seq 2073372183, ack 1016197912, win 65535, options [mss 1436,sackOK,TS val 671609432 ecr 1435897636,nop,wscale 9], length 0
IP TUNNEL_PUBLIC_IP.57321 > 151.101.193.69.443: Flags [.], ack 1, win 2069, options [nop,nop,TS val 1435897651 ecr 671609432], length 0
IP TUNNEL_PUBLIC_IP.57321 > 151.101.193.69.443: Flags [P.], seq 1:518, ack 1, win 2069, options [nop,nop,TS val 1435897651 ecr 671609432], length 517
IP 151.101.193.69.443 > TUNNEL_PUBLIC_IP.57321: Flags [.], ack 518, win 285, options [nop,nop,TS val 671609448 ecr 1435897651], length 0
IP 151.101.193.69.443 > TUNNEL_PUBLIC_IP.57321: Flags [.], seq 1:1425, ack 518, win 285, options [nop,nop,TS val 671609451 ecr 1435897651], length 1424
IP 151.101.193.69.443 > TUNNEL_PUBLIC_IP.57321: Flags [.], seq 1425:2849, ack 518, win 285, options [nop,nop,TS val 671609451 ecr 1435897651], length 1424
IP 151.101.193.69.443 > TUNNEL_PUBLIC_IP.57321: Flags [P.], seq 2849:4273, ack 518, win 285, options [nop,nop,TS val 671609451 ecr 1435897651], length 1424
IP 151.101.193.69.443 > TUNNEL_PUBLIC_IP.57321: Flags [P.], seq 4273:5487, ack 518, win 285, options [nop,nop,TS val 671609451 ecr 1435897651], length 1214
IP 151.101.193.69.443 > TUNNEL_PUBLIC_IP.57321: Flags [R], seq 2073377670, win 0, length 0
IP 151.101.193.69.443 > TUNNEL_PUBLIC_IP.57321: Flags [R], seq 2073377670, win 0, length 0
IP 151.101.193.69.443 > TUNNEL_PUBLIC_IP.57321: Flags [R], seq 2073377670, win 0, length 0
IP TUNNEL_PUBLIC_IP.57322 > 151.101.193.69.443: Flags [S], seq 250459534, win 65535, options [mss 1436,nop,wscale 6,nop,nop,TS val 264874640 ecr 0,sackOK,eol], length 0
IP 151.101.193.69.443 > TUNNEL_PUBLIC_IP.57322: Flags [S.], seq 1998695891, ack 250459535, win 65535, options [mss 1436,sackOK,TS val 443883638 ecr 264874640,nop,wscale 9], length 0
IP TUNNEL_PUBLIC_IP.57322 > 151.101.193.69.443: Flags [.], ack 1, win 2069, options [nop,nop,TS val 264874653 ecr 443883638], length 0
IP 151.101.193.69.443 > TUNNEL_PUBLIC_IP.57321: Flags [P.], seq 4273:5487, ack 518, win 285, options [nop,nop,TS val 671609501 ecr 1435897651], length 1214
IP TUNNEL_PUBLIC_IP.57321 > 151.101.193.69.443: Flags [R], seq 1016198429, win 0, length 0
IP 151.101.193.69.443 > TUNNEL_PUBLIC_IP.57322: Flags [R], seq 1998695892, win 0, length 0
IP TUNNEL_PUBLIC_IP.57323 > 151.101.193.69.443: Flags [S], seq 2696764619, win 65535, options [mss 1436,nop,wscale 6,nop,nop,TS val 291229027 ecr 0,sackOK,eol], length 0
IP 151.101.193.69.443 > TUNNEL_PUBLIC_IP.57323: Flags [S.], seq 3935317826, ack 2696764620, win 65535, options [mss 1436,sackOK,TS val 926841068 ecr 291229027,nop,wscale 9], length 0
IP TUNNEL_PUBLIC_IP.57323 > 151.101.193.69.443: Flags [.], ack 1, win 2069, options [nop,nop,TS val 291229040 ecr 926841068], length 0
IP 151.101.193.69.443 > TUNNEL_PUBLIC_IP.57323: Flags [R], seq 3935317827, win 0, length 0
IP TUNNEL_PUBLIC_IP.57324 > 151.101.193.69.443: Flags [S], seq 3399970170, win 65535, options [mss 1436,nop,wscale 6,nop,nop,TS val 3259210450 ecr 0,sackOK,eol], length 0
IP 151.101.193.69.443 > TUNNEL_PUBLIC_IP.57324: Flags [S.], seq 421652706, ack 3399970171, win 65535, options [mss 1436,sackOK,TS val 443883721 ecr 3259210450,nop,wscale 9], length 0
IP TUNNEL_PUBLIC_IP.57324 > 151.101.193.69.443: Flags [.], ack 1, win 2069, options [nop,nop,TS val 3259210462 ecr 443883721], length 0
IP 151.101.193.69.443 > TUNNEL_PUBLIC_IP.57324: Flags [R], seq 421652707, win 0, length 0
IP TUNNEL_PUBLIC_IP.57325 > 151.101.193.69.443: Flags [S], seq 520414487, win 65535, options [mss 1436,nop,wscale 6,nop,nop,TS val 4096690292 ecr 0,sackOK,eol], length 0
IP 151.101.193.69.443 > TUNNEL_PUBLIC_IP.57325: Flags [S.], seq 1371963069, ack 520414488, win 65535, options [mss 1436,sackOK,TS val 2253923934 ecr 4096690292,nop,wscale 9], length 0
IP TUNNEL_PUBLIC_IP.57325 > 151.101.193.69.443: Flags [.], ack 1, win 2069, options [nop,nop,TS val 4096690304 ecr 2253923934], length 0
IP 151.101.193.69.443 > TUNNEL_PUBLIC_IP.57325: Flags [R], seq 1371963070, win 0, length 0
IP TUNNEL_PUBLIC_IP.57326 > 151.101.193.69.443: Flags [S], seq 3838108390, win 65535, options [mss 1436,nop,wscale 6,nop,nop,TS val 3121968079 ecr 0,sackOK,eol], length 0
IP 151.101.193.69.443 > TUNNEL_PUBLIC_IP.57326: Flags [S.], seq 1948995739, ack 3838108391, win 65535, options [mss 1436,sackOK,TS val 1474468012 ecr 3121968079,nop,wscale 9], length 0
IP TUNNEL_PUBLIC_IP.57326 > 151.101.193.69.443: Flags [P.], seq 1:155, ack 1, win 2069, options [nop,nop,TS val 3121968092 ecr 1474468012], length 154
IP TUNNEL_PUBLIC_IP.57326 > 151.101.193.69.443: Flags [.], ack 1, win 2069, options [nop,nop,TS val 3121968092 ecr 1474468012], length 0
IP 151.101.193.69.443 > TUNNEL_PUBLIC_IP.57326: Flags [.], ack 155, win 285, options [nop,nop,TS val 1474468024 ecr 3121968092], length 0
IP 151.101.193.69.443 > TUNNEL_PUBLIC_IP.57326: Flags [.], ack 155, win 285, options [nop,nop,TS val 1474468024 ecr 3121968092], length 0
IP 151.101.193.69.443 > TUNNEL_PUBLIC_IP.57326: Flags [P.], seq 1:8, ack 155, win 285, options [nop,nop,TS val 1474468024 ecr 3121968092], length 7
IP 151.101.193.69.443 > TUNNEL_PUBLIC_IP.57326: Flags [F.], seq 8, ack 155, win 285, options [nop,nop,TS val 1474468024 ecr 3121968092], length 0
IP TUNNEL_PUBLIC_IP.57326 > 151.101.193.69.443: Flags [F.], seq 155, ack 9, win 2069, options [nop,nop,TS val 3121968104 ecr 1474468024], length 0
IP 151.101.193.69.443 > TUNNEL_PUBLIC_IP.57326: Flags [.], ack 156, win 285, options [nop,nop,TS val 1474468038 ecr 3121968104], length 0
IP 151.101.193.69.443 > TUNNEL_PUBLIC_IP.57326: Flags [R], seq 1948995747, win 0, length 0
IP 151.101.193.69.443 > TUNNEL_PUBLIC_IP.57326: Flags [R], seq 1948995748, win 0, length 0
IP 151.101.193.69.443 > TUNNEL_PUBLIC_IP.57322: Flags [S.], seq 1998695891, ack 250459535, win 65535, options [mss 1436,sackOK,TS val 443884669 ecr 264874653,nop,wscale 9], length 0
IP TUNNEL_PUBLIC_IP.57322 > 151.101.193.69.443: Flags [R], seq 250459535, win 0, length 0
IP 151.101.193.69.443 > TUNNEL_PUBLIC_IP.57323: Flags [S.], seq 3935317826, ack 2696764620, win 65535, options [mss 1436,sackOK,TS val 926842097 ecr 291229040,nop,wscale 9], length 0
IP TUNNEL_PUBLIC_IP.57323 > 151.101.193.69.443: Flags [R], seq 2696764620, win 0, length 0
IP 151.101.193.69.443 > TUNNEL_PUBLIC_IP.57324: Flags [S.], seq 421652706, ack 3399970171, win 65535, options [mss 1436,sackOK,TS val 443884737 ecr 3259210462,nop,wscale 9], length 0
IP TUNNEL_PUBLIC_IP.57324 > 151.101.193.69.443: Flags [R], seq 3399970171, win 0, length 0
IP 151.101.193.69.443 > TUNNEL_PUBLIC_IP.57325: Flags [S.], seq 1371963069, ack 520414488, win 65535, options [mss 1436,sackOK,TS val 2253924964 ecr 4096690304,nop,wscale 9], length 0
IP TUNNEL_PUBLIC_IP.57325 > 151.101.193.69.443: Flags [R], seq 520414488, win 0, length 0
^C
50 packets captured
50 packets received by filter
0 packets dropped by kernel

PART 2/3

 

[HELLO_wORLD]

And this show the TCPMSS rules are being hit:

root@HERMES:~$ iptables -L -n -v -t mangle
Chain PREROUTING (policy ACCEPT 1583K packets, 133M bytes)
(…)           

Chain INPUT (policy ACCEPT 773K packets, 78M bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DROP       tcp  --  brwan  *       0.0.0.0/0            0.0.0.0/0            tcp flags:0x20/0x20

Chain FORWARD (policy ACCEPT 772K packets, 54M bytes)
 pkts bytes target     prot opt in     out     source               destination         
 109K 4822K TCPMSS     tcp  --  mw0    *       0.0.0.0/0            0.0.0.0/0            /* milkywan */ tcp flags:0x06/0x02 TCPMSS set 1436
    0     0 ACCEPT     tcp  --  mw0    *       0.0.0.0/0            0.0.0.0/0            tcpmss match 1437:1500
 106K 4692K ACCEPT     tcp  --  mw0    *       0.0.0.0/0            0.0.0.0/0            tcpmss match 1436
 8354 1868K ACCEPT     tcp  --  mw0    *       0.0.0.0/0            0.0.0.0/0           
 3271  408K ACCEPT     udp  --  mw0    *       0.0.0.0/0            0.0.0.0/0           
 1456  116K ACCEPT     all  --  mw0    *       0.0.0.0/0            0.0.0.0/0           
  131  8384 TCPMSS     tcp  --  *      mw0     0.0.0.0/0            0.0.0.0/0            /* milkywan */ tcp flags:0x06/0x02 TCPMSS set 1436
    0     0 ACCEPT     tcp  --  *      mw0     0.0.0.0/0            0.0.0.0/0            tcpmss match 1437:1500
  131  8384 ACCEPT     tcp  --  *      mw0     0.0.0.0/0            0.0.0.0/0            tcpmss match 1436
  383 49563 ACCEPT     tcp  --  *      mw0     0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     udp  --  *      mw0     0.0.0.0/0            0.0.0.0/0           
 1251 75060 ACCEPT     all  --  *      mw0     0.0.0.0/0            0.0.0.0/0           

Chain OUTPUT (policy ACCEPT 627K packets, 51M bytes)
(…)

Chain POSTROUTING (policy ACCEPT 1517K packets, 112M bytes)
 pkts bytes target     prot opt in     out     source               destination

A few more pieces of info on my config:

root@HERMES:~$ ip route
default via ISP_GATEWAY dev brwan
ISP_PUBLIC_IP_SUBNET/22 dev brwan  proto kernel  scope link  src MY_ISP_PUBLIC_IP
TUNNEL_IP_SUBNET/30 dev mw0  proto kernel  scope link  src MY_TUNNEL_IP
192.168.0.0/24 dev br0  proto kernel  scope link  src 192.168.0.1

root@HERMES:~$ ip route list table 100
default via REMOTE_SIDE_TUNNEL_IP dev mw0  proto static

root@HERMES:~$ ip rule
0:    from all lookup local
199:    from 192.168.0.7 lookup 100
200:    from 192.168.0.1 lookup main
32766:    from all lookup main
32767:    from all lookup default

What am I missing?

PART 3/3

 

[sfx2000]

Connection reset by peer

Consider what that error message means - the other end sent a RST packet - you've sent the packet, but the other end is not recognizing the connection...

Looks like a routing issue

 

[sfx2000]

Since I am sending several devices through the tunnel, and setting manually MTU for each one would be a pain, I came with this solution:

You shouldn't have to adjust MTU - Path MTU does this for you automatically end to end, from the faucet to the sink drain...

 

[sfx2000]

Handy tool to keep around...

https://www.speedguide.net/analyzer.php

 

[HELLO_wORLD]

Handy tool to keep around...

https://www.speedguide.net/analyzer.php

I will look into that tomorrow (it is late here in France…) thank you!

I agree the problem is a routing one, since the curl through the tunnel from the router is ok, but not from the LAN device (for which the router is forwarding to the tunnel).

Now, at this point, I spent hours trying to figure that one out, unsuccessfully so far.
I have a hard time to understand why some sites are giving this error, and not others.

 

[HELLO_wORLD]

From my ISP connection:

« SpeedGuide.net TCP Analyzer Results »
Tested on: 2022.08.09 06:39
IP address: 2.59.xxx.xxx
Client OS/browser: Mac OS (Safari 15.5)
 
TCP options string: 020405b4010303060101080a6b87bae80000000004020000
MSS: 1460
MTU: 1500
TCP Window: 131712 (not multiple of MSS)
RWIN Scaling: 6 bits (2^6=64)
Unscaled RWIN : 2058
Recommended RWINs: 64240, 128480, 256960, 513920, 1027840
BDP limit (200ms): 527 Mbps (53 Megabytes/s)
BDP limit (500ms): 211 Mbps (21 Megabytes/s)
MTU Discovery: ON
TTL: 52
Timestamps: ON
SACKs: ON
IP ToS: 00000000 (0)

From the tunnel:

« SpeedGuide.net TCP Analyzer Results »
Tested on: 2022.08.09 06:42
IP address: 45.13.xxx.x
Client OS/browser: Mac OS (Safari 15.5)
 
TCP options string: 0204059c010303060101080ac5e47cbf0000000004020000
MSS: 1436
MTU: 1476
TCP Window: 132416 (not multiple of MSS)
RWIN Scaling: 6 bits (2^6=64)
Unscaled RWIN : 2069
Recommended RWINs: 63184, 126368, 252736, 505472, 1010944
BDP limit (200ms): 530 Mbps (53 Megabytes/s)
BDP limit (500ms): 212 Mbps (21 Megabytes/s)
MTU Discovery: ON
TTL: 50
Timestamps: ON
SACKs: ON
IP ToS: 00000000 (0)

 

[R. Gerrits]

If you do the curl on the router, does that traffic even flow though the forward chain? I thought it would use the output chain.
And in that case you are only changing tcpmss for the LAN traffic and not the router traffic.

So this might explain the different behaviour between LAN and router.

 

[HELLO_wORLD]

If you do the curl on the router, does that traffic even flow though the forward chain? I thought it would use the output chain.
And in that case you are only changing tcpmss for the LAN traffic and not the router traffic.

So this might explain the different behaviour between LAN and router.

It should not hit the FORWARD chain when I do it from the router but as you rightfully say the INPUT/OUTPUT ones.
I did have the TCPMSS rules for INPUT and OUTPUT before. I just put them back:

iptables -t mangle -w -I OUTPUT -o mw0 -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1436
iptables -t mangle -w -I INPUT -i mw0 -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1436

Same result:
– success from router: curl -sv --interface mw0 https://serverfault.com >/dev/null
– failure from LAN device

Also, if I change --set-mss 1436 by --clamp-mss-to-pmtu in FORWARD (and also in INPUT and OUTPUT), the results are exactly the same.