What are the Real-World LAN Transfer Speeds in MB/s for 2.5Gbe?

[kelllogg9]

Title says it all.
At the moment i am only getting between 170MB/s to 217MB/s which seems small. I would have expected the real-world to be between 298-305.

 

[thiggins]

Assuming Ethernet, both ends of your test connection must have fast machines and hard drives can't be involved at either end, i.e. memory to memory.

How are you testing and with what protocol (TCP/UDP)?

 

[kelllogg9]

iperf3, tcp, and udp, and yes its via mem-to-mem.
What are ur speed ranges in MB/s?

 

[thiggins]

Please show your iperf3 command. Are you using multiple connections?

I have no data handy. But assuming % of overhead stays constant (and I'm not sure it does), max TCP/IP rate for 1 GbE is 943 Mbps. 2.5x that is 2358 Mbps . 294 MB/s.

 

[ColinTaylor]

2.37Gb/s

Be careful when looking at Windows' "MB/s" numbers because Windows uses base 1024 whereas Ethernet data rates are base 10 (i.e. 2,500,000,000 bits/s). Plus of course the usual overheads for whatever protocol you're using.

2.5GbE Networking on Linux -- Virtualization Review

After detailing using 2.5GbE NICs in his home lab on Windows 10, Tom Fenton does the same thing on Linux.

virtualizationreview.com

2.5GbE Networking: Setup and 1GbE Performance Comparison -- Virtualization Review

Tom Fenton details setting up and using a 2.5GbE NIC and compares its performance to a 1GbE NIC, finding 'The cost is the bugaboo in the equation.'

virtualizationreview.com

 

[TheLostSwede]

The target it my DIY OMV NAS with a 10Gbps Aquantia card in it and regular old spinning rust, using SnapRAID.

Realtek 2.5Gbps

Aquantia throttled to 2.5Gbps.

Aquantia at 10Gbps.

 

[TheLostSwede]

iperf3, single workload, Realtek 2.5Gbps to Aquantia 10Gbps. Four parallel workloads deliver the same overall throughput.

C:\iperf3>iperf3 -c 192.168.1.250
Connecting to host 192.168.1.250, port 5201
[ 4] local 192.168.1.27 port 55539 connected to 192.168.1.250 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 282 MBytes 2.37 Gbits/sec
[ 4] 1.00-2.00 sec 277 MBytes 2.32 Gbits/sec
[ 4] 2.00-3.00 sec 276 MBytes 2.31 Gbits/sec
[ 4] 3.00-4.00 sec 276 MBytes 2.31 Gbits/sec
[ 4] 4.00-5.00 sec 272 MBytes 2.28 Gbits/sec
[ 4] 5.00-6.00 sec 275 MBytes 2.31 Gbits/sec
[ 4] 6.00-7.00 sec 275 MBytes 2.31 Gbits/sec
[ 4] 7.00-8.00 sec 275 MBytes 2.31 Gbits/sec
[ 4] 8.00-9.00 sec 278 MBytes 2.33 Gbits/sec
[ 4] 9.00-10.00 sec 279 MBytes 2.34 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 2.70 GBytes 2.32 Gbits/sec sender
[ 4] 0.00-10.00 sec 2.70 GBytes 2.32 Gbits/sec receiver

iperf Done.

 

[L&LD]

So, in MB/s.

2.37 gigabits / second = 296.25 megabytes / second

Gbps to MB/s Converter (Gigabit per second to Megabyte per second) (gbmb.org)

Less the ~2.8% TCP/IP overhead = 287.955 MB/s

Or, with 'efficiency' at ~97.33% = 288.34 MB/s

Or, with Ethernet, VLANs and AES,
the 'efficiency' is at ~94.68% = 280.49 MB/s

The above assumes you're maximizing the data payload for each packet.

This also assumes a local network setup too. With HTTP, for instance, you can lose
an additional 25% more (see quote below). This is a good reason to get maximum
ISP connection speeds, even if you think you don't need it (but still want to minimize
your required, online time).

TCP Over IP Bandwidth Overhead - Packet Pushers

Summary​

So, as demonstrated, for data payloads in excess of the common TCP payload maximum segment size (the MSS) of 1460 Bytes, the TCP over IP bandwidth overhead is approximately 2.8%. This equates to an ‘efficiency’ of 97.33% (1460/1500) – in other words, that’s how much bandwidth is left for actual data if you’re putting as much data in each packet as possible.

However, keep in mind that for very small data payloads (common with applications such as Telnet, TN3270 mainframe emulation and SSH) the TCP over IP bandwidth overhead can be higher than 4,000%.

If you add Ethernet (and VLAN tagging) into the mix (see the calculations from Wikipedia here) then the throughput of a 100Mb link is 100 X 0.9733 (TCP/IP efficiency) x 0.9728 (Ethernet (with tagging) efficiency) which equals 94.68Mbps, which I assume means the combined efficiency is 94.68%.

Add in your security protocol (AES would drop the figure to 87.7%), session and application overheads and you start to see where all your precious bandwidth is going, if it is precious. If you’re not filling every packet to the brim you can imagine the loss can get even higher pretty quick.

With HTTP for instance, a large number of headers, large cookie values, multiple cookies and the like can literally consume a further 25% of your bandwidth without you even trying, or realising.

If you are looking for an argument as to why you should use Jumbo frames in a LAN environment, I don’t think there is one; still, like I said, in some circles ever % matters.