Troubleshooting 2.5 Gbps Network Speed Issue with NAS and RT-AX86U Router

[Stylishh1]

I have a RT-AX86U router with one 2.5 Gbps port that I can configure either for WAN (internet) or LAN (internal network). In my setup, I want to connect it to a NAS server with Proxmox, which also has a 2.5 Gbps network card (FENVI 2.5G PCI-E) and SSD drives for faster file transfer. And that's exactly what I did.

When I ran tests with iperf3, it turned out that the connection only reaches a maximum of 1.46 Gbps, which is unsatisfactory to me.

Troubleshooting:

• I assumed it was the fault of the Chinese network card in the NAS server, so I ordered the TX201 TP-Link, but the same issue occurred.
• I tested the FENVI 2.5G PCI-E card from the NAS server in another computer running Windows, but the same issue happened.
• I installed Windows 11 on the NAS server, and again, the same problem.
• I replaced the cables with other CAT5e cables (tested 5 of them), but the same issue.
• I ordered 3 flat CAT7 cables, and once again, the same issue.

I’m now at a loss. On the router, I can see that the port is operating at 2.5 Gbps. On the NAS server, when I run the command to check the speed ethtool enp1s0 | grep -i speed, I get the result: Speed: 2500Mb/s

I’m really stuck and don’t know what to do next.

 

[bbunge]

You may have discovered, as I have, that the 2.5 Gb port on the AX86U is less than optimal. Same with the 2.5Gb port on the AX86U Pro. I have tried to use those ports for Ethernet backhaul with tested CAT 5e cables and, in time, have had the connection fail. Even connecting the 2.5 Gb port to a 1 Gb port has caused connection issues never mind the data transfer rate.
So, best of luck with your endevors. I doubt that you will find a solution. May be better off to stick with using 1 Gb as this is sufficient for most home networks.

 

[ColinTaylor]

Running the iperf server on the router is not a valid test because iperf will become CPU limited. You need to test LAN to LAN transfers, which admittedly is quite difficult as you'd have to use multiple clients and aggregate the results.

You will also get better results from iperf if you use parallel streams, e.g. iperf3 -c 192.168.1.1 -t 120 -P 8

 

[degrub]

And if running iperf on windows, you should be using iperf 2 according to the developers. On linux and unix, iperf3 is appropriate.
Yes, you must use parallel streams to saturate the link.
You could also avoid the router all together by using a 2.5 GHz switch in place of the asus port. This would avoid loading up the router switch as well and is more likely to give you packet transfer at line speed. It would give all your devices a 2.5 GHz uplink to the router as well, although a new bottleneck will show up.

 

[Tech9]

for faster file transfer

Going where? Aggregate traffic only to multiple devices with this single 2.5GbE port router.

 

[SwampKracker]

My fastest iperf result:

user:~$ iperf -c 10.10.10.4 -w 416K -P 4
------------------------------------------------------------
Client connecting to 10.10.10.4, TCP port 5001
TCP window size: 416 KByte
------------------------------------------------------------
[ 1] local 10.10.10.2 port 34304 connected with 10.10.10.4 port 5001 (icwnd/mss/irtt=14/1448/469)
[ 4] local 10.10.10.2 port 34334 connected with 10.10.10.4 port 5001 (icwnd/mss/irtt=14/1448/127)
[ 3] local 10.10.10.2 port 34320 connected with 10.10.10.4 port 5001 (icwnd/mss/irtt=14/1448/465)
[ 2] local 10.10.10.2 port 34312 connected with 10.10.10.4 port 5001 (icwnd/mss/irtt=14/1448/463)
[ ID] Interval Transfer Bandwidth
[ 4] 0.0000-10.0129 sec 27.8 GBytes 23.9 Gbits/sec
[ 3] 0.0000-10.0130 sec 25.0 GBytes 21.4 Gbits/sec
[ 2] 0.0000-10.0129 sec 25.5 GBytes 21.9 Gbits/sec
[ 1] 0.0000-10.0129 sec 26.1 GBytes 22.4 Gbits/sec
[SUM] 0.0000-10.0001 sec 104 GBytes 89.7 Gbits/sec

 

[degrub]

Try -P 10 or higher until it saturates.

 

[tgl]

Running the iperf server on the router is not a valid test because iperf will become CPU limited.

Exactly. The CPU inside that router does not have enough oomph to run iperf3 at 2.5Gbps. This is true not only of most Asus gear but most dedicated network hardware from any manufacturer. They're made so that the high-speed data paths go through specialized ASICs and never hit the CPU at all. This lets the maker get away with a cheaper CPU.

As said, if you want to test a scenario with a single iperf3 run reaching 2.5Gbps, you need a 2.5G-capable switch and at least one more 2.5G-capable end device. But a more realistic test given the gear you have is two or three 1G end devices concurrently running iperf3 to multiple iperf3 instances on the NAS. Can you approach 2.5G total throughput that way?

 

[Digilog]

Apparently its still set to the generic interface settings in the router.
try

ip link set enp1s0 mtu 2500 txqueuelen 2500

if it can handle jumbo frames:

ip link set enp1s0 mtu 9000 txqueuelen 5000

 

[Digilog]

The CPU inside that router does not have enough oomph to run iperf3 at 2.5Gbps.

it actually has nothing to do with the cpu. The first culprit would be soft irq buffers not being big enough as both Linux and BSD have this set low. So you should see a limitation somewhere around 1.75Gb/s passing it across net interfaces unless they have modified it. But that is If it was truly the hardware slowing it down instead of interface settings.

 

[ColinTaylor]

@Digilog Please don't post nonsense about a device you clearly know nothing about.

 

[Digilog]

@Digilog Please don't post nonsense about a device you clearly know nothing about.

Well then they can stop using Linux OS then.

There are only two places it is not optimized (even the BSD spin off) in networking: interface buffers and memory allocation of buffers and software irq buffers,

But I'll let you guys flop like a fish out of water since you want to insult a Linux Dev.

Second look at what is going on, the parameter in netdev_budget should be increased to 600 in /etc/sysctl.conf if setting the buffer on the interface is not the issue. Because routing from one interface to another uses irq software buffers from ram and not the cpu.

 

[ColinTaylor]

Well then they can stop using Linux OS then.

There are only two places it is not optimized (even the BSD spin off) in networking: interface buffers and memory allocation of buffers and software irq buffers,

But I'll let you guys flop like a fish out of water since you want to insult a Linux Dev.

Second look at what is going on, the parameter in netdev_budget should be increased to 600 in /etc/sysctl.conf if setting the buffer on the interface is not the issue. Because routing from one interface to another uses irq software buffers from ram and not the cpu.

Please stop. You're just digging a deeper hole for yourself. It's embarrassing.

 

[Digilog]

Please stop. You're just digging a deeper hole for yourself. It's embarrassing.

your lack of understanding is not my concern.

as if

 

[Digilog]

You could also avoid the router all together by using a 2.5 GHz switch in place of the asus port. This would avoid loading up the router switch as well and is more likely to give you packet transfer at line speed. It would give all your devices a 2.5 GHz uplink to the router as well, although a new bottleneck will show up.

This is the method you guys should wire these network: Router (at fastest lan port) -> switch . If you are networking between computers and want performance over 1.75gb. This presumes that the router is built like a gateway server instead of how the cheap routers are built with a WAN interface and a LAN interface with a hardware switch to many LAN ports. (which ironically would never have this networking bandwidth issue)

Rarely a gateway server has multiple ports configured as a switch for a network, and in that situation, they are usually ran to different areas with switches on them and then wires connected between these switches to service inside networking.

 

[Tech9]

gateway server

What is a gateway server??