New motherboard with 10Gbe and only getting max 5Gb/s

[Pauluk]

Hi all

So recently purchased a new Asus motherboard x670e , it has a 10Gbe port on it, as did my previous Asrock motherboard x570. I`ve got it connected directly to another PC with a Asus 10Gbe PCI card , and you can see the iperf results below.
Main use is to connect to my Qnap NAS 10Gbe port via a 10gbe switch, which when running tests previously on my old motherboard i`d get around 10Gb/s. But on this setup i`m struggling to get 3Gb/s to my NAS via the switch.
I`ve tried uninstalling/reinstalling drivers and pretty much disabling and enabling everything in the adapters (Marvell) advanced settings. But nothing improves the speed. I have forced it to connect at 10Gb in there aswell.
Tried disabling Rec Segment Coalescing and Energy efficient Ethernet but no difference. I`ve actually gone down the whole list disabling things.

The only slightly better result i can get is i setup a dual boot to windows 10 (i`m using windows 11) , and I would get around 5Gb/s (instead of 3Gb/s to the NAS) But still no where near the 10Gb/s i`d expect.

Any ideas? Thanks

 

[Tech Junky]

Start at layer 1 which is the cable. Once you rule that out move up the OSI stack.

 

[L&LD]

Did you try new cables?

 

[Pauluk]

Start at layer 1 which is the cable. Once you rule that out move up the OSI stack.

Yes tried two cables cat 6e and 7.

 

[Pauluk]

Did you try new cables?

Yes as mentioned above. Also I did work fine before with the same cables and setup just new motherboard has changed.

 

[drinkingbird]

Yes as mentioned above. Also I did work fine before with the same cables and setup just new motherboard has changed.

Onboard NICs are not known for being stellar. During your iperf, go into windows perfmon (on both PCs) and look at all your CPU cores, see if one is pegged at 100%. Task manager only shows you an average of all cores so you have to go into perfmon. A single iperf is not multi threaded, unless they've changed something. You can test two simultaneous iperfs and see if it spreads across two cores and you get a higher total throughput.

10G/s of "dummy" packets is a lot for a PC to generate, then add onto that the onboard NIC may not have a great offloading chipset and relies on the main CPU.

I'm assuming you've tried the latest NIC drivers. Start with whatever Asus has on their site, and if that doesn't improve, try the ones from Intel directly (if they're newer).

Have you previously tested the other PC? What card is in that - could be hitting a PCIe limit depending on the revision and number of lanes on that card. Maybe that is what is limiting your test and not the new MB?

When you got the new motherboard I'm assuming you reinstalled the OS so check and see if maybe something else changed, you used a newer version of firewall/antivirus, etc. Try disabling your FW/AV including windows defender and see if throughput gets better (and/or CPU use gets lower).

EDIT sorry just noticed the 10G is Marvell not Intel. They're not exactly known for their NICs... - is that the same brand that was on the old mobo?

 

[Pauluk]

Onboard NICs are not known for being stellar. During your iperf, go into windows perfmon (on both PCs) and look at all your CPU cores, see if one is pegged at 100%. Task manager only shows you an average of all cores so you have to go into perfmon. A single iperf is not multi threaded, unless they've changed something. You can test two simultaneous iperfs and see if it spreads across two cores and you get a higher total throughput.

10G/s of "dummy" packets is a lot for a PC to generate, then add onto that the onboard NIC may not have a great offloading chipset and relies on the main CPU.

I'm assuming you've tried the latest NIC drivers. Start with whatever Asus has on their site, and if that doesn't improve, try the ones from Intel directly (if they're newer).

Have you previously tested the other PC? What card is in that - could be hitting a PCIe limit depending on the revision and number of lanes on that card. Maybe that is what is limiting your test and not the new MB?

When you got the new motherboard I'm assuming you reinstalled the OS so check and see if maybe something else changed, you used a newer version of firewall/antivirus, etc. Try disabling your FW/AV including windows defender and see if throughput gets better (and/or CPU use gets lower).

EDIT sorry just noticed the 10G is Marvell not Intel. They're not exactly known for their NICs... - is that the same brand that was on the old mobo?

Thanks for that.

I tried perfmon , % Processor Time, and there is one core (out of 32) that does max out for a few seconds then drops away an another one maxes out, then after about 30secs they all drop to around 30% or lower and stabilise.

I check my NAS and that has 4 cores, 3 are around 20% and one is 60% when iperf is running.

Tried the latest drivers, from Asus and also Marvell. I also went back one version to see if that helped but no different.

The PCi card is an Asus https://www.asus.com/supportonly/xg-c100c/helpdesk_download/?model2Name=XG-C100C
Same brand as my motherboard on both PC`s.

I tested this in my main PC and then put it into another PC and connected mine and this together with a CAT 7 cable and ran iperf.
When I installed the new motherboard I did re-use the current Windows 11, which I did have some issues with drivers at first but seems to have settled down now. I did install a Windows 10 dual boot clean and installed the same drivers onto that. That gives a better performance but still about 5-6Gb/s. That might have defender enabled by default but apart from that its a clean install.

I tried contacting Asus and they were completely unhelpful, suggestion I go back to where I purchased from for warranty. I do still have my old Asrock motherboard but it pains me to reinstall that to check that works ok.

 

[drinkingbird]

Thanks for that.

I tried perfmon , % Processor Time, and there is one core (out of 32) that does max out for a few seconds then drops away an another one maxes out, then after about 30secs they all drop to around 30% or lower and stabilise.

I check my NAS and that has 4 cores, 3 are around 20% and one is 60% when iperf is running.

Tried the latest drivers, from Asus and also Marvell. I also went back one version to see if that helped but no different.

The PCi card is an Asus https://www.asus.com/supportonly/xg-c100c/helpdesk_download/?model2Name=XG-C100C
Same brand as my motherboard on both PC`s.

I tested this in my main PC and then put it into another PC and connected mine and this together with a CAT 7 cable and ran iperf.
When I installed the new motherboard I did re-use the current Windows 11, which I did have some issues with drivers at first but seems to have settled down now. I did install a Windows 10 dual boot clean and installed the same drivers onto that. That gives a better performance but still about 5-6Gb/s. That might have defender enabled by default but apart from that its a clean install.

I tried contacting Asus and they were completely unhelpful, suggestion I go back to where I purchased from for warranty. I do still have my old Asrock motherboard but it pains me to reinstall that to check that works ok.

100% for 30 seconds is your culprit (assuming that is happening during the iperf run). 10 gigs is a lot for anything to handle, you can't just accept it as a given that it will always hit 10G no matter what you throw at it. Iperf is particularly CPU intensive as it has to generate 10G/s worth of fake packets and then the network card has to transmit them.

Most likely you'll see close to 10G of aggregate performance if you run multiple parallel transfers and it takes advantage of multiple CPU cores.

If you really want 10G performance nearly all the time, look for a server grade NIC from Intel, which will have its own onboard ASIC and processor. Doesn't help with the part about iperf having to generate a lot of packets but offloads the majority of the networking from the main CPU.

We're basically back where we were when 1G started becoming common. I have an old laptop from 2008 or so that can't come anywhere close to 1G on iperf, but my desktop with an intel server NIC in it can do it with barely any effort. Newer laptop is in between, can hit 1G no problem but CPU up around 40-50% on one core.

New motherboard = new install of windows. You will have ongoing strangeness if not. Not saying that is the reason for not hitting 10G, but it may be part of it.

 

[Pauluk]

100% for 30 seconds is your culprit (assuming that is happening during the iperf run). 10 gigs is a lot for anything to handle, you can't just accept it as a given that it will always hit 10G no matter what you throw at it. Iperf is particularly CPU intensive as it has to generate 10G/s worth of fake packets and then the network card has to transmit them.

Most likely you'll see close to 10G of aggregate performance if you run multiple parallel transfers and it takes advantage of multiple CPU cores.

If you really want 10G performance nearly all the time, look for a server grade NIC from Intel, which will have its own onboard ASIC and processor. Doesn't help with the part about iperf having to generate a lot of packets but offloads the majority of the networking from the main CPU.

We're basically back where we were when 1G started becoming common. I have an old laptop from 2008 or so that can't come anywhere close to 1G on iperf, but my desktop with an intel server NIC in it can do it with barely any effort. Newer laptop is in between, can hit 1G no problem but CPU up around 40-50% on one core.

New motherboard = new install of windows. You will have ongoing strangeness if not. Not saying that is the reason for not hitting 10G, but it may be part of it.

Sorry no it’s only a couple of seconds then it’s down to 30%. Then if I restart it’s just constantly at 30%.

I think I’ll move over fully to the new install, just was trying not to have to reinstall all my software which will take me days to move across.

Thing I find strange is that I’m using same setup apart from new motherboard and although I haven’t checked recently I was getting a full 10gb/s up and down with my old motherboard. But now I’ve tried a PCIe board and getting similar poor performance I’m confused.
Thanks

 

[Tech Junky]

Task manager only shows you an average of all cores

If you right click in the CPU section it will break them out into individual cores/threads.

NIC from Intel

The aquantia ones do just as well for half the price.

run multiple parallel transfers

I see this on data for TB 4 for my enclosure and drive testing. ST maxes around 700MB/s where multiple hits the expected 3.1GB/s.

New motherboard = new install of windows.

Saw this on another thread for a 2.5 nic that wasn't hitting it's potential by 500mbps and a reinstall for it to where it should be. Windows does some crappy stuff when it comes to certain things.

Just for sanity sake I would boot into Linux on both sides and test with no MSFT interference. This at least rules out the hardware.

 

[drinkingbird]

Sorry no it’s only a couple of seconds then it’s down to 30%. Then if I restart it’s just constantly at 30%.

I think I’ll move over fully to the new install, just was trying not to have to reinstall all my software which will take me days to move across.

Thing I find strange is that I’m using same setup apart from new motherboard and although I haven’t checked recently I was getting a full 10gb/s up and down with my old motherboard. But now I’ve tried a PCIe board and getting similar poor performance I’m confused.
Thanks

The onboard NIC is a PCIe also. PCIe doesn't mean much, it is the chipset on the board (whether onboard or addin) itself. Maybe that MB has issues with the PCIe bus or bridge to CPU or storage etc too.

 

[Pauluk]

Saw this on another thread for a 2.5 nic that wasn't hitting it's potential by 500mbps and a reinstall for it to where it should be. Windows does some crappy stuff when it comes to certain things.

Just for sanity sake I would boot into Linux on both sides and test with no MSFT interference. This at least rules out the hardware.

I did think of this, I looked into trying to do a linux usb boot up but I got lost down some rabbit hole. Any guides on how to do this without too much time/effort.
thanks

 

[drinkingbird]

If you right click in the CPU section it will break them out into individual cores/threads.

Yeah I like/am used to perfmon but changing task manager to "logical processors" works too.

The aquantia ones do just as well for half the price.

There are several decent server class NICs out there, I've found you can get intel stuff used fairly cheap often times but obviously it isn't the only choice. Again I'm just used to theirs and setting up VLANs and Teaming/LAG is very easy to do etc. Even broadcom and realtek have made some pretty good onboard and add in NICs (but they can really be hit or miss).

 

[drinkingbird]

I did think of this, I looked into trying to do a linux usb boot up but I got lost down some rabbit hole. Any guides on how to do this without too much time/effort.
thanks

Just go to ubuntu.com and download the usb installer. During boot, you choose whether to install or just boot from it. Very easy. Stick with 22.04 desktop version. Very stable.

 

[Tech Junky]

I did think of this, I looked into trying to do a linux usb boot up but I got lost down some rabbit hole. Any guides on how to do this without too much time/effort.
thanks

Just grab a copy of Rufus and burn to the USB. Rufus can download the ISO withing itself or you can grab one yourself from vendor websites as well.

 

[Tech Junky]

Teaming/LAG

Or bonding and bridging. RTL is my onboard port that requires blacklisting for the 2.5 driver module inorder to use the correct one.

As for brands it's price and preference. Also, I run 5ge and the quad port card was $200 vs Intel at a much higher stocker price. For 10ge though it's debatable on which performs better and has lower thermals.

 

[Pauluk]

Just grab a copy of Rufus and burn to the USB. Rufus can download the ISO withing itself or you can grab one yourself from vendor websites as well.

Thanks i gave this a go. So running a speedtest on my Qnap I was getting around 6Gb/s download and 2.5Gb/s upload. So similar to the windows 10 build i tried but a slightly slower upload.
So to recap have tried my Asus board to Qnap with 10GBe, and also to another PC with an Asus 10GBe pci card (direct connection). Best I can get is Windows 10 clean build with 6Gb/4Gb down/up speeds.

 

[drinkingbird]

Thanks i gave this a go. So running a speedtest on my Qnap I was getting around 6Gb/s download and 2.5Gb/s upload. So similar to the windows 10 build i tried but a slightly slower upload.
So to recap have tried my Asus board to Qnap with 10GBe, and also to another PC with an Asus 10GBe pci card (direct connection). Best I can get is Windows 10 clean build with 6Gb/4Gb down/up speeds.

Seems reasonable for an inexpensive 10Gbe desktop class card. Keep in mind your hardware (not just the NIC, everything) has to be capable as well.

You may see better speeds by doing a large file transfer which does not require the devices to generate large amounts of dummy data.

 

[Pauluk]

Seems reasonable for an inexpensive 10Gbe desktop class card. Keep in mind your hardware (not just the NIC, everything) has to be capable as well.

You may see better speeds by doing a large file transfer which does not require the devices to generate large amounts of dummy data.

Yeah thanks but thing is i use to get 10gb/s with my old Asrock x570 motherboard, so i`m not sure whats happened since. Although i havent checked speeds for about a year so could be something unrelated.

 

[drinkingbird]

Yeah thanks but thing is i use to get 10gb/s with my old Asrock x570 motherboard, so i`m not sure whats happened since. Although i havent checked speeds for about a year so could be something unrelated.

Newer doesn't always equal better (in fact these days with all the chip shortages from the last year, often quite the opposite). Look at Samsung, their 970 Evo Plus SSD which had killer benchmarks, they couldn't get the controller, swapped it with another brand, and did some trickery with cache to make it look like the performance was similar, but then when doing a large transfer it would fall off sharply partway through.