A very strange problem with GT-AX6000 and QNAP QSW-1105-5T switch

[L&LD]

It's your USB 2.5GbE Ethernet adaptor that is messing things up for you.

 

[Moan]

It's your USB 2.5GbE Ethernet adaptor that is messing things up for you.

All right, but how? It's easy to blame some device, but the entire problem makes no sense. This adapter is not attached to the router. The switch / the router should know exactly where the traffic is coming from/to. Even if it was a 40 years old Ethernet port in some antic PC, the switch and the router should still be able to know the right direction.

As I mentioned in my first post, I didn't notice this issue when I was using my old router (which also had the 2.5Gb LAN port). This is why I'm trying to understand what's wrong before judging anything.

In previous answers I read that it's the switch creating MAC tables and being responsible for forwarding. So why would the USB Ethernet adapter be the problem with it? Is switching depended on adapter drivers or not?

 

[MDM]

USB 2.5GbE Ethernet adapter, could: go into some power save mode (or the USB port!), be changing the MAC address each time, something 3rd...?

 

[Moan]

Power saving is disabled, the MAC address doesn't change. What I noticed on this switch is that after power on, the network (on PC) appears for 1-2 seconds, then disappears and appears again after 2 seconds. It's like this switch is reconnecting devices twice after reboot? Or maybe it's when my PC reaches the router.

A perfect solution would be forcing to re-check devices ports on the network. Is there any Asus function that could simulate the same behavior as the switch does after restart? Or it's the switch that would only be able to do it?

 

[L&LD]

I don't know how, but it should be easy for you to check if it is doing that. Remove it from the equation and test.

Nothing else on the network can cause this issue as you've described so far. Therefore, it must be the unknown (the USB adaptor).

 

[ColinTaylor]

On the New NAS I'm also using Pihole, Nextcloud, Wireguard (obviously I also tested with disabled wireguard after fresh reboot).

Are you sure the NAS isn't trying to send data to the internet when you have this problem thereby reducing the bandwidth to your PC?

The original Ethernet port (1Gb/s) from the New NAS is unplugged, and this device has no Wi-Fi at all.

And here's a screenshot when the Switch bug is not present:
So easily getting 2.5Gb/s then.

In your "working" screenshot you were getting 1.59Gb/s from the router and only 884Mb/s from the New NAS. Try removing the USB adapter from the New NAS and use the original 1GbE port. Then test again and see if you get consistent results. Even though the New NAS would only have 1GbE you should still be able to hit 2.5Gb/s in combination with the router on your PC.

 

[Moan]

Are you sure the NAS isn't trying to send data to the internet when you have this problem thereby reducing the bandwidth to your PC?

No, my Internet upload speed is only 100Mb/s, so that would be way slower. It just goes to the Asus router and then back to the PC as a local connection transfer. That NAS is unreachable from Internet anyway.

In your "working" screenshot you were getting 1.59Gb/s from the router and only 884Mb/s from the New NAS. Try removing the USB adapter from the New NAS and use the original 1GbE port. Then test again and see if you get consistent results. Even though the New NAS would only have 1GbE you should still be able to hit 2.5Gb/s in combination with the router on your PC.

Sorry, that screenshot was misleading and had result of a different test performed before. I'm getting over 2 Gb/s from the New NAS (as in the screenshot from the post above): https://www.snbforums.com/attachments/iperf3-png.51771/

I'm not going to revert back to the 1Gb/s port though. It's not like I'm performing simultaneous transfers 100% of the time. I just wanted to have everything working perfectly since I got the new router and started testing how things work.

I think I managed to improve my "workaround" script. I will post later the updated version, so maybe someone with the same issues will find it useful.
What I'm generally doing is:
- Detecting if eth1 is down and setting a variable to true.
- When the eth1 becomes "up" again I'm calling the USB unbind/bind function in the driver to reset it. This usually doesn't work, but has effect at the second step...
- After 90 seconds of connection, I'm calling the "make install" function in the driver source folder. It performs rmmod, unattaches the USB, removes the .so driver file, copying the driver again and is calling modprobe. The entire process takes just 1-2 seconds, and the adapter becomes connected again with the fixed Switch bug.

So I think the key was to reconnect a device with 90+ seconds delay after the Switch is rebooted. This gives me a question if the Switch or Router itself is not an issue. It simply seems to not perform handshakes correctly on restart.

Anyway, I'll post the workaround script when I test it a bit longer that it fixes it in 100% of cases.
To be honest, the Switch restarts are rare. I just noticed it these days due to stormy weather, and I usually plug off the power strip to make sure nothing bad happens It's more like I want to have everything working perfectly and fully automated.

By the way, after the Switch power cycle, the Asus log is the following:

Jul 17 08:47:16 kernel: eth5 (Int switch port: 6) (Logical Port: 6) (phyId: 13) Link DOWN.
Jul 17 08:47:16 kernel: br0: port 5(eth5) entered disabled state
Jul 17 08:47:27 kernel: eth5 (Int switch port: 6) (Logical Port: 6) (phyId: 13) Link Up at 2500 mbps full duplex
Jul 17 08:47:27 kernel: br0: port 5(eth5) entered blocking state
Jul 17 08:47:27 kernel: br0: port 5(eth5) entered listening state
Jul 17 08:47:29 kernel: br0: port 5(eth5) entered learning state
Jul 17 08:47:31 kernel: eth5 (Int switch port: 6) (Logical Port: 6) (phyId: 13) Link DOWN.
Jul 17 08:47:31 kernel: br0: port 5(eth5) entered disabled state
Jul 17 08:47:34 kernel: eth5 (Int switch port: 6) (Logical Port: 6) (phyId: 13) Link Up at 2500 mbps full duplex
Jul 17 08:47:34 kernel: br0: port 5(eth5) entered blocking state
Jul 17 08:47:34 kernel: br0: port 5(eth5) entered listening state
Jul 17 08:47:36 kernel: br0: port 5(eth5) entered learning state
Jul 17 08:47:38 kernel: br0: port 5(eth5) entered forwarding state
Jul 17 08:47:38 kernel: br0: topology change detected, propagating

So as I noticed earlier. The QNAP QSW-1105-5T is making two connects/disconnects on restart. @L&LD could you please check if it's the same for you after you power on/off this switch? Maybe the problem is actually that it's being done twice on reboot. If it doesn't happen to you, then maybe my QNAP switch is from some bad revision with the bug, and I would need to send it for warranty...

 

[ColinTaylor]

I'm not going to revert back to the 1Gb/s port though. It's not like I'm performing simultaneous transfers 100% of the time. I just wanted to have everything working perfectly since I got the new router and started testing how things work.

I wasn't suggesting you permanently change to using the 1GbE port. I was only suggesting it as a temporary diagnostic step to try and determine whether the problem is specific to using the USB adapter.

 

[Moan]

I wasn't suggesting you permanently change to using the 1GbE port. I was only suggesting it as a temporary diagnostic step to try and determine whether the problem is specific to using the USB adapter.

Thank you for the idea! I was so focused on getting the 2.5Gb speeds that I forgot to verify other things.

So I connected the Switch to the Original (1Gb/s) Ethernet port in the Odroid device. The 2.5Gb was unplugged. I set a different IP address for that adapter (192.168.1.136 instead of 192.168.1.135).
Seems like the bug is exactly the same!

As you can see, when the transfer is paused, I'm getting ~1.7Gb/s from the router's iperf.

When the transfer is resumed, I'm still getting the same maximum speed:

So it's definitely not a problem with the USB Ethernet adapter, nor the NAS IP address... The USB adapter and the original Ethernet ports are using totally different network drivers as well.

I suppose it might be something wrong in Debian itself, stupid Switch, or Asus doing something weird.


EDIT:
Can somebody who has the same QNAP QSW-1105-5T switch please confirm that after power off / on, there is a double message in the Asus Router's log?

EDIT:
1 more thing why I'm 100% sure that the transfer goes by the Asus router: New Nas -> Switch -> Router -> Switch -> My PC:

I started file transfer with the bugged speed, and then I unplugged the Ethernet cable from the switch to the router... And my PC instantly started transferring the file with the higher speed. After I connected the Asus router again (and Iperf), I got the full 2.5Gb/s in the Task Manager.

I wonder if there's some possibility on Merlin, to delay actions made after "Link Up" is detected? Maybe if I added some little sleep there, it would help?

 

[ColinTaylor]

Were those two tests done when "the problem" was happening or in "normal" conditions? Apart from the unbalanced load the throughput to the PC is the same, ~1.7-1.8 Gb/s.

 

[Moan]

Were those two tests done when "the problem" was happening or in "normal" conditions? Apart from the unbalanced load the throughput to the PC is the same, ~1.7-1.8 Gb/s.

When the problem was happening (after the switch reboot). Without the problem I'm always getting full 2.5Gb/s in the Task Manager (NAS + Router's Iperf).

I updated my previous post with some extra information.

I should also check what happens if I connect the Router to the Switch with the 1Gb/s port instead of the 2.5Gb/s one.

 

[Moan]

OK...

So it's a bit different than what I thought.

On the Router when I was performing Iperf from the New NAS it was always maxing at ~2.35Gb/s, even with -P 1. So even after the switch reboot, speed wasn't reduced between them.

I connected the Switch with the router, using the 1Gb/s port.

Iperf from router was maxing at ~1Gb/s (as expected).

When I started transferring a file from the New NAS at the same time, it still maxed at ~1.7 - 1.8Gb/s!

So there is no way that the transfer was going by the router, because I would get max. 1Gb/s then.

I apologize for guessing wrongly, but as you guys read this problem is extremely weird to the point that I had no logical explanation of what's happening, and that was the most logical explanation.

I also noticed, that I can sometimes "fix" this Switch reboot bug, when I unplug My PC ethernet cable from the Switch, and connect it again few seconds after the switch restart.

I suspected that My PC might have a buggy 2.5Gb ethernet drivers. I reinstalled the Microsoft's default one, but it didn't improve anything. Then I again installed the newest Realtek version, disabled all possible Energy Saving options (that was done before, too, but I had to reconfigure after the drivers installation). It didn't help.

I didn't want to blame My PC before, because the same problem is reproducible on my laptop with Ethernet USB adapter that is totally different than the card in My Desktop PC.

Either way, nothing seemed to help.

My final blame is this terrible QNAP QSW-1105-5T Switch. I didn't mention it before, but since I bought it on Amazon around December, I was experiencing a problem with ports fully locking up (and not transferring any data). After few weeks I sent it back to Amazon, and they replaced it with a new one. After that, I didn't notice the full port locking problem anymore, so I assumed the Switch just works fine. After noticing this bug with slow transfers after reboot, I suppose that these switches are just bad, and the fact that it's impossible to update them makes it worse. I'm not sure if I should try to send it for warranty, or it won't change anything.

It seems that ironically, the fact that I'm using the USB Adapter in the New NAS allowed me to workaround this stupid Switch issues. Thanks to that I can simulate some kind of a hardware reconnect ~90 seconds after link is up. So maybe I should just use it as it is now?

It would be helpful if somebody who owns the same QNAP switch made some tests to see if the same problem occurs. I think this switch might be incorrectly reducing max. speeds for ports after reboot, and it fixes itself.

------------------

It also seems like I'm getting better speeds when I iperf the router on My New NAS. It goes 2.35 Gbits/sec then. Maybe it's just better between Linux devices, or maybe just that port always gets limited speed with My PC only...

 

[ColinTaylor]

It also seems like I'm getting better speeds when I iperf the router on My New NAS. It goes 2.35 Gbits/sec then. Maybe it's just better between Linux devices, or maybe just that port always gets limited speed with My PC only...

I see your PC has multiple network adapters. Disable (not just unplug) all but the one connected to the switch and see if that's makes any difference.

 

[sfx2000]

The New NAS is an Ondroid N2 device. I have kernel 6.3.11-arm64 on it, and it's "Debian GNU/Linux 11 (bullseye)". To get the 2.5Gb/s port, I use external USB RTL8156B device. I own two of these (they're from different manufacturers, and they look different, but RTL8156B is the same) and both of them are affected by the same bug.

Well - couple of concerns...

I assume it's this one - https://www.hardkernel.com/shop/odroid-n2-with-4gbyte-ram-2/

1) the Amlogic S922x is basically a TV box SoC, so your mileage may vary
2) Realtek's USB3 2.5Gb chips - again YMMV... it's meant to be a client device under windows, so linux is not a priority for them...

 

[Moan]

Are you guys ready for a cherry on a cake?

The reason of all these problems is: Realtek's Windows 11 Driver!

I started Windows 8.1 on My PC, to realize that the problem is not occurring there.

After I switched back to Windows 11, I installed the Windows 8 driver from: https://www.realtek.com/en/componen...0-1000m-gigabit-ethernet-pci-express-software
- Name: Win8, Win8.1 and Server 2012 Auto Installation Program
- Filename: Install_Win8_8.1_8088_12222021.exe
It was impossible to install it on Windows 11 using the .exe, so I opened this .exe file with 7zip and extracted it as an archive. Then in Device Manager -> Network Adapters -> Realtek PCIe 2.5GbE Family Controller ->Driver -> Update Driver -> Browse My Computer for drivers -> Let me pick from a list... -> Have Disk... -> Browse... I selected a file from the extracted folder: WIN8/64/rt630x64.inf.
This results in MUCH BETTER iperf speeds from the router on My windows machine:




... and of course the Switch Reboot bug is fixed.


I also tried the Windows 10 driver from: https://www.realtek.com/en/componen...0-1000m-gigabit-ethernet-pci-express-software
- Name: Win10 Auto Installation Program (NDIS)
- Filename: Install_Win10_10065_05032023.exe

It seems to also work fine with iperf from the router:

... and the Switch Reboot bug seems not to occur, but I'll test it more to make sure.


At the end, just to confirm that Windows 11 driver is bad, I again installed it from: https://www.realtek.com/en/componen...0-1000m-gigabit-ethernet-pci-express-software
- Name: Win11 Auto Installation Program (NetAdapterCx)
- Filename: Install_PCIE_Win11_11.13.0424.2023_05052023.exe

And yeah, the iperf speed is much slower:

... and the other problems with the switch reboot, which I don't need to repeat.


So the main issue was Windows 11 driver. Seems like it uses this stupid "NetAdapterCx", even for the USB Ethernet adapter on my laptop.
I have no idea what it is doing wrong, but I don't want to waste more time on it. I'm just happy that I found better drivers. I hope they'll work just as fine in games, without adding any latency, blue screens, etc.

Well - couple of concerns...

I assume it's this one - https://www.hardkernel.com/shop/odroid-n2-with-4gbyte-ram-2/

1) the Amlogic S922x is basically a TV box SoC, so your mileage may vary
2) Realtek's USB3 2.5Gb chips - again YMMV... it's meant to be a client device under windows, so linux is not a priority for them...

To be honest, it's a really nice device for the NAS, especially when I added the 2.5Gb ethernet adapter.
As it can be seen in this screenshot:

With the good drivers on Windows, I'm often getting over 212MB/s. This is not bad at all. And the beautiful thing is this device consumes like no electricity, is not heating and it's only loud from the external hard drives (when they're running, but in my low-use scenarios they're sleeping most of the day). So I wouldn't judge based on the CPU used. It works as expected and performs very well for these tasks And my Windows 11 machine with Ryzen 9 5950X had problems after the switch reboot

I'm still not sure why Windows drivers don't like when the switch reboots. The most annoying thing was, that this "bug" was probably occurring at some Windows startup. This is why I started noticing it, because in normal use scenario that switch is normally running 24/7 with no issues.


Anyway, I hope somebody will find this thread helpful in the future. Conclusion is simple: avoid "NetAdapterCx" drivers on Windows 11.