A morality tale: The Importance of flow control

[azazel1024]

So a thread either on here or over on Anandtech got me thinking about flow control a bit. I've never bothered playing with it on a network. Just leaving the defaults, and oh hey, things seem to work okay.

Well in testing my network config, I delved in to testing flow control too and I am glad I did. What happens next will shock you

My network setup consists of a semi star layout.

I have a core network switch of a TP-Link SG-2216 with all ports filled. Branched off is a Trendnet TEG160sw with only 3 ports currently filled, plus a dual port uplink to the SG-2216. I also have a 5 port DLink DGS1100-05, plus 3 routers (2 in AP mode) off the SG2216 as well, feed either themselves, or multiple devices (well, only the DGS1100 feed multiple, the APs have nothing hung off them currently, other than WLAN clients and the router is only to WAN).

I wanted to test my uplink between switches to see if SMB Multichannel would work cross switches. I couldn't think of a reason why not, but I've seen that SMB Multichannel does not work with adapters teamed (its an either or, though I guess if you had ENOUGH NICs, you could have two seperate teams of adapters and each team could combine for SMB Multichannel, but since I don't have 4 extra ports laying around to combine on my server and desktop, I can't test that theory, unless someone wants to donate some Intel adapters to the cause...)

Pulled the cables on my desktop at the switches and plugged it in to the TEG160sw, so that it had to go across the 2 port uplink to the SG-2216 and then to the server. No dice, I got 20-30MB/sec of performance. This is compared to ~230MB/sec of performance intraswitch with the two NICs on both machines.

I thought, dag nabit, SMB Multichannel is bunk across switch...wait a second...hmmm, the switch ports say flow control is disabled on the switch, I wonder if I...oh, look, 117MB/sec now. Hmm, so SMB Multichannel doesn't work across switches, but with flow control on, it'll work at single port speed. Okay, that is something. Wait, upload from server to desktop is 20-30MB/sec. WTF?

More playing, if I connected one port to each switch I could get 230MB/sec. Screwy. I checked my other switch, oh, flow control is off on that one, lets turn it on. Screwy results. Okay, lets DISABLE flow control on my Intel NICs...oh, 230MB/sec ACROSS switches, lets enable it on the NICs, 117MB/sec on both switchs? WTF? Checked again a few minutes later, 236MB/sec on both switches.

I played with a few other NICs I had in other machines, they are all working better (and with flow control enabled on them).

Moral of the story, sometimes things need to settle (I guess windows wasn't kicking in SMB Multichannel with the NIC settings changed, at least not for a minute or two). Also Flow control is VITAL for the proper functioning of things.

With flow control disabled, if I had both Intel NICs enabled, but only ONE of them connected, I'd get 20-30MB/sec. If I had both enabled, I'd get 230MB/sec, but that is the only time things seemed to work great. I'd be hit or miss on other adapters working properly. Sometimes I could get 114-116MB/sec, and sometimes I'd get odd 20-50MB/sec performance. With flow control enabled on EVERYTHING, EVERYTHING seems to work better.

Also single port speeds are higher too. Previously my max was with the Intel NICs and I'd hit 117.5MB/sec ultimate max. I am now hitting 119-120MB/sec with a single NIC and teamed I hit 238MB/sec (this is with 9k jumbo frames). Most of my other adapters now work consistently at full speed and also seem to have picked up a good 1-2MB/sec of performance also.

Which makes me wonder, why the heck is flow control being disabled on the switches the default behavior? This was the case with all 3 semi-managed switches I had.

As a test, I tried disabling flow control on EVERYTHING (as all NICs default behavior has been to have it on, and all switchs was off, I set them all to off), things behaved a little better than with the NICs having it on and the switchs off, but with everything set for flow control to be one, everything performs MUCH better.

So, the morality tale ends with, flow control is rather vital for the proper functioning of things apparently.

 

[System Error Message]

they're disabled by default because when they first came out things didnt work properly. For example if you had a server connected to a gigabit port, a 100Mb/s client and a gigabit client, with flow control turned on and the 100Mb/s client communicating with server the gigabit client ends up being limited to 100Mb/s instead. This may still be a problem.

Protocols like TCP have their own control mechanism and sometimes flow control interferes by making TCP to keep lowering its speed with the TCP connection thinking that bandwidth is full if packets get held up. For flow control to work the end points and every device in between must have it turned on.

Some applications have support for flow control while some dont. I guess it depends on what you use it for.

 

[azazel1024]

At least in my testing, if flow control was disabled on everything, it caused problems. It worked a lot better than some things being disabled and others enabled, but having it enabled across everything works significantly better than any other setup.

I haven't tried introducing a 10Mbps client, as I don't happen to have any (and I haven't forced any clients down to that, well, other than sleep state downlinked), but with mixed use 100Mbps and 1000Mbps to the same network end point it seems to work just fine.

 

[System Error Message]

Its good to know they fixed that problem hopefully. If different link speeds and control mechanisms are compatible than using flow control would benefit to prevent packet drops.

However flow control must be enabled on everything in your network for it to work properly. Flow control only prevents transmission when bandwidth is full. It may have other functions but i havent seen or read about any. I have used flow control in the past and it is recommended not to use it on older switches that do not have any recent firmware updates. My new switch just arrived so i can use flow control. It seems the mikrotik CRS is more than i bargained for and has a steep learning curve. At least i can finally be rid of the netgear prosafe switch. I now can use 9K and higher MTUs with SFP+on the ports and 65535 MTU on higher layer stuff.

 

[azazel1024]

I'd deffinitely agree with "must be enabled on everything on your network to function properly". When I had the switches set to off and NICs with it on, it was causing serious issues.

For example, my laptop would sometimes bounce between 30-60MB/sec connecting to my server (both with flow control enabled). Disabling flow control on the laptop jumped caused it to ramp up from 30-40MB/sec initial to 108-114MB/sec after about 2-3s for Rx, but Tx it hit 108-114MB/sec immediately. Disabling flow control on my Intel NICs caused it to pretty much perform at 108MB/sec out of the gates for Tx and Rx. With flow control enabled on the NICs and on the switches, the laptop performs at 116MB/sec Rx and Tx immediately and constantly.

There didn't seem to be any issues between my desktop and server with both having the same Intel NICs with flow control enabled on them, but disabled on the switches. I did however notice that if I DISABLED an adapter, I'd get only 20-30MB/sec performance, instead of the expected 117MB/sec I'd see if I physically REMOVED an adapter (interestingly, if I also disabled an adapter in BOTH machines at the same time, I'd get 117MB/sec). With flow control disabled on only one end, the performance was worse, however receiving from the server...117MB/sec, just transmitting was impacted. On the otherside of things, if I was initiating the transfer from the server to my desktop with one NIC disabled, 20-30MB/sec, but receiving from my desktop (initiated from the server), 117MB/sec.

Flow control disabled on both ends and I could get roughly 117MB/sec performance with one adapter disabled and still get roughly 230MB/sec of performance if both adapters were online. Flow control enabled on everything however, and I get 119-120MB/sec of performance from a single adapter and I am peaking at 238MB/sec with both adapters online.

So with flow control on, on everything, everything seems to be working both perfectly and faster than the best case scenario with flow control disabled.

Still seems odd to me that that default behavior in 3 semi-managed L2 switches from different manufacturers was to have flow control disabled, but in 3 different manufacturer's NICs and 5 different NIC chipsets (1 Intel chipset, 1 QCA chipset and 3 Realtek chipsets) between the 3 manufacturers, the default behavior is flow control enabled (AFAIK tell, I have a Marvell and two Broadcom chipsets in the semi-managed switches).

 

[System Error Message]

Your results are correct. With flow control you get consistant bandwidth because flow control uses the pause frame when link is full so packets do not get lost and wait in your computer but without it you do not get consistant bandwidth because packets get lost and have to be retransmitted because of networking gear dropping/returning them due to full link.

To properly test this you should test traffic in bidirectional transfers to see if having full links on both directions can interfere with flow control commands. This means that you need to send and receive at the same time.

If you use windows to test make sure to disable autotuning and other network related features to prevent windows networking from failing under stress since it will definitely happen when you stress hardware as far as you can.

As i mentioned before because of the uncertainty of how well flow control works manufacturers disable them by default.

WHen i meant problems with multiple speeds i meant if both the 100Mb/s client and gigabit client communicated with gigabit server all at the same time.

dont forget to test using TCP based applications. SMB is most likely UDP.

 

[azazel1024]

I tested mixed 100Mbps and 1000Mbps to my server and it had no issues. I also tested full duplex traffic (the advantage of having SSDs on both ends, I don't like doing artifical TCP/IP testing, and I am too lazy to setup RAM disks). The numbers are fairly representative when doing bi-directional max traffic for everything I posted. With flow control on I can hit around 238MB/sec up AND down at the same time from my desktop to the server. With it turned off, it drops to around 230-235MB/sec up and down. With a single link and some of the other NICs I get the odd much slower speeds mentioned. Flow control enabled only part of the way and I get really attricious speeds.

SMB can be both/either. Standard is port 445 over TCP, but it can run on several different ports on the 120-140 range over UDP for SMB direct over NETBIOS. This is NOT a common method of SMB though, but it is supported in the SMB protocol.

 

[W4RH34D]

Flow control or $1,000 in extra nics for LAG on the server. Gee, let me think.

 

[coxhaus]

Do by chance you have a homeplug device? Does flow control work fine across it?

 

[W4RH34D]

I am not sure the homeplug is connected to an AP that only does wireless.

 

[System Error Message]

flow control is between communication so the homeplug must support it too but between 2 homeplug devices it may not work well because the speed varies and it relates to sending the pause frame vs asking to resend data which is what i guess the homeplugs use between each other and for wifi. For the pause frame to work the transmission must be reliable that it will receive whereas wifi and homeplug have packet losses that they have to resend data instead.

 

[W4RH34D]

I'll try it out tonight when no one is using it.

 

[thiggins]

I explored Flow Control a few years back and confirmed some problems with mixing 100 and Gigabit NICs on the same switch that were fixed by disabling Flow Control
http://www.smallnetbuilder.com/lanwan/lanwan-features/30212-when-flow-control-is-not-a-good-thing

 

[azazel1024]

Hmmm, at least with my setup I have several 100Mbps devices connected to my switches that have flow control on, with no problems.

Doesn't mean it can't produce problems, but my stuff seems to be working fine with it on, on all ports.

I have vaguely duplicated the situation of the test. Example, transfering from my server to my desktop (dual 1Gbps links) while also streaming to my Apple TV (100Mbps link) doesn't result in anything other than a few MB/sec off the top of the transfer to my desktop. Now maybe this is because the Apple TV isn't using flow control at all, but my desktop and all switches on my network certainly are/have it enabled as does my server.

All of my 100Mbps devices are generally "configureless devices" like a network printer, MoCA bridge, Apple TV. Only one that is configurable is my Airport Express, but no options to enable/disable flow control on that. So for all I know all 100Mbps are using flow control, or they might all not support it. All gigabit devices that do support it have it enabled (all switches, laptop, tablet UBS3 GbE adapter, desktop and server).

 

[W4RH34D]

one of our links is on bad wire, flow control kills the connection. We need a new line drop there, but I'm assuming fancy protocols like this need the equipment in ideal condition.

 

[azazel1024]

Very well could be the case.