Bizarre issue when measuring upload speeds

[RamGuy]

I have had the feeling that my ISP haven't really provided me with the right upload speeds for the past few months. I'm paying for a 24 / 24bit fiber connection and my download speeds has always been right where they are supposed to, but my upload speeds has been somewhat lower.


So I decided to test this out before complaining to my ISP and I was met with some bizarre results which I can't get my head around.

The entire network is configured like this:

1: D-Link DGS-3010G switch which convert from fiber to Ethernet. My ISP has configured the switch is such a way that port5 has my internet connection mapped with the right speed limitations. Thereby we are able to configure our own local network just the way we'd like, with whatever hardware we choose as the switch is providing us with a raw public IP-address with regulated speeds and that's it.

2: Cisco RV 220W Router, configured without wireless and features CAT7 cabling towards the DGS-3010G.

3: Cisco SRW2024-K9 Switch, featuring CAT6a cabling towards the router. This the main hub on our network, all computers and other network equipment and devices are hooked right into this switch.

4: Netgear WNDR4500 in access point mode. Handling both our 2.4GHz 802.11 (3x3) and our 5.0GHz 802.11n (3x3) broadcasting. Featuring CAT7 cabling towards the SRW2024-K9 switch.



That's the essentials of our local network and how it's been configured. While testing I disable all security features in the router, all security features on all individual systems, I reboot each device beforehand and make sure there is nothing draining network resources while testing.


These are the different systems I used for testing:

Machine #1 is my primary desktop computer, featuring Rampage III Extreme, Intel Core i7-980X and Intel Pro/1000 ET Dual NIC and is running LACP towards the SRW2024-K9 running CAT7 cabling. Windows 7 64-bit.


Machine #2 is my HTPC. This is a Mac Mini Mid-2011 model running Windows 7 64-bit and features a Broadcom BCM57765 Gigabit NIC for cable and Broadcom BMC4331 NIC for wireless testing.


Machine #3 is my MacBook Air 13" Mid-2011 model running Mac OS X Mountain Lion Developer Preview 3 (technically Preview 4 with yesterdays update) and is featuring Broadcom BCM4322 NIC for wireless testing.

Machine #4 is my MacBook Pro 17" Early-2011 model also running Mac OS X Mountain Lion Developer Preview 3 (Technically Preview 4 with yesterdays update) and is featuring Broadcom BCM4331 NIC for wireless and Broadcom BCM57765 for cable testing.



I use Speedtest.net for measuring my network connection, normally I use Hafslund server located in Oslo as Hafslund is the one who built our fiber connection in the first place, but this time I also tested with Devenia LTD also located in Oslo just to see if they would provide different results. I ran a total of about 50 tests on each system, in a chronological matter (first Machine 1, then Machine 2 etc..) over a span of about one hour. I did of course never run two tests at the same time, as that would be silly.


And the results baffled me quite a loot, and too me it doesn't make much sense. Here are the results (I made a rough estimate of the average results of each machine).


Machine #1 (My desktop) with cable:
Hafslund: 24-25mbit down and 16-17mbit up.
Devenia: 24-25mbit down and 16-17mbit up.


Machine #2 (Mac Mini Mid-2011) with cable:
Hafslund: 24-25mbit down and 6-8mbit up.
Devenia: 24-25mbit down and 10-11mbit up.

Over Wireless:
Hafslund: 24-25mbit down and 15-16mbit up.
Devenia 24-25mbit down and 15-16mbit up.


Machine #3: (MacBook Air 13 "Mid-2011) over wireless:
Hafslund: 24-25mbit down and 22-24mbit up.
Devenia: 24-25mbit down and 23-24mbit up.


Nr4 machine (MacBook Pro 17 "Early 2011) over wireless:
Hafslund: 24-25mbit down and 23-25mbit up.
Devenia: 24-25mbit down and 22-24mbit up.

Over Cable:
Hafslund: 24-25mbit down and 22-23mbit up.
Devenia: 24-25mbit down and 21-22mbit up.



The results are quite clear, none of my Windows systems are able to utilise more than 14-16mbit at the same time as all my OS X systems are able to push up towards the 24mbit limit.

How does this make any sense? I booted up my Windows 8 Consumer Preview 64-bit partition on my MacBook Air 13" Mid-2011 model and ran the very same tests and then it was all of a sudden down on 13-15mbit up. So just a few minutes earlier while running OS X Mountain Lion it got 21-23mbit upload speeds, but when testing while running Windows 8 Consumer Preview 64-bit it drops down to 13-15mbit.


This doesn't make any sense, but it's clearly not my ISP doing something wrong. How should one troubleshoot such a bizarre problem? If there was only one isolated system I would see the logic behind it, but here we are talking Windows providing slower upload speeds on my network across various systems, where as OS X is not. That doesn't make any sense to me whatsoever.

 

[stevech]

Firstly, some measurement tools are suspect, such as speedtest.net - because the servers it offers have widely varying loads and the route from you to the server varies each time a measurement is run. Choosing a consistent server (often not the closest) and averaging heavily is about all you can do.

My cable modem provider like most, rate-limits the upstream to burst high speed for a few seconds then reduce the rate dramatically. The idea is that the fast burst takes care of most web page GETs quickly but sustained transfers like FTPs and torrents won't hog so much bandwidth. Speedtest.net doesn't run upstream long enough for this to truly show up.

take note of the clients' varying connection mode/speed:
20MHz vs. 40MHz (usually 20MHz)
WiFi bit rate, e.g., 802.11g max is 54Mbps which leads to a net IP layer yield, at best, of about 60% of 54Mbps.

WiFi bit rate declines with weaker signals - EITHER to or from the router (either direction), and the client's transmitted signal is the weakest link and unreported by most routers.

Competition (with neighbors' WiFi if they are heavily used) and/or other 2.4GHz signal-emitters - this would affect all clients. Channel-change on router (1, 6 or 11).

Client device's TCP/IP stack config: "RWIN". Google that. In Windows, this isn't an issue since early Win XP days.

 

[RamGuy]

Thanks for your reply. Of course Speedtest.net is no ideal, the best would be to have a server at my neighbour or something which had 100mbit connection as see how fast we could grab things of my file-server etc..


BUT, as I have ran the test so many times, over a short period of time and you can CLEARLY see that ALL Windows machines are in the 13-16mbit department and NEVER reaches 20mbit and ALL OS X systems are always in the 21-24mbit department and NEVER drops below 20mbit upload speeds one should be able to conclude that there is something really awkward going on.

When it comes to rate-limiting upload speeds that's something consider illegal here in Norway, and no ISP have done that for years, and running fiber connections of 24 / 24mbit there wouldn't really be any logical reason for limiting the uploads in the first place as there is much headroom left in the infrastructure.


Various connection speeds and technologies is also a valid argument, but you have to consider I have tested on multiply system, with both cable and wireless with the exact same results. Not to mention all wireless clients are 802.11n capable, my wireless access point is configured for N-only and the rated speeds while testing was either 300mbps or 450mbps (not that those numbers matters anyway). Another thing to note is that there is no other 5.0GHz network close by at all, and there is only one other 2.4GHz noticeable in my area and with there being a distance of minimum 80 meters to the nearest neighbour there shouldn't be much interference in the first place.

 

[jdabbs]

Investigating jumbo frames or TCP settings (like stevech suggested) would be my next step.

The performance difference between OSX and Windows systems is pretty telling. Let's say there's an issue like jumbo frames being supported on the Windows boxes and not the OSX ones. Something has to reencapsulate those frames for transit over the Internet, and that can certainly impact throughput. Upload is affected but not download, so it is something about the data that is being sent that is probably off.

 

[overdrive31]

Might be something as simple as the Bonjour service. Disable it on the Windows machines and retest. If Bonjour is required for network tasks, you may have to live with the limitation.

 

[RamGuy]

Thanks for your suggestions, I've had Jumbo Frames enabled on both the SRW2024-K9 switch and the router for a very long time and have had both my file-server and all my desktop Windows system configured with 9k jumbo frames, but non of my OS X systems nor my HTPC nor any other system on the network has been using jumbo frames.

So I tried to disable it on all systems, on the switch and on the router. It didn't make any difference to the results. I tried disabling bonjour on my desktop but still the same results with it disabled.



When it comes to the network card settings on my Windows systems they are mostly on stock settings. I took my time configuring some of the settings on my desktop like going from one side scaling queue up to eight but it didn't make any difference either.

 

[overdrive31]

Maybe it is a symptom of QoS, try disabling it on the Cisco Router.

 

[RamGuy]

There is no QoS on the router, I tried disabling it on the switch without any luck.

 

[overdrive31]

According to the SNB Router Finder, your router does indeed have QoS.