DSLReports Speed Test Now Measuring Buffer Bloat

[Chrysalis]

Because download and upload traffic are being buffered at different points. Upload is most likely being buffered at your modem, while download is being buffered at some device within your ISP's network.

Its not quite like that, I come across a lot of posts claiming ingress QoS isnt possible but it is, here is the reason why.

There is 3 things that determine throughput.

Send window
Congestion window
Receive window

The first 2 are indeed at the sender, but the 3rd is at the downloader. You can police downstream traffic by capping the receive window which is how downstream QoS works.

 

[Nullity]

Its not quite like that, I come across a lot of posts claiming ingress QoS isnt possible but it is, here is the reason why.

There is 3 things that determine throughput.

Send window
Congestion window
Receive window

The first 2 are indeed at the sender, but the 3rd is at the downloader. You can police downstream traffic by capping the receive window which is how downstream QoS works.

Ingress QoS does not exist. You cannot schedule the priority of incoming packets. You can rate-limit, kinda, by dropping packets and triggering TCP's congestion control algorithm, but that is not QoS. It also is not fully controllable, unlike packets being sent.

Think of it like the Postal Service. You only have full control over what you send. You cannot fully control when a reply will arrive. You can make requests, but that is all.

It is much more complex than that but a book like Tanenbaum's "Computer Networks" or white papers on network scheduling algorithms will explain it much better than I ever could.

For a concise explanation of QoS/traffic-shaping, my favorite tutorial is http://www.linksysinfo.org/index.php?threads/qos-tutorial.68795/
He (toastman) even practices what he preaches.

 

[Chrysalis]

I am talking about rate limiting to stop isp's buffering downstream traffic, not shaping, dont confuse QoS and shaping as the same thing, shaping is just a form of QoS whilst policing is another form of QoS, downstream policing definitely does exist, even various software packages like ftp clients allow you cap downstream, speeds.

I dont know if tomato can police downstream, if it doesnt thats bad for them, its no indicator of whats possible or not, I have downstream QoS on my router which is effective.

 

[Nullity]

I am talking about rate limiting to stop isp's buffering downstream traffic, not shaping, dont confuse QoS and shaping as the same thing, shaping is just a form of QoS whilst policing is another form of QoS, downstream policing definitely does exist, even various software packages like ftp clients allow you cap downstream, speeds.

I dont know if tomato can police downstream, if it doesnt thats bad for them, its no indicator of whats possible or not, I have downstream QoS on my router which is effective.

I think we both understand each other. We are getting into advanced details that are beyond the simple intention of this thread, though. Complex traffic-shaping topics are better suited elsewhere, imo.

 

[Syaoran]

Not too bad!

 

[sfx2000]

I'm starting to wonder if the Broadband operators are learning how to game the DSLReports speed test - last month or so, the uplink speed (from my client to their server) is consistently 300 percent more than what I would expect to see...

 

[Nullity]

I'm starting to wonder if the Broadband operators are learning how to game the DSLReports speed test - last month or so, the uplink speed (from my client to their server) is consistently 300 percent more than what I would expect to see...

Local speed-test server ran by your ISP or your ISP's provider (or their provider, etc)?

Even when I test with ISP-hosted speed-tests, I get the same results, except when there are problems.

 

[sfx2000]

Local speed-test server ran by your ISP or your ISP's provider (or their provider, etc)?

Even when I test with ISP-hosted speed-tests, I get the same results, except when there are problems.

DSLReports speed test thingy...

Going to speedtest.net or Comcast;s speediest thingy (Comcast's is nice as it does ipv6 vs. ipv4)...

They're more consistent with my provider/service level (CoxHSI 50/5 mid-tier)

 

[punkinduster]

Wonder why quality and speed are so bad. Is it because I only subscribe to 10m/800k? I set QOS to 9.9m/750k.

 

[Nullity]

Wonder why quality and speed are so bad. Is it because I only subscribe to 10m/800k? I set QOS to 9.9m/750k.

Speed is a relative measurement. As long as it approximately matches the speeds you pay for, disregard the rating.

For proper QoS, you need to measure your download/upload rates with QoS disabled. The bitrates you put into the QoS settings must be below the rates you observed during the QoS-disabled tests. How much below depends on a few things, but 3-15% is a reasonable amount. Some subtract up to 30%+, especially on download.

Your bufferbloat seems fine though, so you may either have QoS working properly, or your ISP (and possibly their CPE modem/router) has limited problems with bufferbloat.

 

[Haggster]

I've come back to the test today for the first time since this summer and have to say it's improved a lot.

So I've reviewed my speeds and especially my QoS settings again and found the following extremely strange:

First some background:
- my connection is rated 20mbit/s upstream
- the real life speed I'm getting is 18.?mbit/s upstream (it used to be 19.? in the past sometimes, but for some reason I'm not getting that anymore, but let's ignore that)

now, I was playing with the QoS settings and was testing some "crazy" values, such as 10 (half of my rated speed) and also 5 mbit/s as the limit.

Then, when I executed the test, the buffer bloat got worse if I set it to 10 and horrible when I set it to 5:

- basically, starting from around 10 mbit/s as the limit and going lower, the buffer bloat values got bad
- with 5mbit/s they got into the red area with very bad results
- then I tried values very close to my maximum upload speed, first 16, 17, then 18 mbit/s and they were fine. I almost got the impression that 18 was the best value with the least buffer bloat

Before this I had also verified that *without* any QoS I get horrible buffer bloat values (basically grade "F" consistently).

So my conclusion is that the QoS definitely does its job preventing buffer bloat but for some reason I can enter very high limits and its working fine with those, while the values get worse if I use low values from around 50% of my theoretical speed and lower. This result is the opposite of what people who claim to know about QoS theory say and the opposite of what I'd expect as well. Has anyone here had the same results? Would there be any technical explanation for this?

I found that it's also very important to run the test on a device with sufficient CPU power and resources; the machine I ran the test from is an i5 Haswell laptop with 8 gigs of RAM and an SSD and gives me the fastest and most consistent throughput of all of my devices at home, so it's not something that influences the test results because of any client related problem.

As a final test I completely maxed out my connection with the above mentioned laptop via an upload and download speed test while at the same time consistently pinging a server on the internet from my tablet; the ping times were completely unaffected by the speedtest that was running on the laptop - and that while I was using a QoS limit with 18mbit/s up (which, again, is almost identical to my real life maximum upload speed), so that was another proof that the QoS is working fine with that high value, since I've done that test before and without QoS the pings would simply time out.

EDIT: I forgot to mention that while conducting those tests I even switched on the "Hi-Res BufferBloat" option after a while in order to get an even better picture. Using that option it will show the pings in real time and pings 10 times per second, therefore the buffer bloat indicator jumps around less and gives a clearer picture of the actual values while the test is being carried out.

 

[Nullity]

I've come back to the test today for the first time since this summer and have to say it's improved a lot.

So I've reviewed my speeds and especially my QoS settings again and found the following extremely strange:

First some background:
- my connection is rated 20mbit/s upstream
- the real life speed I'm getting is 18.?mbit/s upstream (it used to be 19.? in the past sometimes, but for some reason I'm not getting that anymore, but let's ignore that)

now, I was playing with the QoS settings and was testing some "crazy" values, such as 10 (half of my rated speed) and also 5 mbit/s as the limit.

Then, when I executed the test, the buffer bloat got worse if I set it to 10 and horrible when I set it to 5:

- basically, starting from around 10 mbit/s as the limit and going lower, the buffer bloat values got bad
- with 5mbit/s they got into the red area with very bad results
- then I tried values very close to my maximum upload speed, first 16, 17, then 18 mbit/s and they were fine. I almost got the impression that 18 was the best value with the least buffer bloat

Before this I had also verified that *without* any QoS I get horrible buffer bloat values (basically grade "F" consistently).

So my conclusion is that the QoS definitely does its job preventing buffer bloat but for some reason I can enter very high limits and its working fine with those, while the values get worse if I use low values from around 50% of my theoretical speed and lower. This result is the opposite of what people who claim to know about QoS theory say and the opposite of what I'd expect as well. Has anyone here had the same results? Would there be any technical explanation for this?

I found that it's also very important to run the test on a device with sufficient CPU power and resources; the machine I ran the test from is an i5 Haswell laptop with 8 gigs of RAM and an SSD and gives me the fastest and most consistent throughput of all of my devices at home, so it's not something that influences the test results because of any client related problem.

As a final test I completely maxed out my connection with the above mentioned laptop via an upload and download speed test while at the same time consistently pinging a server on the internet from my tablet; the ping times were completely unaffected by the speedtest that was running on the laptop - and that while I was using a QoS limit with 18mbit/s up (which, again, is almost identical to my real life maximum upload speed), so that was another proof that the QoS is working fine with that high value, since I've done that test before and without QoS the pings would simply time out.

For a QoS (a painfully generic term) setup to specifically fix bufferbloat, the device needs to employ an AQM (CoDel, RED) with the precise job of dynamically keeping the size of networking buffers small. Functional QoS does not automatically mean bufferbloat will be eliminated, though it can be an improvement since buffering is now happening at a different network node/device. Bufferbloat can also get worse... it depends on the size of the networking buffers being used.

Regarding the increase in bufferbloat with lower bitrates: Since non-AQM networking buffers are a static size, like 2Mbyte, if you were to halve the bitrate, that buffer now holds a 2x longer time-span of network traffic.

Example:

Buffer = 10bits
Bitrate = 10bit/sec
Bufferbloat = 1 second

Buffer = 10bits
Bitrate = 5bit/sec (halved)
Bufferbloat = 2 seconds


Bufferbloat is a problem because generic network buffers are primarily built for maximum throughput. Think of 1Gbit routers buffering a 12Mbit ADSL connection... bufferbloat is virtually assured.

 

[Haggster]

Thanks for the explanation, so this makes sense now! I think I will keep these new settings, because now I get more upload speed and also less buffer bloat.

For a QoS (a painfully generic term) setup to specifically fix bufferbloat, the device needs to employ an AQM (CoDel, RED) with the precise job of dynamically keeping the size of networking buffers small.

Too bad that Asus keeps using almost five year old 2.6. kernels, otherwise we'd at least be able to use CoDel & co. And third-partyWRT firmwares are not an option for people who want hardware acceleration and the DPI engine.

 

[Nullity]

Too bad that Asus keeps using almost five year old 2.6. kernels, otherwise we'd at least be able to use CoDel & co. And third-partyWRT firmwares are not an option for people who want hardware acceleration and the DPI engine.

Yeah. Though, some interesting things are included in AsusWRT though, like ESFQ, an apparently improved SFQ net sched algo capable of some buffering customizations and "head drop buffering" (rather than tail) along with a few RED implementations. One of the problems is the complexity of configuration, especially when compared to CoDel's "no knobs" setup.


John is attempting to improve bufferbloat with his fork by trying a few things. I dunno how it is going though, since I currently use pfSense.

 

[Easy Limits]

On my AC56U using Comcast I went from "F" to "A" by just turning on QOS Adaptive.

My bufferbloat went from C to A with this on an AC56U.

 

[ApK]

I must say I don't really understand bufferbloat. I only heard of it this week for the first time, and from skimming, my take-away is:
1. Excessive bufferbloat can make even a fast connection seem seem choppy or slow to respond.
2. It really only manifests a problem when your connection is near saturation.
3. I may be wrong.

I recently had my service increased from 25Mbps to 150. I tried the dslreports speed test and got an F for bufferbloat.
I turned on QoS on my RT-N66U/Merlin 378.56_2, and got an A for bufferbloat, but went down from >150Mbps to about 110Mbps.
I suspect, as some have said above, the RT-N66U may not have the horsepower to do QoS at that full speed?

Anyway, I've decided to turn QoS off and enjoy my full speed until I see some problem that I can attribute to bb. Am I being selfish and helping to break the Internet?

 

[RMerlin]

I must say I don't really understand bufferbloat. I only heard of it this week for the first time, and from skimming, my take-away is:
1. Excessive bufferbloat can make even a fast connection seem seem choppy or slow to respond.
2. It really only manifests a problem when your connection is near saturation.
3. I may be wrong.

I recently had my service increased from 25Mbps to 150. I tried the dslreports speed test and got an F for bufferbloat.
I turned on QoS on my RT-N66U/Merlin 378.56_2, and got an A for bufferbloat, but went down from >150Mbps to about 110Mbps.
I suspect, as some have said above, the RT-N66U may not have the horsepower to do QoS at that full speed?

Anyway, I've decided to turn QoS off and enjoy my full speed until I see some problem that I can attribute to bb. Am I being selfish and helping to break the Internet?

Let me try to explain the issue in my own words, rather than with technical mojo:

Initially, the main goal of a network was to transfer a lot of data as fast as possible (which was, not THAT fast at all). Networks weren't always reliable (cut offs, retransmition of lost packets, etc...), the speed of all the elements (starting with the CPUs) wasn't always good enough to keep up. So, at various stages in a network you will have a buffer, which will compensate for temporary loss of connectivity, or the CPU being temporarily unable to keep up. That way, if there is a slowdown on one end, the buffer can compensate by keeping the destination fed. This is identical to what audio streaming does (remember the good old days of Realplayer?).

These days, processing power increased, and the speed at which we can transfer data also increased, so buffers also increased in size, to keep up with all of this. Speed mismatch also increased, when an ISP's 100 Gbps pipe is feeding your 10 Mbps cable connection. Another change occurred over the years: people are now starting to use an increasing number of time-sensitive applications over a network. VoIP and online gaming are the two biggest cases that come to mind. This means that the data sent from point A ends up at various transit points sitting in a buffer, and waits there as the data ahead of it is still being sent. This introduces a delay between the time the data is sent from A, and it reaches B. Too much buffering equals too much delay, which will negatively impact the performance of your VoIP or online gaming.

One way to limit the impact is to reduce the buffer sizes. This will potentially have some drawbacks (potential drop in speed, interrupted flow as there is a temporary hiccup in the connection, etc...).

Another is to ensure that your portion of the pipe is never fully utilized, so there is never any data waiting in the buffer. That's where setting up Adaptive QoS with the speed limited slightly below that of your connection will help.

And another method is to more intelligently manage that buffer, what ends up there, what to send first and what to queue for later. Queue management is what things like Codel do.

All of these have one big cave eat however: it only applies to the portion of the network that YOU control. Buffering occurs at many levels, including (but not limited to):

- The ISP's network, between their uplink and their customers
- The router between the ISP and your WAN
- Your modem
- Your router
- The network card of your computer
- The networking stack of the OS on your computer


One extremely simplified way to explain this:

John has 10 apples, he wants to give them to Lucy. John drinks too much coffee and Lucy just woke up, so she's not as fast as John. Each time Lucy gets an apple, she must count it out loud, so Marc can write it down. A few ways this can be done:

1) John can throw all 10 of them at Lucy, which will miss most of them
2) John can throw them one by one as fast as he can, and if it takes too long for her to put them down, she'll miss some
3) John can put them on a table between both of them as fast as he can, and Lucy picks them up as fast as she can. If she's too slow, no worry - the table will keep them during that time. However, it means she won't be counting them as soon as John has handed them out - there will be a delay between the apple leaving John's hands, and the moment she calls it out loud.


The table here, is the buffer.

(PS: don't quote my post, something in it triggers the forum's spam filter...)

 

[Rich Brown]

I must say I don't really understand bufferbloat. I only heard of it this week for the first time, and from skimming, my take-away is:
1. Excessive bufferbloat can make even a fast connection seem seem choppy or slow to respond.
2. It really only manifests a problem when your connection is near saturation.
3. I may be wrong.

I recently had my service increased from 25Mbps to 150. I tried the dslreports speed test and got an F for bufferbloat.
I turned on QoS on my RT-N66U/Merlin 378.56_2, and got an A for bufferbloat, but went down from >150Mbps to about 110Mbps.
I suspect, as some have said above, the RT-N66U may not have the horsepower to do QoS at that full speed?

Anyway, I've decided to turn QoS off and enjoy my full speed until I see some problem that I can attribute to bb. Am I being selfish and helping to break the Internet?

You've got the right idea. Bufferbloat happens *at the bottleneck* of the connection. This is usually your connection to your ISP. You'll definitely see the latency if you're saturating the link (that's how DSLReports Speed Test does its test).

But you'll also get the latency/lag on a micro scale when you're browsing the web. That's because your link to the internet is either 0% used (when idle) or 100% used (during a packet transmission/reception). Even when your line is otherwise idle, loading web pages (which can now easily hit 2 mbytes these days) can cause a significant buffer to develop. That holds up other session's data while the web page data arrives, and makes your VoIP choppy, your game laggy, and DNS/TCP connections get sticky.

UNLESS... your router has a mechanism for intelligently managing the data flowing through it. CoDel and the associated fq_codel examine the times that packets are queued, and offer feedback to the "big senders" to let them know to slow down. This maintains responsiveness for low-traffic flows, and ensures fairness amongst the large flows. Typically, you set the fq_codel parameters to a couple percent below the rated link speed, and the algorithm does the rest.

Asus routers have some vaguely-described algorithm that seems to require a bigger "discount" to link speed to achieve the good bufferbloat grades. But a lot of people seem happy with it.

And to answer your final question: No, you won't break the internet. And if you're content with the performance you're seeing, leave your router alone and expend your energies elsewhere.

 

[tsunami2311]

THis what I get pay for 100/40, I get 120/40 with QOS off, QOS set to 130/25 but seeing router maxes out at around 100mbit(100% cpu) max i seen was 110 but that rare with QOS on, me setting it to 130/25 dont really mater

This is Wired lines for what ever reason wifi on the surface 3 i have get 130 with QOS on which is odd, I limited most wireless devices to about 30mbit more most part since so it being able to do 130mbit with QOS is moot lmao

 

[Nullity]

THis what I get pay for 100/40, I get 120/40 with QOS off, QOS set to 130/25 but seeing router maxes out at around 100mbit(100% cpu) max i seen was 110 but that rare with QOS on, me setting it to 130/25 dont really mater

This is Wired lines for what ever reason wifi on the surface 3 i have get 130 with QOS on which is odd, I limited most wireless devices to about 30mbit more most part since so it being able to do 130mbit with QOS is moot lmao

QoS bitrates must be set below your observed throughput. So instead of 130/25, it should be ~95/10, according to your speedtest results.

Though, unless you have a need for QoS, you have no bufferbloat so I would disable QoS.