An issue with the QoS implementation? Buffer / latency related.

[Radical_53]

I'm always looking for ways to improve my connection, keep the latencies as "tight" as possible and give everyone on my network the best possible experience.

Since I first ran the ICSI "Netalyzer", one particular part of the result rose my attention: Network buffer measurements.
As soon as I turn on the QoS, the uplink buffer virtually explodes. It is being increased tenfold and usually ends at something like 3-3.5 seconds

Now, at first I thought it was the QoS per se. As, when I turn it off, the issue would go away. But:
By increasing the upload "bandwidth" on the router to something very large, far larger than my connection, the excessive buffer will get smaller and smaller.
My connection allows for 1MBit upload, not very much but I get along. As soon as I set 2MBit on the router, the buffer will be down from 3.5 seconds to something like 690ms. 5MBit? 290ms. 50MBit? 140ms.

As I just found out, it even happens when I turn on the QoS that's native to my NIC. Pretty weird.

Could anyone with a little more insight to the core of this matter shed some light on it? I really love a working QoS, but I can't exactly see where (and why) this buffer arises once I put some "realistic" values in.

 

[L&LD]

Do you see any real world differences with the different setups?

That would be my only concern. If it's just some numbers that go higher or lower without affecting something real - it doesn't matter too much, huh?


I am using the RT-N66U router with 374.42 RMerlin firmware on it and enabled QoS with 1GB up/down and 0.5MB up/down bandwidth and ran the Netalyzer tests with each configuration.

The 1GB up/down config let Netalyzer report few issues.

The 0.5MB up/down config showed that Netalyzer returned errors with QoS enabled and such a low ceiling for bandwidth.

Actually; with the 1GB settings, Netalyzer shows up and down speeds far below what my ISP is rated for. It reported 6Mbps up and 7Mbps down.


My actual ISP is 50/10 and I get 53/11 (Mbps) consistently.

I am going to leave QoS on at the 1GB up/down settings and see if I notice any real world improvement over QoS off.


Just quickly browsing right now with QoS 1GB on seems just as quick as with QoS off. Let's see how it will behave with multiple users after a couple of days.

 

[Radical_53]

Hard to say, really. No "huge" issues, apparently, but I'm always a little worried about latency & gaming. The tool said the high buffer would, or could, be an issue with games, and as you may know... there's always a bad feeling, if it "feels" like you're a second behind the game, reading a comment like that from Netalyzer.
That's my only concern. I don't have anything "huge", just the feeling that some games seem to be behind a second or two. Nothing others wouldn't report but, honestly, something I'd love to be able to fix.

 

[L&LD]

I don't do online gaming, but I would think that that lag is there for all.

Note: I didn't need to wait for 2 days to see how the system was responding - I disabled QoS and am back to my speedy network again.

The issues I saw? Just little things like pauses before a page loads fully (or before it displays anything) and the very real difference of logging into different forums - I thought is was the first forum that was at fault. But they couldn't all be, right? I would log in and the screen would turn white (waiting?).

With QoS enabled at 1Gbps up/down, I tried the Ookla speedtest and my speeds were exactly what they were before I enabled QoS (that contradicts what Netalyzer reported), so to me that suggests that any single thing that Netalyzer marks as 'bad' may or may not be; it would take a human to determine which side of the line it was.

I turned QoS off and I am now browsing at full speed again. Not only with what Ookla speedtest says, but also what my senses tell me too (on real websites).

How fast is your ISP connection? To increase the responsiveness of the internet, I've found that I had to upgrade the upload speed of my ISP service.

 

[RMerlin]

Just out of curiosity - try setting a higher priority rule for UDP port 53 (DNS).

 

[Radical_53]

Thanks for that! I'll try that out once I get home tonight. Would it be a "real" fix or something to fool the test? So to say, at least.

@l&ld: There's no upgrade available to my right now, otherwise that would've been the first thing I'd have done. The only alternative is cable and I'd rather not do that.

I don't have any issues with QoS, like the ones you've experienced, but there's some randomness to games that *could* have to do with it.
A short lag during an online game can have all sorts of root causes and I'm trying to fix as much as possible on my side.

 

[connorm]

i have used netalyzer. i know alot about bufferbloat.

i can not say this will work for your connection, everyones ISP and there situation is different, but for my 100/5 cable connection there were 2 things that worked for me

first thing i should say is that i always only used qos for upstream, NEVER for downstream, i always set my downstream at 1000 so its not affected.

using default settings on asus QOS. and settings my upload at 80% would help, but it wouldnt help that much. this was about the most basic but least effetive way for me.

i then slowly got away from default settings with asus/merlin WRT and started adjusting QOS settings a bit. this was a good learning experience and it allowed me to limit bandwidth to certain devices but i dont ever believe it improved my latency under load changing any settings from default

the main difference i noticed was when i switched from merlin-wrt to tomato shibby.

i used the bandwidth limiter set at like 5.2mbit when i get 5.5mbit up

i maybe increased my ping by like 15-25ms under load. a decent bit but it was good.

the only thing that can beat my RT-AC66U running Tomato Shibby with a basic bandwidth limiter. is my ubiquiti ERL running the 3.4 linux kernel with FQ-CoDel and HTB.

this kinda stuff aint available on 99% of routers unfortunately. but i have been pushing for everyone to implement FQ-CoDel

but with a Ubiquiti ERL Running Beta 1.5.0 with a custom FQ-CoDel + HTB Traffic-Shaper script this is what my upstream look like under full load

pinging my CMTS for this test since i am on cable

Reply from 174.7.244.1: bytes=32 time=6ms TTL=254
Reply from 174.7.244.1: bytes=32 time=7ms TTL=254
Reply from 174.7.244.1: bytes=32 time=7ms TTL=254
Reply from 174.7.244.1: bytes=32 time=8ms TTL=254
Reply from 174.7.244.1: bytes=32 time=8ms TTL=254
Reply from 174.7.244.1: bytes=32 time=14ms TTL=254 - upload test start
Reply from 174.7.244.1: bytes=32 time=14ms TTL=254
Reply from 174.7.244.1: bytes=32 time=12ms TTL=254
Reply from 174.7.244.1: bytes=32 time=13ms TTL=254
Reply from 174.7.244.1: bytes=32 time=13ms TTL=254
Reply from 174.7.244.1: bytes=32 time=16ms TTL=254
Reply from 174.7.244.1: bytes=32 time=12ms TTL=254
Reply from 174.7.244.1: bytes=32 time=11ms TTL=254
Reply from 174.7.244.1: bytes=32 time=12ms TTL=254
Reply from 174.7.244.1: bytes=32 time=13ms TTL=254
Reply from 174.7.244.1: bytes=32 time=14ms TTL=254
Reply from 174.7.244.1: bytes=32 time=12ms TTL=254
Reply from 174.7.244.1: bytes=32 time=13ms TTL=254
Reply from 174.7.244.1: bytes=32 time=15ms TTL=254
Reply from 174.7.244.1: bytes=32 time=11ms TTL=254
Reply from 174.7.244.1: bytes=32 time=12ms TTL=254 - upload test end
Reply from 174.7.244.1: bytes=32 time=7ms TTL=254
Reply from 174.7.244.1: bytes=32 time=8ms TTL=254
Reply from 174.7.244.1: bytes=32 time=7ms TTL=254
Reply from 174.7.244.1: bytes=32 time=7ms TTL=254
Reply from 174.7.244.1: bytes=32 time=8ms TTL=254
Reply from 174.7.244.1: bytes=32 time=7ms TTL=254
Reply from 174.7.244.1: bytes=32 time=8ms TTL=254

 

[Radical_53]

Can you tell me anything about the miraculous "buffer bloat"?

With my connection I always have to "cap" downstream also as even something like a phone running updates could stall the connection and bring everything else to halt.
With all the family using this single connection, I have to use QoS in order to get working results for everyone.

It's really just that possible (random, due to spikes?) latency that led me to further examine the issue. I play online games, games where latency is *very* vital, and my son plays online games, too.
The connection I have here isn't very fast, bandwidth-wise, but it has a very good latency according to tests.
That's the one thing I'd like to "conserve" while gaming, at all times.

@rmerlin: I just tried out your suggestion, thanks again! With that rule set the buffer gets halfed, more or less. It drops from 2300ms to 1200ms. It's still much higher than with the extremely high down-/upstream values (or without QoS altogether) though. Still can't make out what to do with this.