[Release] FreshJR Adaptive QOS (Improvements / Custom Rules / and Inner workings)

[el pescador]

Quick question.
If i turn the QoS off and on at a later time is it correct that it changes form fq-codel to sfq?

Ive followed all instructions and got the script

 

[FreshJR]

@MarCoMLXXV

I use an overhead value of 0 for Comcast.
Comcast, from my expierence, bandwidth limits after the overhead at the modem. So while overhead is present, we are not limited on it.

The webui priorities are still very much in effect. Read my explanation to jack yaz a few pages back.

As for the QOS values, run a ping test from the router while performing a speed test. Your ping should not spike, (10-15ms is acceptable) during the upload or download portion of the test . If it does you have to lower your limits. If it doesn't you can raise them.

As for custom traffic rules, you have to know what recieving port the application is using or the server ip range the traffic is comimg from. Other rules get significantly more complicated. Examples for the common traffic identification methods are included with the script.

Traffic allocation between devices fighting for traffic with the same group is currently performed poorly. I have a fix for that, but I do not intend to provide support or maintence, as it's slightly more complicated to install and possibly troubleshoot.

@el pescador

That is standard behavior

 

[strangeluck]

Thanks for your reaction @john9527. I've read different percentages throughout this thread and elsewhere on the web. Is 90% of max bandwidth enough room? Or should I leave more margin? And do these percentages actually limit the maximum available throughput (i.e. if theoretically only one category is being used, will throughput be actually capped at 90% or will it utilize the maximum available bandwidth?

You will only use bandwidth up to 100% of the value you set manually for your QoS limits. So if for example you have 100mbps service and your manual bandwidth value is set to 90mbps you will be capped at 90mbps bandwidth regardless of how many categories are seeing activity. A good way to determine the best value for your connection is to perform a speedtest on a site like dslreports and monitor the bufferbloat results. First try with QoS disabled, then start adding headroom until you achieve consistently good results.

 

[FreshJR]

A good way to determine the best value for your connection is to perform a speedtest on a site like dslreports and monitor the bufferbloat results.

While dslreports may be easier to use, the pingtest in the routerUI coupled with a simultanous speedtest will be significantly more accurate from my expierence.

EDIT: I take my statement back!

Under DSLReports settings they have an "Hi-Res BufferBloat" toggle.

After turning that on, my results are exactly inline with a webUI or command prompt ping test.
Without that checked, the results weren't really that accurate.

 

[strangeluck]

While dslreports may be easier to use, the pingtest in the routerUI coupled with a simultanous speedtest will be significantly more accurate from my expierence.

True. And didn't mean to step on your toes, I was constructing my own reply when you responded.

 

[Vexira]

Thanks for your reply, @Vexira. So, if I understand you correctly, I should leave the values at 15/150 and only apply the preset overhead? I thought the overhead was a percentage, but I understand from you it's an amount of bytes? Per package or?



I understand what you're saying, but aren't we doing the same thing when editing the variables in the script? It's kinda tricky as I want to get several kinds of traffic an equal amount of bandwidth, but the 'Customize' option in the WebUI doesn't allow to put two categories on the same priority level, yet the script does.

Thanks for the offer. Customizing rules is not for now, currently not really a need for it, but if I can't figure it out, I might reach out to you!

its an amount in bytes per packet transmitted its, more accirate than 90% because it adds a real world value to shaper it tells it to deduct that value, test it and see it it helps with just that if it doesnt re apply the 90% contrary to popular beleif, ive never seen any improvemnt in latecy with my connection set to 90% bandwith though, ive notice adptive qos by defualt, drops avalable bandwith by a notice able amount, but that just me i even tried 85%, my case is diffrent to others, i suspet its more to do with network congestion since we suffer form it a lot here in aus.

 

[MarCoMLXXV]

Thanks for the input, to all who answered my questions. Always nice to have some different views on a certain subject. Having said that, and I don't mean to be ungrateful for your help, the different opinions haven't really made it easier although I am more familiar with the several definitions. Now all I need is some low-traffic network quality time with my router for some trial and error to see what works out best in my situation... To be continued

(you'll still be here, right? )

 

[FreshJR]

Thanks for the input, to all who answered my questions. Always nice to have some different views on a certain subject. Having said that, and I don't mean to be ungrateful for your help, the different opinions haven't really made it easier although I am more familiar with the several definitions. Now all I need is some low-traffic network quality time with my router for some trial and error to see what works out best in my situation... To be continued

(you'll still be here, right? )

So check this explanation out.

View bandwidth limiting / QOS as a funnel. We we pour water inside this funnel. Once we start pouring water inside the funnel faster than it can spill out, it will spill out over the top. The spills are dropped packets which causes terrible bufferbloat.

Your ISP uses this funnel and pipe size to limit your speed according to the plan you pay for. You pay for XX mbits. What adaptive QOS is doing is that it creates our own smaller funnel so the ISP funnel doesn't overflow. When our smaller funnel overflows, we choose non-important traffic to drop, instead of allowing the ISP to randomly drop traffic it chooses. This way we do not receive bufferbloat on priority traffic.

Now diving deeper.

Packets can have various sizes. The largest frame is 1538 bytes (ignoring jumbo packets).
Note this is 1500 packet payload once ethernet overhead is subtracted, or 1460 data payload once IP/TCP overhead is subtracted.

The first 38 bytes is the ETHERNET header.
The next 20 bytes is the IP header (present on every packet).
The next 20 bytes is the TCP header (or 8 bytes if its a UDP header).

This 78 byte per packet is present for everyone. Packet payload can be up to 1500 bytes, but not all packets are that large.

Example1:
Sending TCP packets, total frame size 1538 bytes each, at 100 kB/s

-Sending 65 packets per second.
-Total transfer rate is 100 kB/s
-Data rate is 95 kB/s
-5% bandwidth is used for packet headers

Example2:
Sending TCP packets, total frame size 200 bytes each, at 100 kB/s

-Sending 500 packets per second
-Total transfer rate is 100 kB/s
-Data rate is 61 kB/s
-39% bandwidth is used for packet headers

This 78 bytes per packet overhead IS seen at the router. Our QOS IS calculating for this per packet overhead due to IP/TCP as part of the throughput it is rate limiting.
This per packet overhead is the SAME for every connection since all packets are standard.

Remember the funnel from earlier. On some internet connections the ISP funnel is before the modem. On others the ISP funnel is after the modem.
Now when the modem encodes the the ethernet traffic into the COAX cable (or whatever you may be using), that procedure adds more per packet overhead.
This overhead will be in addition to the ethernet, ip header, tcp header mentioned previously.

This additional overhead, is stripped or added, when traffic leaves or enters the modem respectively.
It is an invisible amount that the router CANNOT see. Even though the router cannot see it, it will still fill up the ISP funnel.

As you can see packet overhead can vary between 5 or 40% of total bandwidth depending on the packet rate. This means it would be inappropriate just to reduce bandwidth by a percent to account for it.

To prevent that, we use WAN overhead to add a byte value to each packet transversing our router to make sure our estimates are accurate. Overall we want to traffic to spill over router side where we can control the spillover in the funnel, instead of spillover ISP side. If extra stuff if being added to the ISP funnel, we have to account for that so it doesn't spillover.

Now when I mention comcast, I claim that comcast placed their ISP funnel after the overhead is stripped for incoming traffic, or before overhead is added at the modem. It is appropriate for me to use ZERO for ISP overhead. Not all providers do the same with their funnels, so with some, you gotta account for the invisible to the router overheads with the WAN OVERHEAD setting.

So its not conflicting opinions, its just different behaviors depending on the ISP. If someone could create/host a speedtest program that tests throughput with both small and large packets (instead of just large packets), WAN OVERHEAD could be tested.

Note:
If you pay for 30mbps, and you speedtest exactly at 30mbps, it is clear that there is ZERO overhead present, since it would have to eat some of that 30mbps.

Funfact:
When you run a speedtest it counts the standard ~78byte per packet overhead as part of the total available bandwidth result.

Yet when you download a file, it will only counts data throughput, as headers have nothing to do with the file.

So speedtest correctly shows 30mbps of available bandwidth, a max possible file (data) transfer speed is 28.5mbps due to required overheads.

@strangeluck
I take my statement back about DSLReports bufferbloat being inaccurate.

Under DSLReports settings they have an "Hi-Res BufferBloat" toggle.

After turning that on, my results are exactly inline with a webUI or command prompt ping test.
Without that checked, the results weren't really that accurate.

 

[MarCoMLXXV]

Thanks for the excellent explanation, @FreshJR. I really appreciate the time and effort you took to write this and it explains most of things I wasn't fully understanding yet.

One thing, that's why I mentioned the contradicting opinions: I have a 150/15 Mbit plan. When doing speedtesting on a quiet home network, results vary between 142 and 155 Mbit downstream and 14.6 to 16.3 Mbit upstream, somewhat lower around the busy hours (it's a cable plan, so the amount of active users in the neighbourhood affects the bandwidth available slightly). When I read 18 bytes per packet versus ~90%, that's a huge difference in margin, yet it sounds like other users both qualify that as the margin needed. But in Dutch we have saying, which seems applicable here: 'Dat is appelen met peren vergelijken', meaning literally 'That's comparing apples with pears'. It sounds (to me at least) as two totally differently variables, which both can influence performance. I don't need an ultra-low latency, no fanatic gamers here, just my 9 year old playing Roblox, which presumably would perform the same on a 56K modem, if I'm looking at the stats from his system. I do want to get rid of the lag on the local network when he's watching youtube or netflix, or my wife is watching VoD on her phone. I mainly surf the web, read and occassionaly download a new Linux distro.

Given the fact that during business hours (when I'm home alone, as I'm disabled) or at night when I can't sleep my speedtests exceed the maximum up- and download speeds of my plan by 5 to 10%, it would be safe (if I understand you correctly) to set the WAN packet overhead to 0?

I did several different speedtests after your previous post, using speedtest-cli from the router (currently the fastest wired connection to the outside, as my main laptop is still, sigh, out for repair), and I used Ookla Speedtest from my iPhone Se with full coverage. while pinging www.google.nl (as I know Google has a giant server farm in the Netherlands to serve Western Europe) in both cases, as you suggested, from the router itself. Out of 100 pings I have two or three spikes at most during the tests, mainly around 30ms, incidentally up to 60 ms. The rest of them is around 9 ms. I've tried 150/15, 135/14, 130/13 and 120/12. The results are basically the same, the peaks seam to occur at the beginning and the end, and around the moment the speedtest switched form download to upload test.

I used dslreports' speedtest and the results are all over the place, I'd almost say unreliable, even though I've provided them with the actual geo location for finding local servers. When initiating the test, Dutch servers (as expected) seem most responsive in the primary pie graph, yet it decides to use servers at longer distances (surround countries), causing a lot of variation in the results provided and bufferbloat is through the roof... I don't know what's causing this much bufferbloat.

For example (even though these are way smaller packets) when using NTPd by @kvic, there's barely any jitter, I have a deviation lower than +/- 80 microseconds with 8 NTP-servers in The Netherlands as well as several surrounding countries, which are polled every 5 minutes 24/7. Jitter is nihil.

What to do? What are the most basic settings for maximum utilisation of available bandwidth, without experiencing internal lag/congestion? Zero WAN Overhead and Maximum Bandwith values as offered by my ISP's plan? Or should I keep (some) margin in there, like 14/145?

Thanks in advance!

 

[MarCoMLXXV]

In addition to the novel above, some illustrations (Not sure wheter they're helpful, but what's a book without some images ):

1. QoS disabled entirely:

2. RMerlin QoS enabled, discipline fq_codel, WAN Overhead = 0 Upload Max = 15 Mbit, Download Max = 150 Mbit:

3. Fresh_JR QoS enabled, discipline fq_codel, WAN Overhead = 0 Upload Max = 15 Mbit, Download Max = 150 Mbit

These test were just done (22:30CEST, which basically still quite a busy time of day, people at home, watching VoD etcera, but outside office hours). I ran them all three times, these are the best scoring results, but bufferfloat varies between every following test.

 

[FreshJR]

In addition to the novel above, some illustrations (Not sure wheter they're helpful, but what's a book without some images ):

These test were just done (22:30CEST, which basically still quite a busy time of day, people at home, watching VoD etcera, but outside office hours). I ran them all three times, these are the best scoring results, but bufferfloat varies between every following test.

For specific bufferbloat performance you can click results and view the advanced results.

My bufferbloat is super constant and low (~ 5-10ms above idle) on the download portion. I would give that an A+.
It just goes a little crazy on the upload portion (~20-60ms above idle)

For me, I think the poor(er) upload performance is killing my grade.

My ping in webUI or command prompt doesnt spike during the upload portion to reflect DSLreports findings, so I dont see where DSLreports is reading this bufferbloat from.

Also, I wonder if the initial bufferbloat spike can be minimized by reducing cburst. I honestly didn't bother experimenting since compared to my non-QOS results, the effect is night and day.

20ms vs 1000ms ping, i didn't bother investigating further.

@pattiri
I could not get your steam issue repeated.
I had steam maxing my connection at 3.6MB/s with gaming 2nd priority.

I started a speedtest and it got ~2mbps ( slightly above the 5% of 30mbps allotted)
I started a fast.com (netflix speedtest identified as streaming) and got ~8mbps (slightly less than 30% of 30mbps allotted)

 

[Vexira]

I s there at way to get a tablet to default to highest quality, video on YouTube, on a tablet, sees that even with out any downloads, for some reason I get lowest quality by default which is odd, considering my 95/38 vdsl2, id think that even with gaming at highest priory if theirs no downloads or gaming traffic, qos should give video streaming the most bandwidth, I have it set to game priority at the moment, short of setting it to video streaming as highest, I'm perplexed is it possibly that android YouTube app traffic is not correctly classified as video streaming?, could this be the same thing that some of the IPTV channels I have on my fetch tv box stream at low quality, while others are in full HD? its been bugging me for a while. I believe that you may have some possible insight into the matter.

 

[maxsteel]

A Suggestion:

- Gaming Protiroizing

- VPN Recognize (Astrill Applet)

- NZB Fix (port 563)

These three items I recall right off the top of my head in regard of optimizing in next script update. I know there are a lot of things you're working on @FreshJR, so thank you in advance.

 

[beboptrumpet]

For those who haven't noticed, On dslreports speed test, if you go into options, there is a box for hi-res blufferboat. Give you a more detailed test result for what we are doing here.

 

[Deplorable]

What do I do if QOS doesn't appear to be doing ANYTHING? I am on spectrum formerly timewarner and no matter what settings I use QOS the bufferbloat test even with high res is an F.

 

[FreshJR]

@MarCoMLXXV
Your 2nd/3rd speedtest result were perfect.
Your results will vary between speed tests results but overall A's and B's are great.
You can rerun the test with hi-res bufferbloat, and that variance will mostly disappear (it did for my tests) and start to match webUI/command prompt results.

I wouldn't limit bandwidth more than you need too, so it looks like 15/150 is appropriate for you. While for me the difference between 29 & 30 is night and day. The initial bufferbloat spike you saw before it rapidly approaches back to +- 15ms of idle is normal.

A Suggestion:

- Gaming Protiroizing

- VPN Recognize (Astrill Applet)

- NZB Fix (port 563)

These three items I recall right off the top of my head in regard of optimizing in next script update. I know there are a lot of things you're working on @FreshJR, so thank you in advance.

Your requests are very user specific.
I don't want performance loss present from a large chunks of custom rules when majority of the users don't have the same needs.
Use custom rules for your specific needs. Asus/TrendMicro is responsible the original large table of rules and they provide updates. This script is a work around for more desirable characteristics and a handful of custom rules.

For those who haven't noticed, On dslreports speed test, if you go into options, there is a box for hi-res blufferboat. Give you a more detailed test result for what we are doing here.

Yes, ever since I noticed this setting, I can finally recommend and agree with DSLReports results.

What do I do if QOS doesn't appear to be doing ANYTHING? I am on spectrum formerly timewarner and no matter what settings I use QOS the bufferbloat test even with high res is an F.

Are you using manual bandwidth settings with lower limits than your max ISP speed.
There was a rule missing to account for https traffic. DSLreports has an https version on their website. Since this https traffic qos bypass issue has been brought up (and answered) multiple times, I included the https rule with the 1.91 script update. No other changes have been made.

 

[Deplorable]

@MarCoMLXXV
Your 2nd/3rd speedtest result were perfect.
Your results will vary between speed tests results but overall A's and B's are great.
You can rerun the test with hi-res bufferbloat, and that variance will mostly disappear (it did for my tests) and start to match webUI/command prompt results.

I wouldn't limit bandwidth more than you need too, so it looks like 15/150 is appropriate for you. While for me the difference between 29 & 30 is night and day. The initial bufferbloat spike you saw before it rapidly approaches back to +- 15ms of idle is normal.




Your requests are very user specific.
I don't want performance loss present from a large chunks of custom rules when majority of the users don't have the same needs.
Use custom rules for your specific needs. Asus/TrendMicro is responsible the original large table of rules and they provide updates. This script is a work around for more desirable characteristics and a handful of custom rules.



Yes, ever since I noticed this setting, I can finally recommend and agree with DSLReports results.



Are you using manual bandwidth settings with lower limits than your max ISP speed.
There was a rule missing to account for https traffic. DSLreports has an https version on their website. Since this https traffic qos bypass issue has been brought up (and answered) multiple times, I included the https rule with the 1.91 script update. No other changes have been made.

I got it squared away now. Only problem I really have now is with openvpn client in router and qos as in can they be used together. It seems with VPN on it defaults to my upload speed rather than download and severely limits bandwidth. Someone in another thread suggested assigning a category for VPN in your script but I'm not using a tunnel from a particular device through the router. I am setting a vpn tunnel for the router itself through which all traffic from the network goes. QOS appears to still correctly identify categories such as netflix and youtube or web surfing just as though VPN was off it just caps bandwidth at what seems like my upload speed rather than download. In another thread I saw where people were having issues with QOS and VPN where it was using their upload speed rather than download speed and it appeared to be a known issue but that was two years ago. Still based my max speed seeming to match upload speed and QOS still correctly identifying the traffic categories with or without VPN it seems like that apparent issue still exists or it is just a huge coincidence. Thoughts? Not possible to use VPN and QOS at same time?

 

[beboptrumpet]

Another good way to test is on your router GUI,
Go to network tools and set it to ping
I usually select google & Set number of pings as 100
-Start ping test then immedatly start a speed test on a different machine
Watch how the ping time changes as the speed test runs.
I like upping the ping number & using the much longer nperf detailed test. Again, watch how the ping times change on the ping test page

 

[Lacrocious]

Hi FreshJR,
I love the script, but need a tweak or two. I mainly use QOS to control backups. My backups end up filling my upload pipe, which affects download pipe negatively. To backup "fast" when traffic is light, I don't want to just limit my backup software to a low speed.

I was using CrashPlan - but I am looking to shift to BackBlaze B2 and ARQ for the client. I have the BackBlaze IP ranges (see: https://help.backblaze.com/hc/en-us/articles/217664588) .

With your default script, all ARC client backups to BackBlaze B2 fall into NetControl. I know you commented on this at various times on how hard it is to identify https traffic that is backup/file transfer traffic. Since I know the IP Addresses, I wanted to do some custom rules. I created rules similar to the following for each IP range:

#  iptables -A POSTROUTING -t mangle -o eth0 -d 206.190.208.0/21 -j MARK --set-mark ${Downloads_mark}

I added the rule just after ##Upload (Outgoing traffic...) and before the 4 ${tc} filters in that section. Does that make sense? It appears to work and most backup traffic seems to be going into the File Transfer section - but I wanted to run it past you as there haven't been many examples of IP Range rules in the thread. Thank you.

- Ron

 

[FreshJR]

@Lacrocious / Ron

Almost but not quite.

(1) Most important is that you have to remove the preceding # from your desired rules.
# is the comment character, any text starting with that is completely ignored when the script is executed.

(2)You are using the wrong rule.
The iptables rules on the upload portion are traffic originating from LAN ranges. Aka, I want all my upload traffic from 5 of MY specified LOCAL devices to always go into a container.

You need the TC rule that deals with WAN destination ranges. (WAN means wide area network, aka anything not on your LAN/local area network/local device). The WAN rule means that you want any upload traffic destined for that WAN iprange to go into a QOS container.

You need this one

${tc} filter add dev eth0 protocol all prio 1 u32 match ip dst 206.190.208.0/21 flowid ${Downloads}

(3) Also if you do need to use an iptables for local devices in the future, look again and you will see that those rules come in sets of two.
First you have a iptables delete and then you hav your iptables append.

You would be missing the delete before the append if it was the correct rule. Iptables allows duplicate rules, so everytime the script runs it needs to delete the old one if present before recreating it again. Slightly sloppy, but I haven't thought of a better way.

(4) Are you on v1.91 by chance? The sections where the rules go are now marked more explicitly.


That rule you have shouldn't be responsible for the identification.

http://www.ipaddressguide.com/cidr is a CIDR calulator.

Your example includes 206.190.208.0 - 206.190.215.255

You can also consider sticking it in the defaults container. That way it won't even choke your other file downloads