[Release] FreshJR Adaptive QOS (Improvements / Custom Rules / and Inner workings)

[Sinner]

Question have you actually read the article, and secondly have you set your per packet overhead value?
I'm just qurious that's all.

Well ive done almost nothing but read articles on this for months and i dont bother with the per packet overhead. Without turning this into forum pvp are you saying theres no logic to what i explained?

 

[Vexira]

Well ive done almost nothing but read articles on this for months and i dont bother with the per packet overhead. Without turning this into forum pvp are you saying theres no logic to what i explained?

No I'm just qurious about if it might help with the dips I believe you mentioned, if you read Merlins posts on it it actually helps qos shape for your connection type.

I'm a heavy gamer both Xbox and PC so yes I understand what you're trying to articulate.

 

[Sinner]

Im in a dbl nat environment. Calculating overhead would be rocket science and more than likly pointless due to the fact that adjust for overhead a hop or two up the line is like adjusting for overhead for hop15 between you and the game server your playing on. Yeh could be done but then your getting into adjusting differently for every web page you visit. Not into that =)

 

[Sinner]

Well my issues with the dips are much less scientific. Its summer and im on 10mb wireless internet. Even heat outside my be a factor

 

[Vexira]

Im in a dbl nat environment. Calculating overhead would be rocket science and more than likly pointless due to the fact that adjust for overhead a hop or two up the line is like adjusting for overhead for hop15 between you and the game server your playing on. Yeh could be done but then your getting into adjusting differently for every web page you visit. Not into that =)

It's meant to adjust only for the connection that your using as in DSL, cable, fiber.

 

[Sinner]

It's meant to adjust only for the connection that your using as in DSL, cable, fiber.

I get that but technically its gigabit wired then.. the next hop is the wireless internet. And the next hop is the isp shared whatever equipment up the street.

 

[Vexira]

Well my issues with the dips are much less scientific. Its summer and im on 10mb wireless internet. Even heat outside my be a factor

Ahh I see what you mean, is there no hard wired connections available, eg DSL or fiber?

 

[Sinner]

No just the wireless in this area. But see what i mean? If you have one ruter behind another is a perfect eample. Router2 connected to router1 1000mbs wire. Router1 wireless 10mb. If your going to adjust overhead based on router1s connection you might as well be adjusting for your isps overhead as well

 

[Sinner]

Its late i need to crash but... imo the best qos setting or recomendation would actually be to look at the speed tests and avg out all the bottom dips and set that in the gui for almost always perfect qos

 

[FreshJR]

ugh ...

@Vexira, just to be clear, I DO NOT recommend a static 0.95 speedtest result to be entered. (What are you even entering sustained speeds or advertised speeds ?!?)

Each user case is different, and 85-95% of sustained speeds has proven to be an excellent range to start tuning,

The two script lines that perform the reduction were solely given to you simply since you made that request.
If I thought the built in reduction was a great idea, I would of added it into the script.

@Sinner's understanding of QOS performance is completely correct.

Once again, WAN overhead almost becomes almost completely irrelevant when entering a portion of an achievable speedtest result into the WebUI.
(The speedtest already has the overheads included while delivering your data!)
I say almost irrelevant, since figuring out and defining your WAN OVERHEAD would help in certain network circumstances, but ultimately the overall proportion of small sized packets to fully sized packets will be VERY VERY minuscule due present day monstrously high provisioned bandwidth.

Sure if you knew

1) WAN overhead
&&
2) Your ISP's **provisioned** speeds (not advertised)
&&
3) Your ISP's network was completely stable

then, yes, could enter those 99% of your provisioned speeds into WebUI. You are completely correct on that end. (You would also have to match bursts)

But

1) You don't know what you are provisioned at. (For example Comcast provisions 20% higher than advertised.)

2) You don't know your WAN overhead. (For example Comcast, does not limit on their encountered overheads, and there are no tests hosted online to determine a users particular situation)

3) You don't you know if your ISP doesn't have other network features active such as a high burst (speedboost) to give your file transfers an initial kick to the butt for the first XX seconds and make everything snappy.

4) You are not aware of any issues along the way that could cause the link rate to drop lower than provisioned

Your method has a metric ton of unknowns!!!

The speedtest method will always gets results since it is based on observation of real performance. The speedtest methd even had the WAN overhead effecting the result during the process.

Unless ISP's start publishing techincal documents/specs when selling internet and then enforcing those speeds as advertised instead of **up to** jargon, then advertised speeds are moot. The entire network chain is simply too damn complex!!

According to lede if using fq-codel it's 95% of speed test values and 80% of advertised values plus you set the over head value.

The % thing was there before they worked out the overhead values for connections.

TLDR:

1) 85-95% of speedtest values I can agree with

2) advertised speeds are completly moot (read this entire post)

3) Entering WAN Overhead + Provisioned Speeds would be nice, but what user has access to this data + the other unknowns.

Question have you actually read the article, and secondly have you set your per packet overhead value?
I'm just qurious that's all.

Before you tell other people if they read your lede forum posts, did you read my responses I made out directly to you?!? In this post and earlier in the thread.

https://www.snbforums.com/threads/wan-packet-overhead-for-qos.40024/page-3

---

This was a lot of information, but I repeated my viewpoint many many times.
I fully welcome a counter argument based on theory or experience.

Until then, my go to response will be the read the initial posts. Those are pretty clear and defined!

 

[Vexira]

ugh ...

@Vexira, just to be clear, I DO NOT recommend a static 0.95 speedtest result to be entered. (What are you even entering sustained speeds or advertised speeds ?!?)

Each user case is different, and 85-95% of sustained speeds has proven to be an excellent range to start tuning,

The two script lines that perform the reduction were solely given to you simply since you made that request.
If I thought the built in reduction was a great idea, I would of added it into the script.

@Sinner's understanding of QOS performance is completely correct.

Once again, WAN overhead almost becomes almost completely irrelevant when entering a portion of an achievable speedtest result into the WebUI.
(The speedtest already has the overheads included while delivering your data!)
I say almost irrelevant, since figuring out and defining your WAN OVERHEAD would help in certain network circumstances, but ultimately the overall proportion of small sized packets to fully sized packets will be VERY VERY minuscule due present day monstrously high provisioned bandwidth.

Sure if you knew

1) WAN overhead
&&
2) Your ISP's **provisioned** speeds (not advertised)
&&
3) Your ISP's network was completely stable

then, yes, could enter those 99% of your provisioned speeds into WebUI. You are completely correct on that end. (You would also have to match bursts)

But

1) You don't know what you are provisioned at. (For example Comcast provisions 20% higher than advertised.)

2) You don't know your WAN overhead. (For example Comcast, does not limit on their encountered overheads, and there are no tests hosted online to determine a users particular situation)

3) You don't you know if your ISP doesn't have other network features active such as a high burst (speedboost) to give your file transfers an initial kick to the butt for the first XX seconds and make everything snappy.

4) You are not aware of any issues along the way that could cause the link rate to drop lower than provisioned

Your method has a metric ton of unknowns!!!

The speedtest method will always gets results since it is based on observation of real performance. The speedtest methd even had the WAN overhead effecting the result during the process.

Unless ISP's start publishing techincal documents/specs when selling internet and then enforcing those speeds as advertised instead of **up to** jargon, then advertised speeds are moot. The entire network chain is simply too damn complex!!



TLDR:

1) 85-95% of speedtest values I can agree with

2) advertised speeds are completly moot (read this entire post)

3) Entering WAN Overhead + Provisioned Speeds would be nice, but what user has access to this data + the other unknowns.




Before you tell other people if they read your lede forum posts, did you read my responses I made out directly to you?!? In this post and earlier in the thread.

https://www.snbforums.com/threads/wan-packet-overhead-for-qos.40024/page-3

---

This was a lot of information, but I repeated my viewpoint many many times.
I fully welcome a counter argument based on theory or experience.

Until then, my go to response will be the read the initial posts. Those are pretty clear and defined!

*sigh* here we go again.

I was referring to the bandwith reduction rule you gave me to change the value to 95% of the speed test values that are entered into the gui , and the 95% that is suggested by lede and yes I read what you posted. I was if you actually read a few posts back, maby I'm too cryptic for others to understand offering the alternative of giving the rules you have me to reduce the gui bandwidth to 95% even less.

In my own personal case I'm aware that my ISP uses ipoe (dhcp) and mac based authentication for connections, hence why I use Mac cloning to get VoIP to work, I'm also aware that vectoring is enabled on my vdsl2 connection, I'm from Australia I'm on one of the connections upgrades the government gave us known as the national broadband network

I believe that @Sinner miss undstood what I wrote, originally I was merely suggesting that the post I responded to could you the reduction rules in the script to allow for the full speed test values to be eneterointo the gui so the reduction rules could preform thier magic as an alternative to re calculating the values every time a speed test is run again, I'm going to assume that wasn't articulated clearly enough by me or it was miss read.

He mentioned spikes so I asked if he set his overhead, I actually ran a speed test on dsl reports and discovered that at the time of running it with just the overhead value alone there was a form of reduction in comparison to the values that I got with no qos set and so overhead set, it was only a suggestion.

I'm also aware that if I had a direct fiber connection I'd be getting full ISP spees since I'm actually very close to the node cabinet, my syc speeds in the modem are 100/40 but due to the age of my copper line and other various factors including the nature of vdsl my real world values are more closer to
95/37 or the raw values from speedtest.net 95.11/37.33 from the last speed test I ran I'm also running an mtu of 1500, interleaving is disabled vectoring or I believe it's called g vectoring is enbaled, yes unlike most people I have a good idea of what my connection is like I might not quite know exactly the specifics.

In regards provisioned bandwidth my connection is according to my ISP capable of 118 on the down load speeds according to thier own speed test, but I'm capped at 100/40 even though myine is cable of 120/50 according to them and my modem stats, like I said if I had a fiber connection I would be getting 100% advertised speeds, unfortunately I'm on vdsl2 so I suffer from losses due to the copper cable that is 20+years old going to my home.


@FreshJR He didn't understand what I was trying to explain originally about the rules of bandwidth reduction to another user which probably brought about they confusion. Yes I did read the links you sent me and thank you for that it explains alot about the accounting for packets. . I originally assumed that @Sinner was on some form of dsl, cable or fiber hence why I asked about the overhead.

 

[Vexira]

No just the wireless in this area. But see what i mean? If you have one ruter behind another is a perfect eample. Router2 connected to router1 1000mbs wire. Router1 wireless 10mb. If your going to adjust overhead based on router1s connection you might as well be adjusting for your isps overhead as well

You just described a double nat scenario, unless your referring to the ISP router. Then again the over head is intend for a different connection type entirely, which means that our whole conversion is based on a misunderstanding.  *sigh *

 

[Vexira]

@FreshJR the 80% is if your using the the advertised speeds 95% for actual speedtest.net values I remember reading it, I suppose it doesn't hurt to start at 80% and go upwards to see if it helps with buffer bloat, I might actually test that soon on my own connection.
Also again that's for the bandwidth reduction rule it works a charm on my pings.

And lastly I have no idea where that came from "just to be clear, I DO NOT recommend a static 0.95 speedtest result to be entered. (What are you even entering sustained speeds or advertised speeds ?!?)"

It seems to me that you are confused as well I'm sure you already know me better that that I don't remember to just ad a static value I was suggesting that the other user could use the rules that I requested and you provided to do the reduction.

"The two script lines that perform the reduction were solely given to you simply since you made that request.
If I thought the built in reduction was a great idea, I would of added it into the script."

I don't see why it isn't a good idea personally, since it sets a base line and if its not to someone's tastes then they can adjust it.
also the reason I requested it was because of this if you look at the bandwidth they set its whole numbers.
Secondly because speedtest.net tends to feed out none whole number values making it a pain to get 95% e.g .if speed test.net tells me my down speed is 95.33 that if you multiply it by 0.95, comes out as 90.5635. that gives me a headache
https://www.asus.com/au/support/FAQ/1008718/
Thirdly I am grateful to you for giving me the commands, it make life easier a little

I'm only using 95% an the moment because I've not see a difference between using it and 90% in my own personal case, and its what the lede document suggested, for some one else 90 might be more beneficial.

@Sinner This is what I mean by rules for band with reduction.
DownCeil="$(expr ${DownCeil} \* 95 / 100)"
UpCeil="$(expr ${UpCeil} \* 95 / 100)"

 

[Sinner]

You just described a double nat scenario, unless your referring to the ISP router. Then again the over head is intend for a different connection type entirely, which means that our whole conversion is based on a misunderstanding.  *sigh *

I was merely explaining the primary reason I don't bother with overhead.. due to my double nat scenario. but long story short I still think it much easier to just speedtest then lower the results slightly and enter into the qos gui.. done

 

[skeal]

Hey @FreshJR and all you guys!! I'm using the alternative script and I have an observation. When I enable the gaming rules (I have 1 gaming ip /32) and turn on the computer, that is the gaming machine, almost all traffic is classified as gaming, but upload only no download. This happened as I was updating games in Steam, and the computer was probably updating itself as well. There was some File transfer classed traffic but not much. The majority of traffic coming from and going to, the gaming rig, was classed as uploading Game traffic. Is this intended or do I have something wrong? I disabled the rules and it seems to have stopped this problem.

 

[dtaht]

@FreshJR - have you taken a look at sch_cake yet? One feature I'm proud of is the docsis and adsl modes where (at least in the lab!) we can hit 100% of the set rate on the uplink. We still recommend 85-90% on shaping the downlink, though.

See: https://arxiv.org/pdf/1804.07617 for some details about the per host fq through nat, etc.

I like to think that per-host fq will reduce the need for extensive gamer shaping rules, and there are also new opportunities to do those better. But your community is in a way better position to evaluate this than we are.

It's been in openwrt for a while and just hit linux net-next.

-- dave taht, cofounder bufferbloat project.

 

[RMerlin]

No cake in 2.6.36.

Sent from my Nexus 5X using Tapatalk

 

[dtaht]

@RMerlin, lordy... this is 2.6 based still? I feel your pain. I guess my question then becomes what are the underlying qdiscs adaptive qos, traditional qos, and freshJR's stuff uses?

 

[johnathonm]

I'm watching this thread descend into anarchy and I think it has a lot to do with FreshJR's very poor forum instructions. I believe that it might be helpful for us to revise and standardize things rather than throwing out a lot of abstractions. Here are my questions and then thoughts:

Questions/Suggestions:
FreshJR's instructions are fairly complex and not very user friendly, especially with the use of scp and the requirement to convert the text file with dos2unix. Couldn't this be done in notepad++ under windows or in nano on the router, so that the conversion step could be eliminated.

Secondly, rather than use scp, for the sake of being user friendly, would it not make more sense to use Winscp to upload the files along with Putty (or use the built in putty within Winscp to install the files)? Or keep it simpler, like Skynet, use curl along with a script to install?

Third, everyone is arguing over what percentage of bandwidth to deduct from their speedtests. It might be helpful to first, pick a speedtest site (I think dslreports speed test is the most reliable and through), then over the course of a few hours (say 3) take measurements. If we aim for a threshold of aggregate data over say two days and maybe 4-5 speed tests per day, we would have say 10 points of data for our speeds. Then we could add up the values for upload/download and calculate the median value for each then set our deduction from there.

Moreover, perhaps we should agree on a universal percentage that we need to reduce from our calculated median bandwidth. I would argue that since 95% is too little and others think 85% is too much, then perhaps we should agree on a standardized value of 90% of the median values. I would think this would cover the needed overhead and it would move us away from the arbitrary DD-WRT range that is the current model.

Personal Questions:

FreshJR's instructions leave out quite a bit as I mentioned above. I want to make it clear this is constructive feedback, not destructive. I have some questions of my own.

First, after properly installing the script and have enabled QOS we need to select the following:

Adaptive
Manual Setting
fq_codel
Our Wan Packet Overhead (this is being argued to be irrelevant, if so then that needs to be stated).
What is ATM?
Input our pre-calulated median upload and download bandwidth.
Then the part which I have no clue about:

We select customize. Now how is this supposed to be setup? What if I changed values in the script? How does this impact the script itself when there are hardcoded values built into the router? This isn't clear at all in the instructions.

 

[Sinner]

I'm watching this thread descend into anarchy and I think it has a lot to do with FreshJR's very poor forum instructions. I believe that it might be helpful for us to revise and standardize things rather than throwing out a lot of abstractions. Here are my questions and then thoughts:

Questions/Suggestions:
FreshJR's instructions are fairly complex and not very user friendly, especially with the use of scp and the requirement to convert the text file with dos2unix. Couldn't this be done in notepad++ under windows or in nano on the router, so that the conversion step could be eliminated.

Secondly, rather than use scp, for the sake of being user friendly, would it not make more sense to use Winscp to upload the files along with Putty (or use the built in putty within Winscp to install the files)? Or keep it simpler, like Skynet, use curl along with a script to install?

Third, everyone is arguing over what percentage of bandwidth to deduct from their speedtests. It might be helpful to first, pick a speedtest site (I think dslreports speed test is the most reliable and through), then over the course of a few hours (say 3) take measurements. If we aim for a threshold of aggregate data over say two days and maybe 4-5 speed tests per day, we would have say 10 points of data for our speeds. Then we could add up the values for upload/download and calculate the median value for each then set our deduction from there.

Moreover, perhaps we should agree on a universal percentage that we need to reduce from our calculated median bandwidth. I would argue that since 95% is too little and others think 85% is too much, then perhaps we should agree on a standardized value of 90% of the median values. I would think this would cover the needed overhead and it would move us away from the arbitrary DD-WRT range that is the current model.

Personal Questions:

FreshJR's instructions leave out quite a bit as I mentioned above. I want to make it clear this is constructive feedback, not destructive. I have some questions of my own.

First, after properly installing the script and have enabled QOS we need to select the following:

Adaptive
Manual Setting
fq_codel
Our Wan Packet Overhead (this is being argued to be irrelevant, if so then that needs to be stated).
What is ATM?
Input our pre-calulated median upload and download bandwidth.
Then the part which I have no clue about:

We select customize. Now how is this supposed to be setup? What if I changed values in the script? How does this impact the script itself when there are hardcoded values built into the router? This isn't clear at all in the instructions.

I totally hear what your saying bro.. Everyone wants an easy 1 solution fits all setup but there isn't one. Most of Freshs stock settings and recommendations are for the most part just "good" for most users. The reason is because everyone's specific qos needs, network setup and internet speeds and reliability are all different.

The last month or so my conn has been on average 40% of what it normally is so even 80% in qos is useless for me right now. Qos can only work if the setting is lower than what you CAN achieve at this moment. so setting it even to 41% when im only achieving 40% of my normal speeds is exactly the same as having QOS off.

As for the install method I personally would love a 1app edits/installs but im not familiar enough with Linux to know which that might be. Please share some ideas on that

Fresh has a pretty comprehensive guide on custom rules as well, some of which is a bit comfusing but the new commands he added to the script are awesome help with that for me so im happy.

Overall qos is a complex thing that needs to be setup at a minimum slightly differently per user so one setup wont be perfect for everyone.