[Release] FreshJR Adaptive QOS (Improvements / Custom Rules / and Inner workings)

[Tekneek]

scp is scp, regardless of the name of the application. Use whatever flavor of scp you want, it all ultimately does the same thing.

https://en.wikipedia.org/wiki/Comparison_of_SSH_clients

I have secure copied the files over and used dos2unix. I have also copied and pasted them into a properly named file opened with nano. If people already know how to do those things, they are probably doing them.

ATM is Asynchronous Transfer Mode. As I understand that, it is relevant for ADSL, ISDN, etc.

 

[johnathonm]

I totally hear what your saying bro.. Everyone wants an easy 1 solution fits all setup but there isn't one. Most of Freshs stock settings and recommendations are for the most part just "good" for most users. The reason is because everyone's specific qos needs, network setup and internet speeds and reliability are all different.

The last month or so my conn has been on average 40% of what it normally is so even 80% in qos is useless for me right now. Qos can only work if the setting is lower than what you CAN achieve at this moment. so setting it even to 41% when im only achieving 40% of my normal speeds is exactly the same as having QOS off.

As for the install method I personally would love a 1app edits/installs but im not familiar enough with Linux to know which that might be. Please share some ideas on that

Fresh has a pretty comprehensive guide on custom rules as well, some of which is a bit comfusing but the new commands he added to the script are awesome help with that for me so im happy.

Overall qos is a complex thing that needs to be setup at a minimum slightly differently per user so one setup wont be perfect for everyone.

I guess I'm just not smart enough to be messing with the QOS stuff. I think the ultimate failure is in how ASUS has setup their QOS page and categorization. You're right there is not a one size fits all solution but there has to be an algorithm that can be used as a baseline by the average user. This all could be handled in an shell script install like we have seen with skynet or AB solution. There are even public ipref servers so during the install the script could download ipref and run benchmarks against the server(s) to determine a baseline value. Yes, it wouldn't be perfect, but right now we're really flying very blind and debating percentages. I wish that Merlin would build in more congestion algorithms and then I wouldn't even want to go near QOS. I am only looking for a reduction in bufferbloat.

 

[johnathonm]

I think, for me, I am just not smart enough to get what's going on with this as a whole. I appreciate all the work FreshJR put into the script, it was clearly a massive undertaking. I just simply don't understand and I feel really dumb for it. I thank you all for listening. Good luck and Godspeed.

 

[Tekneek]

I think, for me, I am just not smart enough to get what's going on with this as a whole. I appreciate all the work FreshJR put into the script, it was clearly a massive undertaking. I just simply don't understand and I feel really dumb for it. I thank you all for listening. Good luck and Godspeed.

Break down each step and develop an understanding. Trying to understand it as a whole isn't going to help. Take one step at a time.

Some reference for most of it is available elsewhere (try Google/duckduckgo.com/etc).

Don't worry about thinking other people understand it better or faster. Take your time and move at your own pace.

 

[FreshJR]

@johnathonm

It is possible to set up a one line install and have the script be setup with default values, but then if you need to make any custom rules you will still need to go back to the original install instructions consisting of using Notepad++, Pscp, & Putty. So might as well use that method in the beginning.

Sure if I implemented a UI then a one line install + editing via UI solution would be possible and nice, but that is simply too much work. It would be way more than you taking the 5 minutes to

1) make an edit with Notepad++
2) upload with pscp
3) set permissions with putty

-Setting EOL conversion via Notepad++ is more steps then a 1 line command in putty.
-Making edits live on the router with nano or vim is significantly less intuitive than editing in Notepad++

--

Sure I can implement a single bandwidth reduction value and have people simply enter peak speeds, but then everyone is losing performance.

As for the your customize order question .... actually read the install instructions!!

3) I recommend the following QOS traffic priority list.

VoIP
Gaming
Others
Web Surfing
Video and Audio Streaming
File Transfers

Even if you decide your own order, it should also work well enough due to other script modifications related to bandwidth allocation.

As for this question

Our Wan Packet Overhead (this is being argued to be irrelevant, if so then that needs to be stated).
What is ATM?

Setting these parameters correctly IS beneficial, but the values are dependent on your ISP's network implementation and that information is not available to end users.

Users are free to google or ignore these parameters. I am in the same situation as you when it comes to these settings, and simply have them turned off, since there is no information for my (or any) particular ISP's actual implementation. In the end feel free to use the recommended preset or leave WAN overhead off, as both solutions are ultimately a guess.

Don't worry that this setting is potentially incorrectly set ( turned off / underestimating / overestimating ) as the end results from this setting will be mostly washed away when fine tuning bandwidth via this step.

1) Use manual bandwidth with limits set to 85-95% of your non-throttled speedtest results

To zone in on the ideal bandwidth setting within this range you need to be aware of these relations .....

--

Anyone is free to fork it and implement and a GUI on top of it. In fact a user already has and implemented 5 hardcoded spots for port rules.

--

Once again the initial post is the inner working, install, and advanced rules. You are free to just follow the install instructions, no one says you have to understand the inner workings or apply advanced rules.

The install instructions are literally 5 lines of code you copy paste ... Feel free to PM a clearer version of the install instructions....

--

Much of the "anarchy" was simply arguing different viewpoints on settings for ideal QOS performance. I welcome any discussion (disagreement included) since ultimately it either leads to a better solution or a better understanding.

The real "anarchy" were my 2-3 outbursts at people not reading the initial posts, which I ultimately regret as ignoring the questions would be a better solution, but what has been done is done. I was frustrated and regrettably vented.

 

[FreshJR]

-- dave taht, cofounder bufferbloat project.

Thanks for all your contributions!!! Big fan.

I watched a couple YouTube videos where gave you explanations and even the one with the bucket demonstration . Very informative!

I like to think that per-host fq will reduce the need for extensive gamer shaping rules

I fully agree where you can have a single htb, to perform limiting, and have a single fq-codel disc per user. This leads to solid performance across the board, without the need for deep packet analysis.

ASUS is approached it from a different angle.

They have a HTB to limit traffic across categories and then they have another HTB within each category to separate traffic per user.

This essentially gives each user 8 queuing disciplines each instead of 1.

It works well for the most part but the cascaded HTB's have a poor configuration, in my opinion, so users within each category cannot be allotted bandwidth in different proportions between each other, effectively.

ASUS is hellbent on making NOTHING adjustable, so this script is simply a hacky workaround to give us partial flexibility in the initial HTB configuration and additional rules for the DPI categorization. Regrettably it is not a full standalone solution.

@RMerlin, lordy... this is 2.6 based still? I feel your pain. I guess my question then becomes what are the underlying qdiscs adaptive qos, traditional qos, and freshJR's stuff uses?

The underlying configuration is cemented as to how Asus has configured Adaptive QOS (this scripts is simply surface modifications on top of it). Even the underlying schedulers are restricted from configuration when using AdaptiveQOS.

To actually implement CoDel, RMerlin had to rename the TC executable into RealTC, have Asus send their configuration what they thought was TC, and then modify those commands on the fly by forwarding them to the actual executable named RealTC.

We do have full flexibility for Traditional QOS but we then would have to forego the use of the DPI engine.

So in the end we do not have much freedom on this side of the pond.

For your question

Out of the box, Asus only offers the outdated SFQ
With RMerlin's mods, we have the choice between SFQ, Codel, fq-Codel

You already heard about our kernel limitations.

--

Once again, Thank You for your contributions!

 

[johnathonm]

Break down each step and develop an understanding. Trying to understand it as a whole isn't going to help. Take one step at a time.

Some reference for most of it is available elsewhere (try Google/duckduckgo.com/etc).

Don't worry about thinking other people understand it better or faster. Take your time and move at your own pace.

After reading both your and FreshJR's words I am going to go at this problem tomorrow with a clear head and the power of the search engine. I apologize if I cam off as pissy. I feel very frustrated with myself and it came across in my post unintentionally. I am struggling with this and you are both absolutely right about being more patient with myself as well as taking my time. I have always had a way of things just coming to me very naturally and for whatever reason as I move into computer science, this field is really pushing me in ways I hadn't encountered. I won't give up on this and I will keep pushing onward.

Thank you.

 

[johnathonm]

Hello FreshJr,

I want to first take ownership of my posting on the forum earlier. I recognize now, as I have stepped away for awhile, that I was taking my own frustration with myself, due to my own lack of understanding inadvertently at you. It was not proper, nor correct, nor even my intention. I was hoping to actually write something that would be helpful but instead my work became destructive.

I want to formally apologize to you for that as it was uncalled for, plus you deserve better from people like me who will or do derive benefits from your free hard work. I owe you personally an apology as well as I disrespected your work, labor, time and more through my post.

This was never my intention but was the result of my own frustration with my own inability to understand clearly what I was working on. This is uncharted territory for me in many ways and I am not used to hitting the limits of my own knowledge this quickly. It is no excuse though for disrespect and that is unacceptable at any time from me. I hope you will accept my sincerest apology and know that I respect you, as well as your work.

I try to hold myself to a very high standard but, being human, I often times fall short. Either way, I believe you are owed an apology. I will be posting this on the forum as I need to also take ownership of my behavior publicly as I may have inadvertently insulted you publicly.

Respectfully,

J

 

[RMerlin]

@RMerlin, lordy... this is 2.6 based still?

Broadcom SDK6.x and 7.x are based on 2.6.36. The more modern HND platform from Broadcom uses 4.1.21.

 

[Sinner]

Hello FreshJr,

I want to first take ownership of my posting on the forum earlier. I recognize now, as I have stepped away for awhile, that I was taking my own frustration with myself, due to my own lack of understanding inadvertently at you. It was not proper, nor correct, nor even my intention. I was hoping to actually write something that would be helpful but instead my work became destructive.

I want to formally apologize to you for that as it was uncalled for, plus you deserve better from people like me who will or do derive benefits from your free hard work. I owe you personally an apology as well as I disrespected your work, labor, time and more through my post.

This was never my intention but was the result of my own frustration with my own inability to understand clearly what I was working on. This is uncharted territory for me in many ways and I am not used to hitting the limits of my own knowledge this quickly. It is no excuse though for disrespect and that is unacceptable at any time from me. I hope you will accept my sincerest apology and know that I respect you, as well as your work.

I try to hold myself to a very high standard but, being human, I often times fall short. Either way, I believe you are owed an apology. I will be posting this on the forum as I need to also take ownership of my behavior publicly as I may have inadvertently insulted you publicly.

Respectfully,

J

Ill help you a bit with the install procedure.. Done this way you can edit the script if needed and get it uploaded in 12 seconds lol.

put pscp, freshs scripts everything into one folder on your hard drive. Maybe make a new folder called "qos'. so this new folder will be at "c:\qos"
now open notepad and paste the following into it and save it as install.txt in the new qos folder as well.

sh /jffs/scripts/FreshJR_QOS -uninstall
pscp -scp C:\qos\xxxxx.txt xxxxxx@xxx.xxx.xxx.xxx:/jffs/scripts/FreshJR_QOS
dos2unix /jffs/scripts/FreshJR_QOS
chmod 0755 /jffs/scripts/FreshJR_QOS
sh /jffs/scripts/FreshJR_QOS -install

those are all the install commands in order. all done in putty except the pscp line is in a dos window. Now on that pscp line you need to change the filename to the latest script filename you have and the login@ipaddress to your router login and IP.. then save this file. once that's done you can just open this and copy/paste the commands for future use.

Edit

Forgot to mention the most important part. Turn qos off before doing any of this and then back on after.

 

[INeedYou]

@FreshJR Thank you for help me by private to fix my VPN Connection, Traffic Download/Upload reverse fixed!

I have installed Asuswrt-Merlin v384.5, with the script FreshJR Adaptive QOS compatible v3 (I use the gaming rules and add the 4 VPN rules to the script: QOS + VPN Client fix via custom rules)

Edit: Deleted the unnecessary information

 

[FreshJR]

Fun little update:

Netflix's speedtest (fast.com) now measures bufferbloat && upload speed !!

This is great since some ISP's throttle netflix servers (unethically) to make Netflix service less appealing / buggy / slow in attempt to have less competition with their own tv service. Now you can see if your ISP is involved in BS shenanigans && as a bonus measure bufferbloat!

(Still doesn't measure quality like dslreports, but it's still a fun little addition)

--

As a fun fact you can run Dslreports + Fast.com at the same time and see if guaranteed bandwidth is working as expected.

I found shaping accuracy to be rock solid with little bufferbloat on both connections.

 

[skeal]

If you test and retest over and over (over days or weeks) at dslreports and average your results then take 95% of that figure and round up to whole numbers...enter those final figures into QOS up down limits. There, you have the mystery solved about those user entered settings. I'm using the alternative script...why?....I use my vpn for video. I do alot of streaming. This test result from dslreports was done this morning. I have no special rules in my setup. I use the examples that FreshJR lists at the top of the script. I also use a "ip to cidr" converter found by googling that term. Let me stress further that all I did was change port numbers or cidr ranges and remove the comments from the examples in the script. The gaming rule that you can enable caused me some problems that we are working through. Here is my results.

Edit: This test was ran while a 5Mb/s stream running through my vpn. The results are not from my vpn link they are preformed outside of the vpn. Even my tests through my vpn to dslreports are pretty darn good as well. (Again with stream running through it at the time of the test)
Edit 2: What are my ISP assigned speeds? 80/15 baby.

 

[jpclarke]

I updated the script but some advice greatly appreciated.

I regularly update the bandwidth figures via an external pi that runs a scripted speedtest, I have set the appropriate marks and assigned the up and down from the device flowid 1:2 in the script.

To get the new figures to then be applied I restart the qos service, this however is causing the upload figures to slowly decrease on every run of the script.

Can anyone confirm if the decrease in upload is caused by the iptables rules being cleared out on the qos service restart but the script does not reapply the iptables rules under this condition?

Also, what is the recommended way to apply the new up and down values without restarting qos service?

Thanks in advance!

 

[FreshJR]

To get the new figures to then be applied I restart the qos service, this however is causing the upload figures to slowly decrease on every run of the script.

You have to confirm the rPI is unthrottled when it is testing its speeds or the results will forever cascade down.

So stick the rPI into 1:2 **unthrottled**, set upload to something very low like 1mbps, and then make sure the rPI is successfully exceeding those limits while testing.

Also, what is the recommended way to apply the new up and down values without restarting qos service?

Spoiler: probably wont work now that I take a closer look

I thought we covered this a few months back, but instead of searching for it, let me cook up another solution.

1) towards the bottom of a script, you can make it accept an additional argument. Let's call it change rates.

'change_rates')
    if [ "$2" -gt "500" ] && [ "$3" -gt "500" ] ; then
        logger -t "adaptive QOS" "Changing download rate: $2"
        logger -t "adaptive QOS" "Changing upload   rate: $3"
        set_all_varabiles $2 $3
        custom_rates 2>&1 | logger -t "adaptive QOS"        #forwards terminal output & errors to logger
    else
        logger -t "adaptive QOS" "Inputs of ($2), ($3) not accepted into -change_rates"
    fi
    ;;

**just noticed the spelling mistake in set_all_varabiles, include this spelling mistake as is or correct all instances of the spelling mistake in the script**

2) You can invoke your newly created argument with two additional parameters.

eg.

FreshJR_QOS -change_rates 50000 10000

where $2 and $3 in[step 1] will take on 50,000 and 10,0000 when evaluated

3) inside the function "set_all_varabiles"

add some additional logic under this area

    DownCeil="$(printf "%.0f" $(nvram get qos_ibw))"         
    UpCeil="$(printf "%.0f" $(nvram get qos_obw))"

add the following

    if [ "$1" -gt "500" ] && [ "$2" -gt "500" ] ; then
       DownCeil="$1"         
       UpCeil="$2"
    fi

**note:**
$2 turned into $1
$3 turned into $2

when passed into this function. This is not an error in the example

4) test

--

There are are multitude of other solutions as well.

I just thought about it some more and I forgot that you also have to modify the ceilings within the 2nd set of limiters cascaded within each class.

Too much work, just keep on restarting QOS as you currently are, but do your speed really vary that much?

If they don't, then just leave it less than your attainable average and call it a day.

 

[Sinner]

Fun little update:

Netflix's speedtest (fast.com) now measures bufferbloat && upload speed !!

This is great since some ISP's throttle netflix servers (unethically) to make Netflix service less appealing / buggy / slow in attempt to have less competition with their own tv service. Now you can see if your ISP is involved in BS shenanigans && as a bonus measure bufferbloat!

(Still doesn't measure quality like dslreports, but it's still a fun little addition)

--

As a fun fact you can run Dslreports + Fast.com at the same time and see if guaranteed bandwidth is working as expected.

I found shaping accuracy to be rock solid with little bufferbloat on both connections.

Ive heard that before but ive noticed netflix doesnt use that much overall. It may buffer up hard when u load a show or movie but after uses mayby 0.5mb to maybe 2-3 on the highest settings. Jus thought id mention that

 

[FreshJR]

Ive heard that before but ive noticed netflix doesnt use that much overall. It may buffer up hard when u load a show or movie but after uses mayby 0.5mb to maybe 2-3 on the highest settings. Jus thought id mention that

I have personally been throttled down to 480p on netflix by comcast in the past.

1080p was impossible
4k was a pipe dream (~5-15mbps)

The speedtest they offer was created for that sole reason!

The move by ISP's worked since now Netflix pays Comcast to deliver data to Comcast customers.

So the ISP's

1) charge user to receive data
2) charge sender to send data
and now
3) charge either sender/user to receive non-throttled data

And somehow #3 did not break the laws while net neutrality was in effect.

The ISP's also received 400 billion of tax payer money to build on a nationwide fiber network....

I truly am speechless. Internet should be like $10bux with all this cash they commanded. I would have no qualms to move it under natural monopoly regulations.

(But this conversation is off topic for the thread at hand)

Google even had a better tool to visualize ISP throttling on YouTube videos.
https://www.google.com/get/videoqualityreport/

 

[Sinner]

I have personally been throttled down to 480p on netflix by comcast in the past.

1080p was impossible
4k was a pipe dream (~5-15mbps)

The speedtest they offer was created for that sole reason!

The move by ISP's worked since now Netflix pays Comcast to deliver data to Comcast customers.

So the ISP's

1) charge user to receive data
2) charge sender to send data
and now
3) charge either sender/user to receive non-throttled data

And somehow #3 did not break the laws while net neutrality was in effect.

The ISP's also received 400 billion of tax payer money to build on a nationwide fiber network....

I truly am speechless. Internet should be like $10bux with all this cash they commanded. I would have no qualms to move it under natural monopoly regulations.

(But this conversation is off topic for the thread at hand)

Google even had a better tool to visualize ISP throttling on YouTube videos.
https://www.google.com/get/videoqualityreport/

Ya its crazy. Ive been having a major headache with mine this summer. Im averaging maybe 50% of what it shud be.. peaking at maybe 70-80 and dropping down to 15% often. Makin it difficult for me to set qos at an even somewhat reliable setting. The often 15% is the killer and no way i can set it at that lvl so i leave it at about 40% now and seems to work ok. I live in an area where only wireless and satelite internet is available so not much i can do. I am considering trying the auto adjustment methods mentioned by a few others but havent decided yet. Im not into the idea of speed tests running in the background all the time just for that. I may come up with something that uses the ocasional ping and if higher than lets say 100ms would reduce qos levels by 10% and check again. Infact something like that in your script might be great for the less techy users. They could just load it and go and it would auto reduce based on pings. For most it wouldnt even have to go back up just stay at a good lvl for theyre conn. The process could maybe start over when qos is restarted tho. Not sure how complex that might be but if yur bored and itchin to advance the script further theres and idea =)

 

[FreshJR]

Im not into the idea of speed tests running in the background all the time just for that.

Me neither.

I may come up with something that uses the ocasional ping and if higher than lets say 100ms would reduce qos levels by 10% and check again.

The issue is that the pings have to be performed under load. So rates would have to be continually monitored and the system would have to be setup to only kick in during full network usuage. (unless you are saying you want to monitor both loaded and unloaded pings for any values higher than 100ms, 24/7 !?!?)

Both scenarios would be possible, but I do not want to start pinging a server without their permission even in the less straining intermittent ping scenario.

This feature would essentially turn into a DDOS attack for the poor victim.

Anyway, without permission I am not going to implement such a feature. I doubt any company will willingly be subject to a ddos attack, and since I do not have my own hosted server this makes it impossible.

 

[Sinner]

Me neither.



The issue is that the pings have to be performed under load. So rates would have to be continually monitored and the system would have to be setup to only kick in during full network usuage. (unless you are saying you want to monitor both loaded and unloaded pings for any values higher than 100ms, 24/7 !?!?)

Both scenarios would be possible, but I do not want to start pinging a server without their permission even in the less straining intermittent ping scenario.

This feature would essentially turn into a DDOS attack for the poor victim.

Anyway, without permission I am not going to implement such a feature. I doubt any company will willingly be subject to a ddos attack, and since I do not have my own hosted server this makes it impossible.

Well i was thinking something less agressive like ping check every minute or two or 5. During long windows updates or netflix or torrent download it would eventually catch the higher ping and adjust. Some along those lines might work well for majority of users that dont want to constantly tweak the setup all the time.id be happy with something like that even cuz id prefer not to mess with it all the time.

You may be right on the ddos thing. I never considered the possibility of millions or users using it. You could however leave the the server that is being pinged blank and configurable. Like the feature disabled unless someone inputs something in script to ping. That would spread it out almost for sure except high majority may use google or something. Personally id recomend isp or something.