[Release] FreshJR Adaptive QOS (Improvements / Custom Rules / and Inner workings)

[medwatt]

Having download traffic within Web Surfing (or anything mis-categorized) shouldn't the end of the world. The guaranteed rates per category should keep everything functioning.

This means all categories UNDERNEATH web surfing will always get their guaranteed rate.
(The only issue with HTTPS downloads in "WebSurfing" would be that "WebSurfing" would use all the excess bandwidth available, and that excess would not be offered to the categories underneath it).

In your position, I would simply tweak the guaranteed bandwidths or lower the priority of websurfing if something not working to your liking and call it a day.

--

Iptables is able to detect download traffic & override the traffic mark into what you desire.

The iptable rule would be

-m mark --mark 0x80120000/0x80380000 -m connbytes --connbytes 1024000 --connbytes-dir both --connbytes-mode bytes -j MARK --set-mark ${Downloads_mark_down}

Currently, I see that connections are ONLY evaulated on initial opening. This was probably done for performance reasons.
If using the above rule, since the download would not initially be a considered a download and iptables would not be re-evaluating the connection at a later point in time, I don't think thenrule would take effect.

In past firmwares, I remember connections would be CONTIUALLY evaluated (maybe I had HW acceleartion disabled). In those firmwares, that rule should take effect and achieve what you wish.

You would have to research as to what firmwares that rule would still work on. Or potentially what parameter in the WebUI is preventing connections from being continually evaluated.

It really may be that HW acceleration only needs to be disabled for that rule to work. I haven’t messed around with that as I don’t want HW accel to be disabled. (It’s okay to do so on slow connections)

The problem is this also slows down normal web browsing tremendously as both downloads and regular web browsing get identified as "HTTPS over TLS SSL". I'm on a 5MBits connection. Probably, I don't have a lot of wiggle room.

 

[L&LD]

The problem is this also slows down normal web browsing tremendously as both downloads and regular web browsing get identified as "HTTPS over TLS SSL". I'm on a 5MBits connection. Probably, I don't have a lot of wiggle room.

Did you disable HW acceleration? Is the issue still there after doing so and rebooting the router?

 

[FreshJR]

Looked into iptables initially vs continually parsing connections.
My hunch was correct.

1) With HW accel on, only the initial packet of every individual connection is evaluated.
2) With HW accel off, every packet passing through he router is evaluated

I updated the wording on previous post to remove speculation.

The rules from the previous post moving bulky WebSurfing into Downloads work. (I tested them)

I do not recommend disabling HW acceleration on most connections since the CPU will not be able to process data that fast. If you want to use those rules, I suggest your connection be less than 25-50mbps.

 

[skeal]

Looked into iptables initially vs continually parsing connections.
My hunch was correct.

1) With HW accel on, only the initial packet of every individual connection is evaluated.
2) With HW accel off, every packet passing through he router is evaluated

I updated the wording on previous post to remove speculation.

The rules from the previous post moving bulky WebSurfing into Downloads work. (I tested them)

I do not recommend disabling HW acceleration on most connections since the CPU will not be able to process data that fast. If you want to use those rules, I suggest your connection be less than 25-50mbps.

How does flow cache fit in to this? On a AX88U you cannot alter any of the acceleration options.
Update: Runner is disabled by QOS.

 

[FreshJR]

How does flow cache fit in to this? On a AX88U you cannot alter any of the acceleration options.
Update: Runner is disabled by QOS.

Sure you can, but maybe not directly from the WebUI.

Most likely setting an nvram variable and rebooting the router will be enough to disable HW acceleration.

Alternatively, enable traditional QOS.
Reboot (Router will boot into non HW accelerated mode).
Switch to adaptive QOS.
(You now have A.QoS without HW acceleration)

 

[PeterV]

Hi!
@FreshJR
Tanhks for your product! (i.e script & environment).

But I have a small question about DNS over TLS (in my case - I'm using stubby on the router, with several DNS providers on other side on port 853) .
You already had some discussion with strangeluck, but I don't have a full undestanding.
So, do I need to do something to mark this traffic as Net Control, or now, and where I can read more about ?

P.S. Excuse me, as usual user, I'm not enough thoughtful ...

 

[Vexira]

So hardware acceleration should be on if your connection is 100/40 or 95/37 ?

 

[FreshJR]

'm using stubby on the router, with several DNS providers on other side on port 853) .

Without FreshJR_QOS ALL traffic generated by the router skips/bypasses QOS.

With FreshJR_QOS most traffic generated by the router is all placed into ${Downloads}.

I made exceptions for router generated traffic on ports 53,123 (DNS / NTP) and choose have those requests skip QOS. Looks like I should add port 853 into the mix.

iptables -A OUTPUT -t mangle -o $wan -p tcp -m multiport ! --dports 53,123 -j MARK --set-mark ${Downloads_mark_up} &> /dev/null #VPN Fix - (Fixes upload traffic not detected when the router is acting as a VPN Client)


iptables -A OUTPUT -t mangle -o $wan -p udp -m multiport ! --dports 53,123 -j MARK --set-mark ${Downloads_mark_up} &> /dev/null #VPN Fix - (Fixes upload traffic not detected when the router is acting as a VPN Client)

I can make modifications so that (53,123 and now 853) goes into NetControl instead of it being 0-rated but I didn't perform A/B testing to see which scenario would be more responsive. The current behavior gets me a 30-33ms response time per query. (every ms does counts for DNS but I didn't test the other configuration).

It also could be worth looking into not only what is happening with DNS requests, but also what is happening to DNS responses directed to the router and their interaction with QOS. (Not quite sure where those end up currently)

--

The conversation you linked had rules in POSTROUTING. Keep in mind:

OUPUT = traffic generated by the router
POSTROUTING = traffic generated by the clients on network

--

So hardware acceleration should be on if your connection is 100/40 or 95/37 ?

HW accleration should ONLY be disabled if you want to use the rules that track cumulative bandwidth per connection and and move traffic from Web -> Downloads once it exceeds 1MB transferred.

You might be able to use it at 100mbps on your router. I don't know. All I know is that mine cant handle more than 25-50mbps without pegging the CPU and creating lag.

If not wanting to use those rules, definitely keep HW acceleration ON unless its causing other problems.

 

[L&LD]

Hi!
@FreshJR
Tanhks for your product! (i.e script & environment).

But I have a small question about DNS over TLS (in my case - I'm using stubby on the router, with several DNS providers on other side on port 853) .
You already had some discussion with strangeluck, but I don't have a full undestanding.
So, do I need to do something to mark this traffic as Net Control, or now, and where I can read more about ?

P.S. Excuse me, as usual user, I'm not enough thoughtful ...

Without FreshJR_QOS ALL traffic generated by the router skips/bypasses QOS.

With FreshJR_QOS most traffic generated by the router is all placed into ${Downloads}.

I made exceptions for router generated traffic on ports 53,123 (DNS / NTP) and choose have those requests skip QOS. Looks like I should add port 863 into the mix.

iptables -A OUTPUT -t mangle -o $wan -p tcp -m multiport ! --dports 53,123 -j MARK --set-mark ${Downloads_mark_up} &> /dev/null #VPN Fix - (Fixes upload traffic not detected when the router is acting as a VPN Client)


iptables -A OUTPUT -t mangle -o $wan -p udp -m multiport ! --dports 53,123 -j MARK --set-mark ${Downloads_mark_up} &> /dev/null #VPN Fix - (Fixes upload traffic not detected when the router is acting as a VPN Client)

I can make modifications so that (53,123 and now 863) goes into NetControl instead of it being 0-rated but I didn't perform A/B testing to see which scenario would be more responsive. The current behavior gets me a 30-33ms response time per query. (every ms does counts for DNS but I didn't test the other configuration).

It also could be worth looking into what is happening not only with DNS requests, but also the DNS responses directed to the router and QOS. (Not quite sure where those end up currently)

--

The conversation you linked had rules in POSTROUTING. Keep in mind:

OUPUT = traffic generated by the router
POSTROUTING = traffic generated by the clients on network

--



HW accleration should ONLY be disabled if you want to use the rules that track cumulative bandwidth per connection and and move traffic from Web -> Downloads once it exceeds 1MB transferred.

You might be able to use it at 100mbps on your router. I don't know. All I know is that mine cant handle more than 25-50mbps without pegging the CPU and creating lag.

If not wanting to use those rules, definitely keep HW acceleration ON unless its causing other problems.

FreshJr, just pointing out a small correction. I think you mean port 853, not port 863? Or am I wrong here?

 

[Sinner]

So hardware acceleration should be on if your connection is 100/40 or 95/37 ?

Is say if you require more qos acuracy turn it off. If you require more qos performance turn it on.

 

[FreshJR]

Is say if you require more qos acuracy turn it off. If you require more qos performance turn it on.

Close.

The scripts default rules and user created rules via the UI don't need an accuracy boost to function as intended. They will function the same with HW acceleration ON and OFF.

The only time you would need turn HW acceleration OFF is if you are wanting to use the rules Medwatt requested on the previous page (aka rules that track bandwidth used per opened connection and filter accordingly).

 

[Sinner]

Close.

With default rules, and any custom rules in the UI, accuracy will remain the same with both HW acceleration ON and OFF.

The only time you would need to disable it is if you want to use the rules Medwatt requested on the previous page.

I was attempting to simplify

 

[Oleg Eremenko]

Hi Fresh,

Just an update, I'm the one with the wonky router behavior from a couple of weeks ago. I kept 384.9 and had adaptive qos turned off (bandwidth limiter on) for a week: no crashes. Uninstalled and reinstalled your script and with adaptive qos now no crashes for 4 days, so back in business, lovin' it.

One question though, how can one move game transferring to the bottom? Some streaming services use port 80 and fall to file transferring, so I have to throttle Steam/Origin from within the apps.

Thanks!

 

[FreshJR]

Hi Fresh,

Just an update, I'm the one with the wonky router behavior from a couple of weeks ago. I kept 384.9 and had adaptive qos turned off (bandwidth limiter on) for a week: no crashes. Uninstalled and reinstalled your script and with adaptive qos now no crashes for 4 days, so back in business, lovin' it.

One question though, how can one move game transferring to the bottom? Some streaming services use port 80 and fall to file transferring, so I have to throttle Steam/Origin from within the apps.

Thanks!

I will make that action a controllable via check box controllable from the WebUI. (Added to the todo list).

If you are comfortable with manually editing the script with notepad++ and uploading it to the router with scp (basically following the manual install procedure from the first page), I can post the lines of code need minor changing.

Alternatively you boost the minimum guaranteed rate for your bottom category even if it means the sum of all the categories will be slightly over 100% This will cause Game Transferring to have LESS excess bandwidth available and your Downloads to have more guaranteed bandwidth.

In practice, **only** the bottom category can have have a guaranteed rate set that causes the sum of all the rates above it to exceed 100% while having ZERO ill effect under all network circumstances.I just didn't want to explicitly state this was possible to reduce confusion.

 

[Oleg Eremenko]

I will make that action a controllable via check box controllable from the WebUI. (Added to the todo list).

If you are comfortable with manually editing the script with notepad++ and uploading it to the router with scp (basically following the manual install procedure from the first page), I can post the lines of code need minor changing.

Alternatively you boost the minimum guaranteed rate for your bottom category even if it means the sum of all the categories will be slightly over 100% This will cause Game Transferring to have LESS excess bandwidth available and your Downloads to have more guaranteed bandwidth.

In practice, **only** the bottom category can have have a guaranteed rate set that causes the sum of all the rates above it to exceed 100% while having ZERO ill effect under all network circumstances.I just didn't want to explicitly state this was possible to reduce confusion.

I guess it's more of a finetune and I don't want to break anything now that I've got it working otherwise without problems, so I'll wait 'til the feature is out officially. Thanks again for your great work!

 

[FreshJR]

Just set file downloads to 30% and call it a day. Leave the others at the default rates.

 

[Agentsicker]

Welcome to FreshJR's Adaptive QOS modification script!

Gaming Rule Note

Gaming traffic originating from ports 80 & 443 is primarily downloads & patches (some lobby/login protocols mixed within)
Manually configurable rule will take untracked traffic, not originating from 80/443, for specified devices and place it into Gaming
Use of this gaming rule REQUIRES devices to have a continous static ip assignment && this range needs to be defined in the script
[/spoiler]

I am new to this script thing , i used putty to install your awesome Mod, and would like to implement the gaming rule, i have a static ip for my ps4, and i know how to figure out the CIDR range. Im just unsure how to define the range into the script? I just installed Notepad ++ but dont know where to start

 

[lilstone87]

Go to the "Classification" tab under "Adaptive QoS" at the top right you will see "FreshJR Mod" click on it. It will open up the rules, and then just add the IP/CIDR under the gaming rule. Which should be the top rule listed.

 

[bimdimbo]

I am new to this script thing , i used putty to install your awesome Mod, and would like to implement the gaming rule, i have a static ip for my ps4, and i know how to figure out the CIDR range. Im just unsure how to define the range into the script? I just installed Notepad ++ but dont know where to start

Hey, I was similar a couple of days ago. If you are just wanting to specify the PS4, you don't need the CIDR address. Just put the IP of your PS4 into the top left box in the 'Iptables Rules' on the Classicication page of FreshJR's mod.

 

[lilstone87]

Hey, I was similar a couple of days ago. If you are just wanting to specify the PS4, you don't need the CIDR address. Just put the IP of your PS4 into the top left box in the 'Iptables Rules' on the Classicication page of FreshJR's mod.

I suggest he put 192.168.1.xxx/32. Which will apply the rule only to that IP. If he has multiple systems, it can be adjusted. I don't recommend just putting the internal ip of the device only. Unless @FreshJR added it so you only need to input the IP alone.