[Release] FreshJR Adaptive QOS (Improvements / Custom Rules / and Inner workings)

[RMerlin]

I'm wondering if my trend micro engine has actuall updated... I see in the logs. Trend micro forward module 1.0.34

However, when I go to the firmware update page I see 1.176?update 8/5/17

One is the engine (the code part of the firmware), the other are the signature definition files, which get updated over the Internet.

 

[beboptrumpet]

Thanks for getting back. I did a factory reset and saw the same result.

Adaptive qos is giving me a headache. The trend micro engine used to recognize & give priority to my favorite game, unfortionately that has all changed. When I first saw it i thought it might have been the culprit.

Guess I'll get to work adding jrfresh's scrips.

 

[FreshJR]

Thanks for getting back. I did a factory reset and saw the same result.

Adaptive qos is giving me a headache. The trend micro engine used to recognize & give priority to my favorite game, unfortionately that has all changed. When I first saw it i thought it might have been the culprit.

Guess I'll get to work adding jrfresh's scrips.

It pretty easy to install. Takes like a minute.

Ideally if you find that game downloads have their own port that doesn't overlap with game traffic, I would add a custom rule to send all that game download traffic To the file transfers category by port.

 

[beboptrumpet]

are you sure black ops 3 uses port 3074, and not port 27017 unless that was a change to only the pc version, cause ifine warfare uses annother ports in the 27000 range but that might just be pc only. I can pull the other port number if need be for game rules.

Xbox One System NAT port 3074 tcp&udp
Black Ops3 was port 3074 for open NAT
Infinite Warfare Xbox port to get open NAT 3076

Open NAT can connect to all players, moderate can connect to open & moderate, & strict can connect only to open NAT players. Knowing that, I would imagine the game traffic is going to those ports.

As you can see, in the pic, UpNp also forwards a few external ports in the 3000's to 3076 internal. These upnp port maps show up in system logs when I end up being host for a pub.


I can think of a couple ways that might work to test & make sure.
- A switch that allows for more detailed traffic monitoring. I think my cheap Netgear might work ....or set up a pic as a network bridge

Id like to try it adding rules for those as a start. I should be able to tell if it worked by looking at the qos stats page. What would the mask be for ports 3074-3076?

 

[FreshJR]

Xbox One System NAT port 3074 tcp&udp
Black Ops3 was port 3074 for open NAT
Infinite Warfare Xbox port to get open NAT 3076

Open NAT can connect to all players, moderate can connect to open & moderate, & strict can connect only to open NAT players. Knowing that, I would imagine the game traffic is going to those ports.

As you can see, in the pic, UpNp also forwards a few external ports in the 3000's to 3076 internal. These upnp port maps show up in system logs when I end up being host for a pub.

View attachment 10082
I can think of a couple ways that might work to test & make sure.
- A switch that allows for more detailed traffic monitoring. I think my cheap Netgear might work ....or set up a pic as a network bridge

Id like to try it adding rules for those as a start. I should be able to tell if it worked by looking at the qos stats page. What would the mask be for ports 3074-3076?

Since 3074 is not divisible by 4 to get that exact range opened would take two rules.

3074/fffe opens 3074 - 3075
3076/ffff opens 3076

Instead I would just recommend a slightly larger range to have the router perform less work.

3072/fff8 would open 3072 - 3079

The chart explaining this on the first page really shouldn't be that terrible to use.

Here's a non valid example:
**NON VALID MASK COMBINATION ** 3074/fffc opening 3074-3077 **NON VALID MASK COMBINATION**
I included it to show explain why it would have no effect. It's simple, 3074 is not divisible by 4 so it's not a valid mask/mark combination.

Also you do not have to buy a dedicated switch to view your requested information. You can simply install more advanced net monitoring apps into the router. The router in essence is like a full computer.

Port forwards complicate things slightly since the router filters egress traffic. I explained these implications. Either way, if a rule is not possible, with the allotted bandwidth changes, the games shouldn't hiccup even if they aren't identified.

 

[Vexira]

Xbox One System NAT port 3074 tcp&udp
Black Ops3 was port 3074 for open NAT
Infinite Warfare Xbox port to get open NAT 3076

Open NAT can connect to all players, moderate can connect to open & moderate, & strict can connect only to open NAT players. Knowing that, I would imagine the game traffic is going to those ports.

As you can see, in the pic, UpNp also forwards a few external ports in the 3000's to 3076 internal. These upnp port maps show up in system logs when I end up being host for a pub.

View attachment 10082
I can think of a couple ways that might work to test & make sure.
- A switch that allows for more detailed traffic monitoring. I think my cheap Netgear might work ....or set up a pic as a network bridge

Id like to try it adding rules for those as a start. I should be able to tell if it worked by looking at the qos stats page. What would the mask be for ports 3074-3076?

i checked again you right ever cod except black ops 3 uses 3074, they changed it in a patch but it might be pc only.

 

[beboptrumpet]

Installed the scipt on 3.68-alpha2 on rt-ac87 Using fq_codel.

My connection normally tests at 150/20 or better After a factory reset, freshJr script, & codel_fq, setting manual bandwidth limits (150/20), and a reboot. I'm getting somewhere in the neighborhood of 20/15. I tried both Speedtest & Dsl reports.
I noticed on the qos statistics page that the download portion of the test goes to file transfer category, and the upload to web surfing. I have file transfer as my bottom category...I'm guessing my download results are related to that. Im thrilled with the consistency, and the blufferboat. I was seeing blufferboats in the 200's...now they stay below 10ms. Thanks JR. Looking forward to 2.0.

Are the last 6 lines in the log a concern?

What does checking the ATM checkbox next to WAN Packet overhead do?

can I ssh and use nano to edit the values. If so, what would I need to type after I get into nano to bring it up?

Also, I saw these entries in the log. Is this a concern?

 

[FreshJR]

I stand firm by the notion that Comcast customers do not have to account for WAN packet overhead. This is pre-modem overhead.
I am pretty sure that Comcast customers are bandwidth limited on modem egress traffic not modem ingress traffic, which is where the overhead is present.

Since Comcast limits at the modem, not before the modem, I do not think it is necessary to account for WAN overhead. Your ISP may vary. I just set rates under my advertised speeds until bufferbloat dropped.

The non-work-converving is a normal message.
I also have seen quantum message as well, but not recently.

Either way, I did not change the quantum value from the one that ASUS supplied, so I guess its normal aswell.

What do you mean you are only getting 20mbps/150mbps? Also you should set your QOS limits to like 80-95% of your max, not your max

 

[Vexira]

I stand firm by the notion that Comcast customers do not have to account for WAN packet overhead. This is pre-modem overhead.
I am pretty sure that Comcast customers are bandwidth limited on modem egress traffic not modem ingress traffic, which is where the overhead is present.

Since Comcast limits at the modem, not before the modem, I do not think it is necessary to account for WAN overhead. Your ISP may vary. I just set rates under my advertised speeds until bufferbloat dropped.

The non-work-converving is a normal message.
I also have seen quantum message as well, but not recently.

Either way, I did not change the quantum value from the one that ASUS supplied, so I guess its normal aswell.

What do you mean you are only getting 20mbps/150mbps? Also you should set your QOS limits to like 80-95% of your max, not your max

How would one go about changing the quantum, every time I see the system log I get a headache I understand if it's not an issue, I do vaguely remember reading something about it can be but I can't remember, also what is quantum for, other than a kernel error.

 

[wickedshrapnel]

How would one go about changing the quantum, every time I see the system log I get a headache I understand if it's not an issue, I do vaguely remember reading something about it can be but I can't remember, also what is quantum for, other than a kernel error.

Short answer, here is how you fix it. They fixed it in Redhat. https://www.redhat.com/archives/libvir-list/2016-February/msg00125.html <- way above my pay grade.

Long answer below

What quantum it is.

"It might be good time to touch concept of quantums now. In fact when more classes want to borrow bandwidth they are each given some number of bytes before serving other competing class. This number is called quantum. You should see that if several classes are competing for parent's bandwidth then they get it in proportion of their quantums. It is important to know that for precise operation quantums need to be as small as possible and larger than MTU.
Normaly you don't need to specify quantums manualy as HTB chooses precomputed values. It computes classe's quantum (when you add or change it) as its rate divided by r2q global parameter. Its default value is 10 and because typical MTU is 1500 the default is good for rates from 15 kBps (120 kbit). For smaller minimal rates specify r2q 1 when creating qdisc - it is good from 12 kbit which should be enough. If you will need you can specify quantum manualy when adding or changing the class. You can avoid warnings in log if precomputed value would be bad. When you specify quantum on command line the r2q is ignored for that class."

Source: http://luxik.cdi.cz/~devik/qos/htb/manual/userg.htm

Basically I think the issue is this algorithm was written at a time when bandwidth speeds in the hundreds of megabytes or gigabit wasn't feasible and letting r2q calculate the quantum value freaks it out when that number is so high.

If rate is less than 80 the quantum is small warning is provided.
If rate is greater than 16000 quantum is big warning is provided

Quantum is calculated as rate (in bytes) / r2q.
r2q is by default 10

For example, my download rate is set at 300Mb you take that (in bytes) divided by the default r2q of 10 and that equals 3,750,000 bytes for the quantum. The hard coded limit of quantum is 60,000 so it throws a warning.

So in order to prevent these warnings for my 300Mb connection, my r2q needs to be 20,000. That will keep the quantum under 16,000 where it cries but I think it would need to be even a bigger number to get my quantum closer to my MTU of 1500. Changing the r2q value would be the easiest option I think since there are a lot of classes and that would suck to change all of them manually.

There are a lot of classes, you can see them by running the following command

tc class show dev eth0

The class in the log " HTB: quantum of class 10011 is big. Consider r2q change." will be shown in the command output truncated like "class htb 1:11" they substitute a : in place of the zeros.

 

[Vexira]

Thanks for that explanation it makes sense now, I wonder if there's a way to get the script to automatically calculate r2q based on input of bandwidth values, I wonder if this is still an issue on the HSFC algorithm, shame we don't have it available as an option to see if it is better than HTB or doesn't have the same issues.

 

[wickedshrapnel]

Thanks for that explanation it makes sense now, I wonder if there's a way to get the script to automatically calculate r2q based on input of bandwidth values, I wonder if this is still an issue on the HFSC algorithm, shame we don't have it available as an option to see if it is better than HTB or doesn't have the same issues.

I was messing with pfSense for a while and HFSC is the best QoS method they have over there and everyone is begging for fq_codel there. https://forum.pfsense.org/index.php?topic=87931.0

I guess fq_codel is the bee's knees as far as performance goes in QoS. I think this is something that would have to be fixed by ASUS rather than patched in a script. It is not breaking anything but I think it could be causing slight delays probably in milliseconds since it is not optimized. it's just more of an annoyance than anything. I like clean logs.

 

[Vexira]

I was messing with pfSense for a while and HFSC is the best QoS method they have over there and everyone is begging for fq_codel there. https://forum.pfsense.org/index.php?topic=87931.0

I guess fq_codel is the bee's knees as far as performance goes in QoS. I think this is something that would have to be fixed by ASUS rather than patched in a script. It is not breaking anything but I think it could be causing slight delays probably in milliseconds since it is not optimized. it's just more of an annoyance than anything. I like clean logs.

I've been reading the beast combination is FQ-Codel +HSFC, as in HSFC as packet scheduler and FQ-Codel as QOS discipline, read some ddwrt people saying that combo is amazing.
Though I have to agree with you it should be up to asus to implement it, but they will only do it if they get a lot of requests, to do so.
If the r2q was auto calculated we wouldn't see any of the r2q errors in the logs and it hope fully would become more accurate.

 

[wickedshrapnel]

I've been reading the beast combination is FQ-Codel +HSFC, as in HSFC as packet scheduler and FQ-Codel as QOS discipline, read some ddwrt people saying that combo is amazing.
Though I have to agree with you it should be up to asus to implement it, but they will only do it if they get a lot of requests, to do so.
If the r2q was auto calculated we wouldn't see any of the r2q errors in the logs and it hope fully would become more accurate.

Cake is another one. It is built off fq_codel supposed to be an improvement of it. https://www.bufferbloat.net/projects/attachments/150817135028_cake-
battlemesh-v8.pdf

 

[Vexira]

Cake is another one. It is built off fq_codel supposed to be an improvement of it. https://www.bufferbloat.net/projects/attachments/150817135028_cake-
battlemesh-v8.pdf

I'm aware of cake, but apparently the current kernels are too old to support it RMerlin told me, or the full feature set of FQ-Codel sadly I wish asus would just port it since qos is closed source. Id love to have it if its possible.
https://www.bufferbloat.net/projects/codel/wiki/Cake/

 

[Vexira]

only those 2 show up in the log with the error I'm looking at the script o see where I can change the r 2q
kernel: HTB: quantum of class 10012 is big. Consider r2q change.
kernel: HTB: quantum of class 10013 is big. Consider r2q change

 

[Vexira]

@FreshJR is there any way to implent what what @wickedshrapnel posted into the script in regards to modding the r2q values from what I read it affects accuracy of the shaper, you would know more about it than me to be honest, but I'm also curious as to what benefits could appear of it is fixed other than the annoying log messages. Is there any way to implement the commands from the links to the script.

Fix and explanations:
https://www.redhat.com/archives/libvir-list/2016-February/msg00125.html
http://luxik.cdi.cz/~devik/qos/htb/manual/userg.htm

 

[Arsenal]

Dumb question, but will this help with Discord traffic being flagged as General rather than VOIP? It doesn’t sound as though there’s an easy way to manually flag Discord traffic based on ports: https://www.reddit.com/r/discordapp/comments/4g3179/is_there_a_way_to_make_discord_not_use_random/

 

[ThiagoCururu]

Hello.

I'm almost losing my sanity over this QoS.

When I'm gaming and there's Netflix or YouTube traffic on my network, I get so many latency spikes that it makes impossible to play games that aren't flagged as Gaming on QoS. To be honest I don't even know if I have any game that is properly identified.
To make things worse, all Steam traffic is considered Gaming, including downloads and updates.

When I first saw this script I thought I would be saved, but unfortunately no.

I followed that tutorial and got it running on my AC68U with firmware 380.67. Checked on System Log and it is working, but as you can see on screenshots below, it's all going downhill.

I'm using a fiber connection with 50/25 Mb speeds (51.23 and 26.1 speeds at dslreports.com before that script. Bufferbloat with A and overall quality on D). Tested on automatic and manual bandwidth of 43/20 speeds. Tried sfq and fq_codel with no results. Overhead of 4, 16, 27 and 28 made no difference.

I really don't know what else I can try. Any help?


Game example: Path of Exile x64 without Steam. Lockstep and multithread DX11.

 

[Vexira]

what is the signature version your running