[Release] FreshJR Adaptive QOS (Improvements / Custom Rules / and Inner workings)

[lilstone87]

My dslreports speedtest is identified as a file transfer. Are you on the latest definitions? I was using chrome.

I only know of one on type of traffic that has bypassed QOS. I can't look into things without more information.

Note: I consitantly get an A for bufferbloat, but a C or a B for quality as my upload speed loves to decide not to be stable.

Well I just run a test on there with Adaptive QoS enabled, using FD_Codel, with manual limits set at 280/28. Here's the test result, and as you see it totally disregarded any limit set, and actually pushed the total limit of my ISP provided speed. The result's are not bad by any means... But still don't like how it disregards the limit's I inputted. I ran a test on say speedtest.net, and it stayed within the limits I inputted.

 

[FreshJR]

Well I just run a test on there with Adaptive QoS enabled, using FD_Codel, with manual limits set at 280/28. Here's the test result, and as you see it totally disregarded any limit set, and actually pushed the total limit of my ISP provided speed. The result's are not bad by any means... But still don't like how it disregards the limit's I inputted. I ran a test on say speedtest.net, and it stayed within the limits I inputted.

Once again, ARE YOU ON THE LATEST DEFINITION!

I am on 1.172. If you are not, do not upgrade yet. I would like to see where you traffic is erroneously going under the old definitions.

 

[lilstone87]

Once again, ARE YOU ON THE LATEST DEFINITION!

Yes I am FreshJR sorry.... I forgot to reply to that part. I noticed the issue. My test on there was running threw HTTPS, as you can see in that image from the test I run. I had to go into settings on there site, and make it use HTTP. So you're 100% correct HTTPS is bypassing QoS rules.

EDIT: just ran that test again HTTP only this time, and here's the results. results were actually worst.

Second EDIT: I actually updated to Merlin's latest build earlier, so I'm actually on definition 1.174

 

[FreshJR]

Yes I am FreshJR sorry.... I forgot to reply to that part. I noticed the issue. My test on there was running threw HTTPS, as you can see in that image from the test I run. I had to go into settings on there site, and make it use HTTP. So you're 100% correct HTTPS is bypassing QoS rules.

EDIT: just ran that test again HTTP only this time, and here's the results. results were actually worst.

Second EDIT: I actually updated to Merlin's latest build earlier, so I'm actually on definition 1.174

This is a 2nd example of the same issue from before. I don't know why QOS is missing these two filter rules out of the box. (ASUS bug?)

${tc} filter add dev br0 protocol all prio 22 u32 match mark 0x80130000 0x803f0000 flowid ${Web}
${tc} filter add dev eth0 protocol all prio 22 u32 match mark 0x40130000 0x403f0000 flowid ${Web}

What happens is that the QOS engine marks some packets as 0x13 or dec 18, but ASUS doesn't sort these marks into any catagory so the packets end up bypassing QOS entirely.

For the time being, everyone can feel free to add the above rules into the script to account for this type of traffic. Put it before the last rule in each upload/downloads traffic rules section of the script.

I am not adding it it to the original script yet by default, since I am not sure what category I want this traffic to go to yet. I need to see if its all https traffic or a subset. RMerlins tables from the first pages says this packet mark is assigned to Network Protocol type traffic.

@RMerlin, where did you get this table from the begging of this thread (format slightly modified by me).

TC Prio  Filter Mark (hex)        Filter Mark (Dec) Catagory Name
1      custom                                       custom rules
2      mark 0x80000000 0x8000ffff (success 0)    -  unidentified
3      mark 0x80000000 0x803f0000 (success 0)    0  IM
4      mark 0x80010000 0x803f0000 (success 0)    1  P2P
5           0x8002000                            2  ??
6      mark 0x80030000 0x803f0000 (success 0)    3  File Sharing
7      mark 0x80040000 0x803f0000 (success 0)    4  Streaming
8      mark 0x80050000 0x803f0000 (success 0)    5  EMail
9      mark 0x80060000 0x803f0000 (success 0)    6  VOIP
10     mark 0x80070000 0x803f0000 (success 0)    7  Database tools
11     mark 0x80080000 0x803f0000 (success 0)    8  Online Games
12     mark 0x80090000 0x803f0000 (success 0)    9  Management tools / protocols
13     mark 0x800a0000 0x803f0000 (success 0)    10 Remote access
14     mark 0x800b0000 0x803f0000 (success 0)    11 Tunneling and proxy
15          0x800c                               12 ??
16     mark 0x800d0000 0x803f0000 (success 0)    13 Web Services
17     mark 0x800e0000 0x803f0000 (success 0)    14 Security update
18     mark 0x800f0000 0x803f0000 (success 0)    15 Web instnat messenger
19          0x8010                               16 ??
20     mark 0x80110000 0x803f0000 (success 0)    17 Business tools
21          0x8012                               18 Network protocol
22          0x8013                               19 Network protocol (https download & https dslreports speedtestwent here)
23     mark 0x80140000 0x803f0000 (success 0)    20 Network protocol
24     mark 0x80150000 0x803f0000 (success 0)    21 ??
25          0x8016                               22 Adult
26     mark 0x80170000 0x803f0000 (success 0)    23 ??
27     mark 0x80180000 0x803f0000 (success 0)    24 Social network
28          0x8019                               25 ??
29          0x801a                               26 Advertisement

The rules without mark preceding them are not included in default ASUS qos rules.

 

[FreshJR]

@brummygit

yup ASUS bug. See previous post for explanation and fix. Not updating original script yet since not sure what other traffic get that packet mark.

Example: https youtube videos get correctly identified as Video Streaming, so wondering if 0x13 is all unidentified https traffic

 

[brummygit]

@brummygit

yup ASUS bug. See previous post for explanation and fix. Not updating original script yet since not sure what other traffic get that packet mark.

Yes, timing issue in my posts thanks. I was trying to help diagnose. Will delete my rather long post now to avoid cluttering the thread

 

[lilstone87]

@FreshJR Thanks again, I will likely just hold off till you know more. But again I appreciate it.

 

[jpclarke]

What's the mark for the net control packets to apply an iptables command for speedtest traffic?

I plan to set the ceiling of the net control packet class to 1GB just before the test and set it back to 5% of total after the test has run.

If you can help with the mark used to allocate traffic to the net control packets class I hope this will get the full speeds reported no matter what else is flowing at the time.

Thanks

 

[FreshJR]

What's the mark for the net control packets to apply an iptables command for speedtest traffic?

I plan to set the ceiling of the net control packet class to 1GB just before the test and set it back to 5% of total after the test has run.

If you can help with the mark used to allocate traffic to the net control packets class I hope this will get the full speeds reported no matter what else is flowing at the time.

Thanks

The marks are the same as the post above, just make sure they end with 1 or they will go to unidentified traffic.

So mark 0x80140001 with mask 0x803f000 will go to net control for download traffic.
Note: Following the pattern, upload has 0x40140001 for the mark and 0x403f0000 for mask for net control traffic.

In the script I only included 0x4 marks since the iptables rule example only dealt with outbound traffic. I will expand the comment on that section to clarify.

Make sure you set the rate at 1gbs, instead of just the ciel when you are changing the class values, since ciel = max it can hit, rate = guaranteed bandwidth before traffic is offered to next QOS container.

Also, if you do not want to keep setting and unsetting your traffic rule, just permanently give it a random mark like 0x87770001. It won't be prioritized above other traffic but it wont be bandwidth limited anymore, as that mark will not have a matching rule.

 

[RMerlin]

@RMerlin, where did you get this table from the begging of this thread (format slightly modified by me).

/tmp/bwdpi/bwdpi.cat.db contains the categories, and /tmp.bwdpi/bwdpi.app.db shows what application goes into what categories.

 

[jpclarke]

The marks are the same as the post above, just make sure they end with 1 or they will go to unidentified traffic.

So mark 0x80140001 with mask 0x803f000 will go to net control for download traffic.
Note: Following the pattern, upload has 0x40140001 for the mark and 0x403f0000 for mask for net control traffic.

In the script I only included 0x4 marks since the iptables rule example only dealt with outbound traffic. I will expand the comment on that section to clarify.

Make sure you set the rate at 1gbs, instead of just the ciel when you are changing the class values, since ciel = max it can hit, rate = guaranteed bandwidth before traffic is offered to next QOS container.

Also, if you do not want to keep setting and unsetting your traffic rule, just permanently give it a random mark like 0x87770001. It won't be prioritized above other traffic but it wont be bandwidth limited anymore, as that mark will not have a matching rule.

Hi,

I have now completed what I was looking to do, essentially I do the following to achieve automatic bandwidth and qos customisation:

Firstly I have setup the marking to have traffic from the rpi going into the network control packet queue.

Just before I run the speedtest on the rpi I set the allowed bandwidth on the network control packet queue to 1GB in both directions.

I then run the speedtest on the rpi which populates 2 text files for up and down speeds, these are are stored on the router, values are stored as 97% of the max obtained.

A script then runs on the router to extract the last entry in each of the text files generated containing the bandwidth measurements and stores these in nvram.

I then reset qos before I run the FreshJR qos script, this then sets the queues to the desired limits and also ensures the network packet queue limits are set back to normal levels (5%).

Thanks for your help on this!

 

[FreshJR]

I then reset qos before I run the FreshJR qos script, this then sets the queues to the desired limits and also ensures the network packet queue limits are set back to normal levels (5%).

Thanks for your help on this!

If you are actually resetting the QOS, then you are losing your container statistics. I do not know if this is important to you but I'm sure you have already discovered that TC changes are active immediately during runtime without a QOS restart required.

The script already has a cron job scheduled to execute a daily persistence check at midnight. This checks if rules have not been overwritten with a simple conditional statement.

With this you may be interested in also making the following changes to your procedure.

-Remove my cronjob and have your inbound/outbound traffic script call my script after it is completed.
-change this conditional statement in my script:

Spoiler

CurrentDownRate1="$(${tc} class show dev br0 | grep -w "1:10" | tr ' ' '\n' | grep "rate" -A1 | tail -n 1)"
if [ "${CurrentDownRate0}" != "${DownRate0}" ] ; then

I've replaced the 1's with 0's so it checks the net control for modification before executing instead of the 1rst catagory.

This would correct your queues without restarting QOS and everything would be done at runtime.
There's no reason for two cronjobs.

At no point does the QOS system have to really be restarted.

I hope the non-constant formatting didn't mess you up in that portion of the script.
Variable "DownRateX" has kBit appended onto the end of it.
Variable "DownCeil" does not have kBit appended onto the end of it.
I'll make it consistent next version.

 

[el pescador]

I dont run your qos system fresh....bit too technical and need time to sit through and install etc.

But on the standard qos ive noticed something very odd on Battlefield 1 PS4.
I watch the qos graphs and notice initially lots of data in the game bracket on upload and download then for some reason it nearly all shifts over to default.
We are talking 200000 bits from gaming to default on both.
Leaving about 300-500 bits in gaming.

Any ideas?

Fifa is quite stable.
Varies between 10k and 20k on both .

 

[lilstone87]

@FreshJR
Well I was able to test at least one download so far on PS4. Downloading a 8GB game, was using two ports/streams to download. One was port 61456, and other was port 60797. So it might be a bit tricky hammering down a couple port's, to set at a lower priority for PS4 download's. I also played couple games of Battlefield Hardline online, and while in matches, it was using port 3659 to pass data. Normally if I'm playing something like Call of Duty online on PS4, it uses port 3074. I will test that in just a bit, and see exactly what port is passing data while in online games.

EDIT: I just tested Call of Duty Black Ops 3 on PS4, and in matches it's using port 3074, like I mentioned most call of duty games do for console, at least.

Also so far while watching my PS4 strictly, the PS4 uses a lot of different ports ranging mostly in the 50k/60k port range, but I did notice exactly one time a port in the 49k range was being used. So it appears the PS4 uses a lot of the higher port ranges to pass normal communication traffic, and for stuff like downloads. But when it comes to most gaming traffic, we're talking lower ranged port's like 3659/3074.

Anyways I don't know if any of this helps at least with the PS4, but I wanted to check it for myself, and figured I would share the info here. As we're all trying to maximize QoS for each of our need's, and mine just happens to be gaming, and wanting it to have game traffic as top priority. With what I have seen so far, it would be nice if we could just prioritize the lower data stream traffic, within what's considered game traffic by QoS itself. But I honestly don't know if it would be possible.

 

[jpclarke]

Even after tagging the rpi to go into network control packet queue and setting the rate and ceiling to 1Gb the speedtest runs are reducing over time.

Any ideas I should check to ensure the speedtest runs get full bandwidth?

Thanks

 

[FreshJR]

@jpclarke PM ur setup. I can read it real quick and check if you are setting up something wrong.

Off the bat, I can see that class 1:1 has a ceil and rate set. So in summary if class 1:1 has bandwidth limit then class 1:10(Net control) will inherit the remainder. So when class 1:1 is imited, class 1:10 will follow its precedence.

@lilstone87 all data helps for anyone making custom rules. I will be doing custom PS4 optimization rules for personal use when destiny2 comes out.

I would like downloads and updates to have lower priority compared to destiny2 game traffic. I will see what data I have to work with.

 

[jpclarke]

Hi,

I have posted the code, anything you can see to get accurate bandwidth measurement working would be appreciated!

Thanks.

 

[Vexira]

@FreshJR
Well I was able to test at least one download so far on PS4. Downloading a 8GB game, was using two ports/streams to download. One was port 61456, and other was port 60797. So it might be a bit tricky hammering down a couple port's, to set at a lower priority for PS4 download's. I also played couple games of Battlefield Hardline online, and while in matches, it was using port 3659 to pass data. Normally if I'm playing something like Call of Duty online on PS4, it uses port 3074. I will test that in just a bit, and see exactly what port is passing data while in online games.

EDIT: I just tested Call of Duty Black Ops 3 on PS4, and in matches it's using port 3074, like I mentioned most call of duty games do for console, at least.

Also so far while watching my PS4 strictly, the PS4 uses a lot of different ports ranging mostly in the 50k/60k port range, but I did notice exactly one time a port in the 49k range was being used. So it appears the PS4 uses a lot of the higher port ranges to pass normal communication traffic, and for stuff like downloads. But when it comes to most gaming traffic, we're talking lower ranged port's like 3659/3074.

Anyways I don't know if any of this helps at least with the PS4, but I wanted to check it for myself, and figured I would share the info here. As we're all trying to maximize QoS for each of our need's, and mine just happens to be gaming, and wanting it to have game traffic as top priority. With what I have seen so far, it would be nice if we could just prioritize the lower data stream traffic, within what's considered game traffic by QoS itself. But I honestly don't know if it would be possible.

are you sure black ops 3 uses port 3074, and not port 27017 unless that was a change to only the pc version, cause ifine warfare uses annother ports in the 27000 range but that might just be pc only. I can pull the other port number if need be for game rules.

 

[FreshJR]

@Vexira

Any qos information is welcome in this thread, including port specific application discussion.

It it useful information for anyone looking to fine tune their expierence.

 

[lilstone87]

are you sure black ops 3 uses port 3074, and not port 27017 unless that was a change to only the pc version, cause ifine warfare uses annother ports in the 27000 range but that might just be pc only. I can pull the other port number if need be for game rules.

@Vexira @FreshJR
I welcome you to check... As I don't PC game these days. I however have gamed a good amount on PS4, and have played all the COD games. But I will say I barely played COD IW, as I didn't like the new one. I don't even currently have it installed on my PS4. So I can't 100% confirm it uses port 3074 to pass gaming packet's once in a lobby. But I can say all previous COD games on PS4, I have personally seen them use port 3074 to pass active match data. I do however think in the past they have listed a port range of like 3074-3076, for people who have multiple PS4's online at same time playing the same game.

With that said, from what I have seen, a lot of the time. Even with two/three PS4 online under the same router playing the same COD game online, they all still try pushing data threw the same 3074 port, instead of one using 3074, other using 3075, etc.. Reason often times you will have people with these types of setup's. Posting online how one system is stating a OPEN nat type, while the second/third console saying Moderate nat type. Because the router itself is opening 3074 via UPnP, or Port Forwarding for the first console playing the game. Then the Second/Third console also request the same port to be open, and is declined, as the first system already has it open. So them system's report back a moderate nat type, instead of requesting say 3075/3076, and they still end up pushing data threw the same 3074 port.

Again with all this, This is talking strictly COD games for PS4 for me. As I stated in a before post, I also played Battlefield Hardline, on my PS4, and it was using port 3659 to pass active game data while in a match. After watching over my PS4 connection the last 2-3 days. A lot of general PS4 traffic, and download's are passing data between ports in the 49k to 65k range. One other thing PS4's party chat, normally passing voice traffic over port 9307.. Which I seen the last couple days while in party chats. But I do know PS4 list I think between 9304-9309 as ports voice chat will use, depending on who you're in voice chat with, and if you have multiple PS4's online at the same time, using voice party chat.