[Release] FreshJR Adaptive QOS (Improvements / Custom Rules / and Inner workings)

[medwatt]

Lol - a huge spike in Net Control. Clearly ASUS is misclassifying that traffic. Probably because it's google.

Well, then, that's exactly what I'm talking about. Is there a way to reclassify traffic ?

Check this other download: http://oxygenos.oneplus.net.s3.amaz...3_OTA_028_all_1801031502_109fde3709984ec8.zip

This should go under "Web Traffic"

 

[brummygit]

Lol - a huge spike in Net Control. Clearly ASUS is misclassifying that traffic. Probably because it's google.

My RT-AC68U on 382.3 Alpha 1 sees this as Web Traffic using signature file 2.050. What signature version are you running?

@deddc23efb I'm not yet using the new FreshJR script

 

[medwatt]

My RT-AC68U on 382.3 Alpha 1 sees this as Web Traffic using signature file 2.050. What signature version are you running?

@deddc23efb I'm not yet using the new FreshJR script

Mine is 1.200 (I'm on firmware 380.69) and it gets identified as Net Control Packets.

 

[deddc23efb]

I poked around a little bit. The 0x8014 classification is an indication of the application class. The 8 is direction and the actual category is 0x14 == 20 decimal. You can grep through the db files in /tmp/bwdpi/*.db to get a glimpse of what ASUS is doing. That classification is for Network protocols.

If you're willing to risk actual network protocols, you could delete that rule and write a new rule that mapped 0x8014 to the web treatment (it looks like you have web set to second priority? 0x11? There are three level of Network protocol classification (18,19,20) that all get mapped to 1:10 treatment.

I think this is truly a problem with the Trend Micro dpi.

<doh - 8013 not 8014 in all of this.>
/dedd

 

[deddc23efb]

My RT-AC68U on 382.3 Alpha 1 sees this as Web Traffic using signature file 2.050. What signature version are you running?

@deddc23efb I'm not yet using the new FreshJR script

That's very interesting. My signature file is 2.050 as well.
The mapping info I was looking at is in the /tmp/bwdpi directory. I'm completely guessing at the number that are thinking about (hex 14 == 20 == things in the db files) . Maybe those files aren't updated by a signature file update and the problem has been fixed in 382.3?

I disabled my FreshJr script and tried that download with after a clean reboot and it still goes to Net Control. I suspect this is a bug that ASUS has fixed and it requires an upgrade to something after 382.1_2 - sounds like 382.3 has the fix. Thanks brummygit!

/dedd

 

[FreshJR]

@medwatt On the previous versions we noticed that ASUS was missing a rule for "mark 0x80130000".

We noticed this because a lot of HTTPS traffic was bypassing QOS entirely. As such we found the missing rule for that classification and created it as shown below, in previous versions of the script.

${tc} filter add dev br0 protocol all prio 15 u32 match mark 0x80130000 0x803f0000 flowid ${Web}        #https traffic    (rule missing from ASUS default rule set)

Fast forward to today, On ASUS's updated firmware this rule is not missing anymore. You can see it present here.

filter parent 1: protocol all pref 22 u32 fh 802::800 order 2048 key ht 802 bkt 0 flowid 1:10
  mark 0x80130000 0xc03f0000 (success 9471)

Since we saw it was not missing on current firmware, I have removed my original, now duplicate, rule. As you pointed out, ASUS has decided to classify 0x8013 traffic as "Net Control", while I decided it was more appropriate to stick that traffic into "Web Surfing".

I didn't notice that ASUS decided to choose Net Control when the introduced the previously missing rule. I still think Web Surfing is way more appropriate. The old rule is coming back from as was setup on previous version of the script.

The "Web Surfing" rule is coming back. Thanks for pointing it out.

@Nodiaque I bet you downloaded the file from pastebin and uploaded it to the router. The below is in caps so it grabs other peoples attention who did the same thing.

THE PASTEBIN DOWNLOAD ADDS INCOMPATIBLE "NEW LINE" CHARACTERS. THIS MAKES THE SCRIPT NOT EXECUTE IN LINUX/SHELL.

Read the first post on how to convert the New Line characters using Notepad++. On previous versions, you probably downloaded the supplied ".txt" file on the forum. That one was directly uploaded from my computer so it had correct new line characters.

 

[deddc23efb]

@medwatt On the previous versions we noticed that ASUS was missing a rule for "mark 0x80130000".

We noticed this because a lot of HTTPS traffic was bypassing QOS entirely. As such we found the missing rule for that classification and created it as shown below, in previous versions of the script.

${tc} filter add dev br0 protocol all prio 15 u32 match mark 0x80130000 0x803f0000 flowid ${Web}        #https traffic    (rule missing from ASUS default rule set)

Fast forward to today, On ASUS's updated firmware this rule is not missing anymore. You can see it present here.

filter parent 1: protocol all pref 22 u32 fh 802::800 order 2048 key ht 802 bkt 0 flowid 1:10
  mark 0x80130000 0xc03f0000 (success 9471)

Since we saw it was not missing on current firmware, I have removed my original, now duplicate, rule. As you pointed out, ASUS has decided to classify 0x8013 traffic as "Net Control", while I decided it was more appropriate to stick that traffic into "Web Surfing".

I didn't notice that ASUS decided to choose Net Control when the introduced the previously missing rule. I still think Web Surfing is way more appropriate. The old rule is coming back from as was setup on previous version of the script.

The "Web Surfing" rule is coming back. Thanks for pointing it out.

@Nodiaque I bet you downloaded the file from pastebin and uploaded it to the router. The below is in caps so it grabs other peoples attention who did the same thing.

THE PASTEBIN DOWNLOAD ADDS INCOMPATIBLE "NEW LINE" CHARACTERS. THIS MAKES THE SCRIPT NOT EXECUTE IN LINUX/SHELL.

Read the first post on how to convert the New Line characters using Notepad++. On previous versions, you probably downloaded the supplied ".txt" file on the forum. That one was directly uploaded from my computer so it had correct new line characters.

If brummygit is correct, then ASUS has done something about this in 382.3 and classifying true network protocol traffic as web might be undesirable (or whatever ASUS is shoving in that class in 382.3) Something to watch out for.

People can fix their files using dos2unix in the ASUS cli as well.

 

[FreshJR]

If brummygit is correct, then ASUS has done something about this in 382.3 and classifying true network protocol traffic as web might be undesirable (or whatever ASUS is shoving in that class in 382.3) Something to watch out for.

People can fix their files using dos2unix in the ASUS cli as well.

Maybe, maybe not.

In 382.3 maybe @brummygit received a definitions update that specifically identifies that specific HTTPS traffic stream as websurfing.

We would have to test other HTTPS traffic streams to see if they don't fall into Net Control.

@brummygit can you post how you 0x8013 rule is setup from TC filter show dev br0?

Can you run a dslreports (https) version speedtest

Asus just might be identifying google drive HTTPS traffic. We should confirm.

While it goes into net control, I don't think it's actually syn and awks getting caught by 0x8013. It didn't create any issues before.

edit:

• on 382.2 signature version 2.050, the google docs document is "Web Surfing" and https dsl reports speedtest is "Net Control". Confirms my suspicions.

@medwatt The updated version of the script is below.

v382.2 version of FreshJR_QOS (typo present, use v382.3)
https://pastebin.com/pA10JSXT

*now put unidentified https traffic into "Web Surfing" instead of "Net Control" like old versions of the script.

 

[deddc23efb]

Maybe, maybe not.

In 382.3 maybe @brummygit received a definitions update that specifically identifies that specific HTTPS traffic stream as websurfing.

We would have to test other HTTPS traffic streams to see if they don't fall into Net Control.

@brummygit can you post how you 0x8013 rule is setup from TC filter show dev br0?

Can you run a dslreports (https) version speedtest

Asus just might be identifying google drive HTTPS traffic. We should confirm.

While it goes into net control, I don't think it's actually syn and awks getting caught by 0x8013. It didn't create any issues before.

Oddly enough on my system the Google TLS traffic is collecting against 0x8014

filter parent 1: protocol all pref 12 u32 fh 803::800 order 2048 key ht 803 bkt 0 flowid 1:10
mark 0x80090000 0xc03f0000 (success 299)
--
filter parent 1: protocol all pref 21 u32 fh 801::800 order 2048 key ht 801 bkt 0 flowid 1:10
mark 0x80120000 0xc03f0000 (success 58)
--
filter parent 1: protocol all pref 22 u32 fh 802::800 order 2048 key ht 802 bkt 0 flowid 1:10
mark 0x80130000 0xc03f0000 (success 33897)
--
filter parent 1: protocol all pref 23 u32 fh 804::800 order 2048 key ht 804 bkt 0 flowid 1:10
mark 0x80140000 0xc03f0000 (success 249435)

htpps://www.dslreports.com/speedtest goes to web surfing for me R382.1_2 .

 

[FreshJR]

Oddly enough on my system the Google TLS traffic is collecting against 0x8014

That's weird. I still decided to revert it. I ran for over a year with 0x8013 going into web surfing instead of net control.
On previous versions of ASUS's firmware that rule was missing entirely, LOL.

 

[medwatt]

That's weird. I still decided to revert it. I ran for over a year with 0x8013 going into web surfing instead of net control.
On previous versions of ASUS's firmware that rule was missing entirely, LOL.

If you can recall, most of my unidentified traffic when downloading was collected against 0x8014 in earlier firmwares. They are now identified against 0x8013 in 382.2

 

[FreshJR]

If you can recall, most of my unidentified traffic when downloading was collected against 0x8014 in earlier firmwares. They are now identified against 0x8013 in 382.2

Are you sure?

This line is straight from v1.92 in the first post.

${tc} filter add dev br0 protocol all prio 15 u32 match mark 0x80130000 0x803f0000 flowid ${Web}        #https traffic  (rule missing from ASUS default rule set)

It was clearly 0x8013 that was missing and giving issues.
While on the other hand 0x8014 is actual net control that should not be touched.

If https traffic is now collected against 8013 and going into net control, then it will go into web downloads now. That is the same behavior I am currently experiencing and the same behavior I experienced in the past.

 

[medwatt]

Are you sure?

This line is straight from v1.92 in the first post.

${tc} filter add dev br0 protocol all prio 15 u32 match mark 0x80130000 0x803f0000 flowid ${Web}        #https traffic  (rule missing from ASUS default rule set)

It was clearly 0x8013 that was missing and giving issues.
While on the other hand 0x8014 is actual net control that should not be touched.

If https traffic is now collected against 8013 and going into net control, then it will go into web downloads now. That is the same behavior I am currently experiencing and the same behavior I experienced in the past.

I did the changes myself. You can check the private conversation I had with you in the past for a pastebin file. Compare it with the one I posted today and you'll see.

 

[deddc23efb]

Yeah I think that was why I was confused earlier too. I'm definitely seeing it go to 0x8014 on 382.1_2. I just disabled FreshJR_QOS and restarted to make sure things were clean and that's what I get. I haven't tried upgrading to anything newer.

I installed the same rule as you are recommending FreshJR, except for 0x8014 and it works fine (as expected.) Not sure what else is getting pushed down to Web rather than Net. This is the problem with ASUS's approach here.

/dedd

 

[FreshJR]

Yeah I think that was why I was confused earlier too. I'm definitely seeing it go to 0x8014 on 382.1_2. I just disabled FreshJR_QOS and restarted to make sure things were clean and that's what I get. I haven't tried upgrading to anything newer.

I installed the same rule as you are recommending FreshJR, except for 0x8014 and it works fine (as expected.) Not sure what else is getting pushed down to Web rather than Net. This is the problem with ASUS's approach here.

/dedd

Can you post a few links that get pushed to 8014. This is odd/

 

[deddc23efb]

The one that medwatt provided is a great example:
https://drive.google.com/uc?id=18CnyqnxlVpLj1qTdt1I5RIhTBi2Uy4xn&export=download

 

[FreshJR]

Currently https traffic from google drive is hitting 0x800d & https version of dslreports is hitting 0x8013.

This grep filter works great to compare success hits upon fresh QOS restart with low traffic.

tc filter show dev br0 | grep "mark" | grep -v "success 0"

Also I made a typo in script v382.2. Ninja edit incoming. (Use v382.3)

 

[brummygit]

Maybe, maybe not.

@brummygit can you post how you 0x8013 rule is setup from TC filter show dev br0?

Can you run a dslreports (https) version speedtest

filter parent 1: protocol all pref 22 u32
filter parent 1: protocol all pref 22 u32 fh 802: ht divisor 1
filter parent 1: protocol all pref 22 u32 fh 802::800 order 2048 key ht 802 bkt                                                                                                              0 flowid 1:10
  mark 0x80130000 0xc03f0000 (success 1930057)

I can't run a speed test at the moment as my network is very busy

 

[h3lmut]

Currently https traffic from google drive is hitting 0x800d & https version of dslreports is hitting 0x8013.

This grep filter works great to compare success hits upon fresh QOS restart with low traffic.

tc filter show dev br0 | grep "mark" | grep -v "success 0"

Also I made a typo in script v382.2. Ninja edit incoming.

Here is 382.2 again, with typo fixed.
https://pastebin.com/pA10JSXT

dsl reports with https is in "net control" category with this script

with ac56u 382.2 Beta 3 (released today)

signature version: 2.050 Updated : 2018/01/06 01:00

 

[RMerlin]

dsl reports with https is in "net control" category with this script

with ac56u 382.2 Beta 3 (released today)

You should specify the exact version of the rules you are using, for future reference. The version can be seen on the Firmware Upgrade page.

Note that the rules from 380.xx (1.xxx) are completely different from 382.xx/384.xx (2.xxx)