[Release] FreshJR Adaptive QOS (Improvements / Custom Rules / and Inner workings)

[FreshJR]

lThroughout development I was more worried about the router's CPU load when generating the list .... I might bump the auto-refresh cap from 750 to 1000 then (to leave some wiggle room for slower BCM4708 @ 800 MHz).

That’s me wiggling at the low end.

But as we discussed earlier in PM.

If cpu usuage becomes a problem there is a way for the router to do ZERO work by offloading all the processing to the device using the webpage.

1)symlink tracked connections plaintext output from /proc into /www

2) preparse appdb into a json object saved as plain text in /www. Repeat this only when the appdb file timestamp changes

Appdb json object can be loaded on page load. (objects and arrays already act as hashtables in Java)

Proc output can be requested & processed anytime line by line for relevant entries.

—
I already cut down on += as much as I could. I probably could get creative with css to remove some more, but I still intend on keeping a 500 connection limit so I fine with current performance.

 

[RMerlin]

1)symlink the tracked connections output from /proc into /www

Don't. Exposing any internal proc interface to the world is a security disaster just waiting to happen...

2) reparse appdb into a json object saved as plain text in /www aswell (repeat this every time the appdb timestamp chnages)

Keep in mind that it means a much larger web page to push to the client. Extra CPU load on the router if using TLS, and you still need a good amount of CPU processing to generate the json. I suspect the current indexed table is more efficient overall, especially as httpd will cache the data, refreshing it only if it gets changed by a rule database update. And we're dealing with native C code versus interpreted code.

 

[RMerlin]

I already cut down on += as much as I could. Probably could get creatives to remove somebody more, but I am keeping a 500 connection limit so I have no need to mess with it any more.

I was particularly looking at this block of code, which is in a loop:

                       code += '<tr class="row_tr data_tr" row_tr_idx="' + i +' ">';
                                code += '<td>' + tabledata[i][0] +'</td>';
                                code += '<td>' + tabledata[i][1] +'</td>';
                                code += '<td>' + tabledata[i][2] +'</td>';
                                code += '<td>' + tabledata[i][3] +'</td>';
                                code += '<td>' + tabledata[i][4] +'</td>';
                                code += '<td>' + tabledata[i][5] +'</td></tr>';

Possible that I have an older version here tho.

 

[FreshJR]

Possible that I have an older version here tho.

No that’s the current version.

I vagely remember performing A/B benchmarking both [subsequent +=] vs [ consecutive + then +=]

Seems the changes disappeared somewhere on the path to release or mabye the results were the same and I left it more readable.

 

[bimdimbo]

Hi. First thanks to freshJR for your work! I've had my AC68U for a while and I'm just starting to look at QOS now that my kid is old enough to be complaining about 'lag' while he's playing Fornite and I'm doing some work related uploads!

I have the latest stable Merlin firmware installed and I've installed the QOS script succesfully. As far as I can tell all is well, I'm just wondering if I've got the CIDR bit right and correctly inputted. My router IP is 192.168.0.1. His XBOX is manually assigned an IP of 192.168.0.184. From what I can figure out the CIDR number that I put in the first row of the 'iptables rules' is 192.168.0.0/24. Is that correct if my subnet is 255.255.255.0?

Appreciate any help if I've got it wrong, I'd never heard of CIDR before tonight and some of the online calculators give different results depending on 'mask bits' or 'CIDR Netmask' etc...

 

[FreshJR]

No /24 is all your devices on the network.

/32 will achieve that single device.

(Since it’s a single device you can also omit /32 and the setting should take just fine.)

 

[bimdimbo]

OK thanks. I've changed it to just 192.168.0.184 for now. Will keep an eye on it tomorrow and see how it goes great stuff.

 

[bimdimbo]

Well have to come on and say thanks again to you freshJR, (and Merlin for the firmware). For the first time in this household I can leave torrents on, watch something in HDR on netflix, and my kids ping doesn't budge at all. Exceptional work, this did not happen using vanilla Adaptive QOS no matter what I tried.

 

[QuikSilver]

Would like to say thanks again @FreshJR , Thanks to the effects of everyone here (@Adamm, Merlin, Lonelycoder, the list goes on (if you were omitted it wasn't intentional). Thanks to the great support for my Asus 86u unit I went ahead and replaced an older DDWRT router at another location with much less bandwidth (18/1.5 (over provisioned thankfully). I'm able to stream TV with the wife while the kids still play games without an issue. Thanks so much!

 

[macster2075]

Does Discord fall in the Voip category?
When we play online, we prefer using Discord for chatting. Also, we do not use or need wifi calling. Even in my case, do you recommend setting Voip as first priority?

 

[Bishi]

Or bursts can be lowered for the streaming, file downloads, & game downloads category by manually modifying the script.
I am thinking about including this as a future toggle in the UI but I am currently busy.

For the time being I can supply an alternative curl link with VERY low bursts (hardcoded) for for users that are have this problem if they are interested in performing A/B testing.

FreshJR do you have a curl to this very low burst version please? I'd be happy to test.
I am still experiencing some lag in games (noticeable in Apex Legends since thats what im playing right now) when someone is using the LG TV for Netflix etc. The caps seem to be working (TV never goes over 15mb/s) and I have a high resolution ping running which doesn't have huge variance (stays about 8ms-11ms for my closest point @ ISP) but its very noticeable in game.
Only thing I haven't tried yet is SFQ instead of fq_codel but i'll try that now.

EDIT: Yeah SFQ didn't make any difference.

 

[L&LD]

Does Discord fall in the Voip category?
When we play online, we prefer using Discord for chatting. Also, we do not use or need wifi calling. Even in my case, do you recommend setting Voip as first priority?

Just monitor it when you're actively using Discord.

 

[L&LD]

FreshJR do you have a curl to this very low burst version please? I'd be happy to test.
I am still experiencing some lag in games (noticeable in Apex Legends since thats what im playing right now) when someone is using the LG TV for Netflix etc. The caps seem to be working (TV never goes over 15mb/s) and I have a high resolution ping running which doesn't have huge variance (stays about 8ms-11ms for my closest point @ ISP) but its very noticeable in game.
Only thing I haven't tried yet is SFQ instead of fq_codel but i'll try that now.

EDIT: Yeah SFQ didn't make any difference.

I think you can do that yourself with the included script, just put the settings how you require.

 

[FreshJR]

FreshJR do you have a curl to this very low burst version please?

Not yet. First I have to push the pending bug fixes. After that I will pushing the low burst version.

I’ve been busy this past week but keep checking in on this thread, i may have a link up by tmrw.

Does Discord fall in the Voip category?
When we play online, we prefer using Discord for chatting. Also, we do not use or need wifi calling. Even in my case, do you recommend setting Voip as first priority?

Check out

1)what app names are generating traffic under bandwidth monitor when discord is in use

2) cross reference that app name with the destination listed in the tracked connection table.

—

The order of the categories is not super important. If you re-read the first post you will see the order corresponds to which categories are offered excess bandwidth first.

If a category is not using it’s bandwidth, it always spills over to the category underneath.

Reread that post slowly. The operation is quite simple.

 

[Bishi]

Not yet. First I have to push the pending bug fixes. After that I will pushing the low burst version.

I’ve been busy this past week but keep checking in on this thread, i may have a link up by tmrw.

Excellent, thank you for all your hard work!

 

[L&LD]

Okay, I'm curious now. What is this low burst version? I've re-read the last couple of pages of posts here, but I must have missed it (again).

EDIT: Okay, Sorry, I found it. Just tired, I guess.

 

[FreshJR]

Okay, I'm curious now. What is this low burst version? I've re-read the last couple of pages of posts here, but I must have missed it (again).

EDIT: Okay, Sorry, I found it. Just tired, I guess.

Burst determines how many packets are zero-rated before QOS kicks in.

Not throttling the first few packets (few kB at most) allows you to reach max speeds quicker and expierence better responsiveness.

For some users, the defaults bursts are simply too high for their connection so this leads to 50-100ms duration spikes in ping when their connection is initially getting saturated.

I kept the default bursts, but will provide a version with lowered bursts.

 

[Sinner]

Does Discord fall in the Voip category?
When we play online, we prefer using Discord for chatting. Also, we do not use or need wifi calling. Even in my case, do you recommend setting Voip as first priority?

Discord falls into general which gets redirected into others by default.

 

[medwatt]

@FreshJR, is there something that can be done for downloads that recognized under "HTTP Protocol over TLS SSL" which is categorized as "Web" traffic. If I download anything through a browser, it gets recognized as such. Even downloading some files in MegaSync (megaupload download manager) is considered web traffic.

 

[FreshJR]

@FreshJR, is there something that can be done for downloads that recognized under "HTTP Protocol over TLS SSL" which is categorized as "Web" traffic. If I download anything through a browser, it gets recognized as such. Even downloading some files in MegaSync (megaupload download manager) is considered web traffic.

Having download traffic within Web Surfing (or anything mis-categorized) shouldn't the end of the world. The guaranteed rates per category should keep everything functioning.

All categories UNDERNEATH web surfing will always get their guaranteed rate.
(The only issue with HTTPS downloads in "WebSurfing" would be that "WebSurfing" will be using all the excess bandwidth available, and non of that excess will be offered to the categories underneath it).

In your position, I would simply tweak the guaranteed bandwidths or lower the priority of websurfing if something not working to your liking and call it a day.

--

As for your original question.

Iptables is able to track bytes transferred per individual connections & you can re-assign a traffic mark based on this information into whatever you desire.

The iptable rule would be

#HTTPS
#This rule moves a connection marked as 0x8010 - 8017 into ${Downloads} when it transfers over 1mb.
-m mark --mark 0x80100000/0xc0380000 -m connbytes --connbytes 1024000 --connbytes-dir both --connbytes-mode bytes -j MARK --set-mark ${Downloads_mark_down}

Additionally you may be interested in this rule to move HTTP traffic away from WebSurfing. (some http CDN transfers are counted as Web Surfing)

#HTTP
#This rule moves a connection marked as 0x800d into ${Downloads} when it transfers over 1mb.
-m mark --mark 0x800d0000/0xc03f0000 -m connbytes --connbytes 1024000 --connbytes-dir both --connbytes-mode bytes -j MARK --set-mark ${Downloads_mark_down}

NOTE:

With HW acceleration on, only the first packet of every connection is parsed.

This means the connection would never be re-evaluated at a later point in time when it transfers more than 1MB so the iptable rule (sending it to downloads) would NEVER hit
​

With HW acceleration off, every individual packet passing through is parsed

Since the connection is continually evaluated, the connection will change categories after it transfers more than 1MB.​