[Release] FreshJR Adaptive QOS (Improvements / Custom Rules / and Inner workings)

[FreshJR]

I'm not 100% sure. The Aruba doesn't register any tracked connections until unplugged but it does show data use in traffic analyzer. I feel like it may have been going into "Others" but I can't say for sure.

Tracked connections happens before any script modifications take place.

The (5-minute) glitch would be that traffic would end up in the wrong pie-chart.

If its not showing up in tracked connections, something else is going on outside of my modifications. Wait for the upcomming fix, but I am not sure if it will resolve your issue.

 

[RMerlin]

Haha I did say that, but I held off the initial release until I had everything implemented.

All of these .X releases in the last two days mostly were minor bug fixes and user requested features.



Nice, I will have to take a look at it, since that is the ONLY one I used. I saw its performance impact negligible at 500 connections.

My sort approach does NOT sort the entire bwdpi_conntack list.
It only sorts whatever will be shown to the user so it's capped at 500 possible entries vs however large the connections table data could get.

Also you can stick the modded qos_stats page over the existing one without installing the script.

(It will function fine, just don't press the apply as that will set two nvram variables. The page and script and completely decoupled sans sharing two nvram variables)

I ended up re-writing most of the table code, and reused my sort code that I had added to table.js, as well as some of my code from the Traffic Monitor page. Also added a few enhancements that I wasn't able to implement before with a Table object.

https://github.com/RMerl/asuswrt-merlin.ng/commit/54755feb1cfbeb16304b87f3e1908927e0dfbc38

I'll tackle list filtering/capping later, it's well past my bedtime here... Probably a few things in need of further tinkering by a brain that's not half asleep.

 

[Grimluhn]

Tracked connections happens before any script modifications take place.

The (5-minute) glitch would be that traffic would end up in the wrong pie-chart.

If its not showing up in tracked connections, something else is going on outside of my modifications. Wait for the upcomming fix, but I am not sure if it will resolve your issue.

Hey so I'm not big on networking or anything. I play Black ops4 on Xbox and the actual in game packets are catagorized as "untracked" and is put into other on your script. So I moved others to the top priority and gave my Xbox highest priority. And gave it a static IP on the Asus LAN tab. Is that all I should do right? And also another question is when my router restarts, they freshjr customize settings are gone and I had to go to putty and re install it? Is that normal? Hope this helps other call of duty Xbox people if they see this thread. Thanks

 

[Bishi]

On a side note, is there a reason its 5 minutes before the script kicks in? Could I override it to a shorter amount?

 

[Vexira]

On a side note, is there a reason its 5 minutes before the script kicks in? Could I override it to a shorter amount?

There used to be an instant script before but it had issues, the five minutes delay is to prevent some issue with the kernel if I remember correctly, or something my brain is too fuzzy.

 

[TicoMan]

Update: Manual install instructions && installers for users on stock AsusWRT have been posted

Update: v8.4 aswell (there was an additional bug report open on the github page I missed)
AppDB - 12003F was not functioning as expected

@TicoMan Are you sure the Sony Download was not actually going into Web Surfing which is above video streaming.
That is the only reason video streaming should/would be choking.
(You have to really observe the rates next to the pie charts for a complete picture).

You have many options to choose experiment with

1) If your game download is going into web surfing then to remove it from that section, as a very aggressive rule you can take all TCP traffic from remote ports 80,443 but only dealing with your game consoles CIDR range and stick it into "Game Transferring"

2) Or slightly less aggressive, you can take all TCP traffic from remote ports 80,443 with a current traffic mark of (13**** or 14****) but only dealing with your game consoles CIDR range and stick it into "Game Transferring".

Not sure which mark your game download was detected as. Do some more investigation if you need 13****, 14****, both, or simply the aggressive rule which swallows ALL the marks.

3) You can also bump the guaranteed rate higher for video streaming, so the new minimum guaranteed rate causes no buffering irreguardless what is happening elsewhere by taking some bandwidth away from other categories.

4) You can put Video Streaming above Web Surfing

There's lots of options at your disposal .... experiment

Typical traffic on ports and protocols was explained, and should be apparent when studying tracked connections. You then can spot more patterns with MARK for specific use cases.

--

Also if you ONLY have one gaming device active at a time.
You can simply stick all its traffic (Game + Game Downloads + ALL) into "Gaming" and place "Gaming" 2nd from bottom. Since it is only one device in the category, the console should be smart enough to PAUSE game transfers while it is actively playing online. Even though you placed it 2nd to the bottom, until it exceeds its guaranteed bandwidth it will still have first dibs on the network access.

--

@TicoMan

Report back, game transfers showing up in "Web Surfing" was your issue.
I can cook up another optional gaming rule to make more assumptions and stick MORE stuff into "Game Transfers"

Hey @FreshJR! sorry for the delay, I was out of town. So traffic was definitely not showing as "Web Surfing", it was all "Game Transfering". Priorities on UI are, Video Streaming, VOIP, Gaming...etc.

Good news is, it seems to be working now . What I did was:

1. Updated your script to latest version.
2. Added my PS4 IP/CIDR to the Gaming class.
3. Added my IPTV box IP/CIDR to a new Streaming class.

Right now PS4 is downloading a game and IPTV running with no buffering.

Appreciate your support and great work with this gem. PS. what was that link to donate a beer? I would like to invite you one.

 

[FreshJR]

@TicoMan
Thanks! I appreciate the support. There is a paypal link is available at the top of the third post.

Looking back, I think you were affected by the whitelisted game download traffic bug when you first reported your issue. IF your connection to the Sony servers was already opened before QOS start or during the 5 minute modification wait the download would become glitched/whitelisted.

A fix is coming for this. (ALL connections started after the initial wait should work reliably)

If it happens again, keep an eye on the rates and pie charts.

@Grimluhn
You didn’t have to move others. The gaming rule would do it for you.

@Bishi
I allow some time for adaptiveQOS to finish setting itself up and apply my modicafions afterwards.

 

[CooCooCaChoo]

There used to be an instant script before but it had issues, the five minutes delay is to prevent some issue with the kernel if I remember correctly, or something my brain is too fuzzy.

Probably waiting for the WAN interface to come up. I have a script to change the initcwnd and initrwnd in wan-start, but won't run properly. Even though it is in the wan-start script file, I have to place a two minute sleep timer before executing my script.

 

[FreshJR]

Probably waiting for the WAN interface to come up. I have a script to change the initcwnd and initrwnd in wan-start, but won't run properly. Even though it is in the wan-start script file, I have to place a two minute sleep timer before executing my script.

Was a different set of issues. For the instant version of the script I was intercepting original QOS calls and overwriting them with modified values upon creation.

(Sounded great in theory)

1) You could NOT concurrenty read nvram while QOS commands were similtanesouly issued or the system could go crazy

2) If command interception process took too long QOS would freak out and restart the entire structuee

I figured out how to disable the Asus checks (same ones spamming the log) and resolve both the above issues. I decided that this approach may not remain working with future updates (reliability concerns) so I pulled development on that end.

As it stands issue #2 still even occurs to a minor extent WITHOUT any external qos modifications. The QOS structure actually resets 2-3 times when it is turned on or restarted.

(It used to not reset at all before that system log error was introduced. Before the log errors a 90sec delay was more than sufficient)

With Asus recently introduced glitchy behavior the delayed start was recently changed from 90sec -> 5min. Watch the pie charts during startup if you want to watch the structure resets for yourself. Of course any script modifications have to be sent after the last reset for them to remain in effect.

Long story short

-I don’t see issues with the delayed start method.

-The hacky solution to fix the Asus errors was shelfed due to concerns on future reliability.

-I won’t be developing the instant method as the structure is finicky WITHOUT any modulations applied.

Just wasn’t worth the headache.

 

[Kingp1n]

@FreshJR ...im I assuming the next update will be a 9.0 finalized or will you need folks to test prior to making final?

 

[RMerlin]

@FreshJR Had a 10 hours long ISP outage. Looks like some clients insisted on spamming the network during the outage, check the number of tracked connections (2130!)

Sadly, the number went down to 150 before I could remove my current hardcoded limits to see how the router and the browser handled such a large list...

 

[FreshJR]

@RMerlin

It should honestly perform fine (estimating 100ms). I just didn’t like the long scroll bar so I’m keeping mine capped.

I will fill my version with 2500 fake entries and let you know if you are curious.

I’ll break the time stamps into three sections.
1) FreshJR mod evaluating all connections for rule overrides
2) Sorting data (double sort - 2 column)
3) Drawing table

 

[RMerlin]

It should honestly perform fine (estimating 100ms).

That one page load I did when that happened didn't seem to be particularly slow - currently capped at 500 entries shown, and auto-refresh disabled as it was above 750 entries (I didn't know how the router CPU would deal with httpd having to re-generate the list every 3 seconds).

I now have realtime list filtering implemented. List dynamically refreshes as you type in any of the filter fields (and you can even use multiple fields at once):

 

[FreshJR]

That one page load I did when that happened didn't seem to be particularly slow - currently capped at 500 entries shown, and auto-refresh disabled as it was above 750 entries (I didn't know how the router CPU would deal with httpd having to re-generate the list every 3 seconds).

I now have realtime list filtering implemented. List dynamically refreshes as you type in any of the filter fields (and you can even use multiple fields at once):

View attachment 16564

Yup realtime filtering shouldn't cause ANY performance drop.

I do realtime filtering 22 seprate times to evaulate if any entries in the "Tracked Connection" table need overrides as a result of script modifications.

Here are some stats.

3000 connections on a 2.4ghz - i3-2400
----------
4ms - parse rules
5ms - sort data
165ms - generate table
177ms - TOTAL
----------
4ms - parse rules
6ms - sort data
229ms - generate table
242ms - TOTAL
 ----------
4ms - parse rules
6ms - sort data
198ms - generate table
211ms - TOTAL
 ----------
4ms - parse rules
4ms - sort data
166ms - generate table
177ms - TOTAL
----------
4ms - parse rules
4ms - sort data
204ms - generate table
215ms - TOTAL
----------

still performs better than 200 connections using tableApi

EDIT:

WOW, it feels like it renders faster on my iPhone. What the hell in the process of pulling stats from the mobile device.

 

[RMerlin]

Yup realtime filtering shouldn't cause ANY performance drop.

Correct, as long it's properly implemented. There are a few more tricks I could implement to make it even more optimal (unfiltered cases add 6 checks per nodes to see if there is any filter value present), but it's already at a point where unless you browse on a Celeron 300A, it shouldn't make a visible difference... Overoptimization can lead to messy code, for the sake of 2-3 ms here and there.

Here are some stats.

Those are very reasonable, considering that's on an i3. Throughout development I was more worried about the router's CPU load when generating the list, as not everyone has a BCM4908 @ 1.8 GHz running httpd. I hadn't realized immediately I had over 2000 entries, so at least the AX88U CPU can deal with 2000 entries just fine. I might bump the auto-refresh cap from 750 to 1000 then (to leave some wiggle room for slower BCM4708 @ 800 MHz).

 

[RMerlin]

WOW, it feels like it renders faster on my iPhone. What the hell in the process of pulling stats from the mobile device.

The JS engine used by the browser will have a fairly significant impact on performance. Some JS functions (like array splicing) is much more optimal in some engines than others, for example.

The iPhone's JS engine is generally doing really well overall.

 

[FreshJR]

Here are the numbers from my iphone

8ms-parse rules
24ms-sort
26ms-generate table
60ms-total

9ms-parse rules
23ms-sort
83ms-generate table
112ms-total

It is rendering at double/triple the speed of my desktop ....
1w tdp vs 65w ..... nice

(No idea why table generation jumps from 26ms -> 83ms)

 

[RMerlin]

Here are the numbers from my iphone

8ms-parse rules
24ms-sort
26ms-generate table
60ms-total

9ms-parse rules
23ms-sort
83ms-generate table
112ms-total

It is rendering at double/triple the speed of my desktop ....
1w tdp vs 65w ..... nice

(No idea why table generation jumps from 26ms -> 83ms)

Any add-on on the PC browser? It could be negatively affecting performance.

 

[FreshJR]

Any add-on on the PC browser? It could be negatively affecting performance.

Only adguard, but I do also adguard running on mobile safari aswell.

Disabling adguard only brings it down ~ 10ms.

Ill just accept the fact the they achieved great single threaded performance on these mobile chips.

 

[RMerlin]

Only adguard, but I do also adguard running on mobile safari aswell.

Disabling adguard only brings it down ~ 10ms.

Ill just accept the fact the they achieved great single threaded performance on these mobile chips.

If you want to further optimize performance, try merging multiple "code +=" into a single one (you can split it accross multiple lines). It might improve performance inside loops. Also, "row_tr_idx" is not a valid HTML attribute, you can remove that from your nodes.