[Release] FreshJR Adaptive QOS (Improvements / Custom Rules / and Inner workings)

[lilstone87]

@FreshJR Here's a image of what the "Traffic classification" page looked like while the download was running. Oddly no large amounts of data shows on the download side. However the upload side it oddly shows 300KB's of data being passed over "Gaming", Which I would think should of been under "Game Transferring" as it relates to the game update being downloaded. Something is for sure off.

Also here's what the "Bandwidth monitor" page showed while the download was running. I don't know what to say.. I don't have any custom rules running either. I just have the one game rule that's already there, attached to my PS4's internal IP/32.

EDIT: Well I removed my PS4's IP from the Gaming rule that is set by default. I started the download back up, and it actually showed the download under "Game Transferring". Look at Traffic Classification 2 image. Still I don't like how the upload data being passed by the game download, is showing under "Gaming" though.

 

[FreshJR]

@lilstone87 thanks for the detailed report.

Can you reenable the glitched rule and send me the output of

iptables -vL -t mangle

To clarify. With the gaming rule enabled, the traffic disappears entirely for QOS?

 

[lilstone87]

@FreshJR Here.

Chain PREROUTING (policy ACCEPT 4201 packets, 426K bytes)
pkts bytes target prot opt in out source destination
304 38288 BWDPI_FILTER udp -- eth0 any anywhere anywhere

Chain INPUT (policy ACCEPT 2226 packets, 169K bytes)
pkts bytes target prot opt in out source destination

Chain FORWARD (policy ACCEPT 1905 packets, 251K bytes)
pkts bytes target prot opt in out source destination
0 0 MARK all -- any br0 192.168.1.0/24 192.168.1.0/24 MARK xset 0x1/0x7

Chain OUTPUT (policy ACCEPT 3862 packets, 3037K bytes)
pkts bytes target prot opt in out source destination
0 0 MARK udp -- any eth0 anywhere anywhere multiport dports !domain,ntp MARK set 0x40030001
0 0 MARK tcp -- any eth0 anywhere anywhere multiport dports !domain,ntp MARK set 0x40030001

Chain POSTROUTING (policy ACCEPT 5753 packets, 3288K bytes)
pkts bytes target prot opt in out source destination
2 216 MARK udp -- any br0 anywhere anywhere multiport sports 500,4500 MARK set 0x80060001
0 0 MARK udp -- any br0 anywhere anywhere udp dpts:16384:16415 MARK set 0x80060001
0 0 MARK tcp -- any br0 anywhere anywhere multiport sports nntp,snews MARK set 0x80030001
0 0 MARK all -- any br0 anywhere anywhere mark match 0x40000000/0xc0000000 MARK xset 0x80000000/0xc0000000
11 2046 MARK tcp -- any br0 anywhere anywhere mark match 0x80080000/0xc03f0000 multiport sports www,https MARK set 0x803f0001
0 0 MARK tcp -- any br0 anywhere 192.168.1.48 mark match 0x80000000/0x8000ffff multiport sports !www,433 MARK set 0x80080001 <--443(?)
0 0 MARK udp -- any br0 anywhere 192.168.1.48 mark match 0x80000000/0x8000ffff multiport sports !www,https MARK set 0x80080001
2 216 MARK udp -- any eth0 anywhere anywhere multiport dports 500,4500 MARK set 0x40060001
0 0 MARK udp -- any eth0 anywhere anywhere udp spts:16384:16415 MARK set 0x40060001
0 0 MARK tcp -- any eth0 anywhere anywhere multiport dports nntp,snews MARK set 0x40030001
0 0 MARK tcp -- any eth0 anywhere anywhere mark match 0x40080000/0xc03f0000 multiport sports www,https MARK set 0x403f0001
0 0 MARK tcp -- any eth0 192.168.1.48 anywhere mark match 0x40000000/0x4000ffff multiport dports !www,https MARK set 0x40080001
0 0 MARK udp -- any eth0 192.168.1.48 anywhere mark match 0x40000000/0x4000ffff multiport dports !www,https MARK set 0x40080001

Chain BWDPI_FILTER (1 references)
pkts bytes target prot opt in out source destination
0 0 DROP udp -- eth0 any anywhere anywhere udp spt:bootpc dpt:bootps
2 788 DROP udp -- eth0 any anywhere anywhere udp spt:bootps dpt:bootpc


From what I can see.... Shouldn't the one port be 443? Not 433? I think someone mentioned this not to long ago as well. I might be totally wrong, just noticed that on that output.

 

[FreshJR]

@lilstone87 argg, I thought I fixed that.
(I think I only fixed one of the two rules)

1)I have to change that port to 443
2)I also have to fix game transferring detection on the upload rule. Should be dports instead of sports.

—

Anyway. Nothing from my end would cause traffic to be whitelisted from QOS.

No idea what happened. Can you replicate white listed traffic again ??

If it can happen again I can take a look into it.
Also see if it happens without my script installed.

—

Edit scratch that. I know what happened.

Iptables rules are created immiedetly after clicking apply. The one iptable table rule was sending it your game download to game transferring.

Game transferring doesn’t exist in TC until AFTER a 5 minute wait. TC also only evaluates connections once.

This means if a game transfer was started during the 5minute wait it would be whitelisted. It’s quite an unfortunate glitch due to how the structure is setup.

I have an idea on how to work around this.

 

[lilstone87]

@FreshJR Oddly after reapplying the "preset" gaming rule you have at the top to my PS4's IP. It's now showing the download under "Game Transferring". Still the upload data from the download showed under "Gaming". Which you're aware of.. I just seen your new reply. I wonder the glitch I ran across, that rule had stayed there from 8.0 to the current 8.8. So maybe it bugged out some how. Also when I went from 8.0 to 8.2 I had to reinstall the script due to not being able to update it via the script menu.

I do remember after I reinstalled the script, that gaming rule still had my PS4's IP listed. So maybe it bugged out because of that. Either way I'm happy it happen... as we seen the 433->443 port error, plus the upload data from the game file download, was being put under "Gaming" which should of been "Game Transferring".

Anyways I hope I didn't drive you to crazy.... @FreshJR LOL

 

[FreshJR]

Nope reread my last post. I got everything figured out.

2 things need fixing and one thing needs some creative thinking.

The reported bug was tricky to figure out what could be causing it. On the surface the rules look fine and bulletproof.

The problem was due to how TC and iptables interact and the timing of when the rules are applied.

 

[lilstone87]

I just reread your editted post
I'm happy we ran across these couple bugs, but in the same sense... I feel bad putting more on your plate. I know this past week you have been pushing out bug fixes a good amount. Just know we all appreciate it. This gamer for sure appreciates what you do Thanks again!! @FreshJR

 

[FreshJR]

It’ll settle down. Pretty sure this is the last bug!!

 

[lilstone87]

I hope so for your sake. You deserve some chill time.

 

[Jack Yaz]

I hope so for your sake. You deserve some chill time.
Netgear R8500 - Running latest, and greatest firmware

You're not running Asuswrt Merlin on that Netgear in your signature...are you?

 

[lilstone87]

@Jack Yaz No LOL... I actually need to remove that, I don't even have it anymore. I currently use a RT-AC3100 as my main router. I do have a Netgear R9000 as a repeater, and to run plex. Also have a Netgear XR500 I have used as my main router, off, and on. However with @RMerlin work, and @FreshJR QoS work. I'm pretty happy with my AC3100 right now.

 

[Jack Yaz]

@Jack Yaz No LOL... I actually need to remove that, I don't even have it anymore. I currently use a RT-AC3100 as my main router. I do have a Netgear R9000 as a repeater, and to run plex. Also have a Netgear XR500 I have used as my main router, off, and on. However with @RMerlin work, and @FreshJR QoS work. I'm pretty happy with my AC3100 right now.

All good then

 

[Adamm]

Small bug, the custom web-page is only ever downloaded during the update function, not during a fresh install (line 1859). This leads to the page not mounting until you run the update function.

 

[FreshJR]

Small bug, the custom web-page is only ever downloaded during the update function, not during a fresh install (line 1859). This leads to the page not mounting until you run the update function.

Actually, the fresh install command in the first post two curl requests. You are not downloading the WebUI page since you are not using the intended install command.

I do see your point.

There is no reason not to have it one curl request, and have the -install function perform the second request itself for consistency between the -install and -update arguments.

 

[Adamm]

Actually, the fresh install command in the first post two curl requests. You are not downloading the WebUI page since you are not using the intended install command.

I do see your point.

There is no reason not to have it one curl request, and have the -install function perform the second request itself for consistency between the -install and -update arguments.

Ah my mistake, I was using the command on your Github readme, I guess its outdated. Regardless its probably best to include it in the install procedure, no need for its own command.

 

[tmcb82]

Is the change in font size in the Application field a bug or by design?

 

[RMerlin]

Is the change in font size in the Application field a bug or by design?

The font size is dynamic - labels that are too long will use a smaller font to be able to fit. It's possible that the size calculation is done on the original label instead of the "custom" label shown here.

 

[FreshJR]

Is the change in font size in the Application field a bug or by design?

The intended action is that font size will be reduced by 25% if the AppDB name is very long.

What happened in the screenshot is that the label was very long and later had its name overwritten to relatively short Rule2. The previous font reduction in remained place.

I will push an update to query the length of the label a later point in the code for proper evaluation.
(Or potentially truncate with an appended ... like I did with long device names)

 

[heinz]

Does this script auto update or is there something required to push new updates to it?

 

[Mutzli]

Does this app auto update or is there something required to push new updates to it?

Use the following command:
/jffs/scripts/FreshJR_QOS -update