[Release] FreshJR Adaptive QOS (Improvements / Custom Rules / and Inner workings)

[Jack Yaz]

@Adamm argg, you are right. (It’s late).
The changes from 8.5 -> 8.6 introduced a mount bug affecting ALL users rather than expanding functionality to support user compiled firmwares.

@Jack Yaz originally the code logic was one block of (if NOT RMerlin) while the other two blocks were (IF RMerlin)

I copy pasted the new alternate RMerlin detection block three times and this turned all three logic statements into (if NOT RMerlin) which was not intended. I forgot they originally differed.

Pushing another fix...

--

Update v8.7 is out
-fix mount bug introduced in v8.6 changes

Apologize for the update roller coaster today. Stuff kept popping up.
Hopefully everything is now settled for a nice calm stretch of time.

Thanks for the clarification. I'll merge your latest changes into my fork

 

[FreshJR]

At line #193 is this a copy/paste error?

And the break at #2150 probably shouldn't be there.

Correct, upload rule line #193 was glitched. Thanks

That break @#2150 was meant to explicitly exit the switch (case) statement and to continue on with the traditional installation.

It was placed there for redundancy just incase the switch statement had "fall-through" logic. That particular switch statement did not have fall-through logic, but sometimes they do.

I will test that portion again right now. But that particular break shouldn't harm anything.

Thanks for the report.

 

[FreshJR]

#update v8.8
-bugfix iptable rule3 (upload portion)
-changes to meet amtm compliance

 

[FreshJR]

Can you add this Update button?
To be able to check and update the script to the latest version by the webui.
(I do not know if it's okay in that place)

Everything from my WebUI is a series of workarounds strung together. Even though it looks the same, I am at a BIG disadvantage compared to other pages since they DO have access to native functions.

They can execute code on the router or grab data. I can’t do either. So no, I can’t trigger the update or any terminal commands from the WebUI. This would require me recompiling the firmware with custom hooks and is not a good direction to take.

The most I can do would be after clicking update, it would set a nvram variable indicating that the script should for updates at its next daily scheduled check. I don’t like this non instance approach.

Don’t worry the update button is not needed.

Everything has settled and there shouldn’t be many more updates.

It usually goes for a LONG while before something or another breaks

 

[iaTa]

Upgraded from v7 to v8.8 on AC68U firmware v380.70.

System log looks fine but script doesn't seem to be working. Usenet traffic on port 563 stays in VOIP/IM instead of File Transferring.

Any ideas? Should I just roll back to v7?

EDIT: Rolled back to v7 and having the same issue. Hmmm.

 

[FreshJR]

Upgraded from v7 to v8.8 on AC68U firmware v380.70.

System log looks fine but script doesn't seem to be working. Usenet traffic on port 563 stays in VOIP/IM instead of File Transferring.

Any ideas? Should I just roll back to v7?

EDIT: Rolled back to v7 and having the same issue. Hmmm.

Try power cycling the router.

Post results of

iptables -vL -t mangle

(How is your traffic appearing in the tracked connections table, is it on the expected Usenet ports)

 

[iaTa]

Try power cycling the router.

Post results of

iptables -vL -t mangle

(How is your traffic appearing in the tracked cooler nextions table, is it on the expected Usenet ports)

Tried power cycling and waiting 5 minutes for all QoS rules to be applied and same result.

iptables -vL -t mangle
Chain PREROUTING (policy ACCEPT 22425 packets, 3939K bytes)
 pkts bytes target     prot opt in     out     source               destination        
  650 98689 BWDPI_FILTER  udp  --  eth0   any     anywhere             anywhere          

Chain INPUT (policy ACCEPT 19101 packets, 3166K bytes)
 pkts bytes target     prot opt in     out     source               destination        

Chain FORWARD (policy ACCEPT 3057 packets, 721K bytes)
 pkts bytes target     prot opt in     out     source               destination        
    0     0 MARK       all  --  any    br0     192.168.1.0/24       192.168.1.0/24       MARK xset 0x1/0x7

Chain OUTPUT (policy ACCEPT 20747 packets, 17M bytes)
 pkts bytes target     prot opt in     out     source               destination        
    0     0 MARK       udp  --  any    eth0    anywhere             anywhere             multiport dports  !domain,ntp MARK set 0x40030001
    0     0 MARK       tcp  --  any    eth0    anywhere             anywhere             multiport dports  !domain,ntp MARK set 0x40030001

Chain POSTROUTING (policy ACCEPT 23804 packets, 18M bytes)
 pkts bytes target     prot opt in     out     source               destination        
    0     0 MARK       udp  --  any    br0     anywhere             anywhere             multiport sports 500,4500 MARK set 0x80060001
    0     0 MARK       udp  --  any    br0     anywhere             anywhere             udp dpts:16384:16415 MARK set 0x80060001
    0     0 MARK       tcp  --  any    br0     anywhere             anywhere             multiport sports nntp,snews MARK set 0x80030001
    0     0 MARK       all  --  any    br0     anywhere             anywhere             mark match 0x40000000/0xc0000000 MARK xset 0x80000000/0xc0000000
   64 48533 MARK       tcp  --  any    br0     anywhere             anywhere             mark match 0x80080000/0xc03f0000 multiport sports www,https MARK set 0x803f0001
    0     0 MARK       udp  --  any    br0     anywhere             anywhere             udp spt:8621 MARK set 0x80040001
    0     0 MARK       udp  --  any    eth0    anywhere             anywhere             multiport dports 500,4500 MARK set 0x40060001
    0     0 MARK       udp  --  any    eth0    anywhere             anywhere             udp spts:16384:16415 MARK set 0x40060001
    0     0 MARK       tcp  --  any    eth0    anywhere             anywhere             multiport dports nntp,snews MARK set 0x40030001
    0     0 MARK       tcp  --  any    eth0    anywhere             anywhere             mark match 0x40080000/0xc03f0000 multiport sports www,https MARK set 0x403f0001
    0     0 MARK       udp  --  any    eth0    anywhere             anywhere             udp dpt:8621 MARK set 0x40040001

Chain BWDPI_FILTER (1 references)
 pkts bytes target     prot opt in     out     source               destination        
    0     0 DROP       udp  --  eth0   any     anywhere             anywhere             udp spt:bootpc dpt:bootps
    0     0 DROP       udp  --  eth0   any     anywhere             anywhere             udp spt:bootps dpt:bootpc

Traffic is showing as NNTP in the app analysis table.

 

[FreshJR]

Iptables is showing zero hits for the built in Usenet rule so it seems your Usenet transfers are on non standard ports.

Since DPI picking it up as Usenet, why don’t you create a rule for the Usenet mark (nntp) into downloads... no brainer

 

[iaTa]

Iptables is showing zero hits for the built in Usenet rule so it seems your Usenet transfers are on non standard ports.

Since DPI picking it up as Usenet, why don’t you create a rule for the Usenet mark (nntp) into downloads... no brainer

All of my usenet traffic is running on port 563 so why would that be the case?

Sorry, create a rule where? I've tried adding a rule using your script menu but you already have a default rule for usenet traffic on tcp 563 in v8. I used to add the rule manually in v7.

 

[FreshJR]

Local or remote port 563?

 

[iaTa]

Remote.

 

[FreshJR]

Remote.

TCP remote port 563?

Because this rule is not hitting.

0 0 MARK tcp -- any br0 anywhere anywhere multiport sports nntp,snews MARK set 0x80030001

Or just look up the mark from the tracked connection table or appdb and redirect that.

 

[iaTa]

Tes TCP remote port 563.

/jffs/scripts/FreshJR_QOS -appdb "NNTP"
NNTP
 Originally:  VoIP
 Mark:        050004

Why would NNTP state originally VoIP?

I added an Appdb rule for mark 050004 to destination downloads and it's working now.

 

[FreshJR]

Why would NNTP state originally VoIP?

I added an Appdb rule for mark 050004 to destination downloads and it's working now.

Because the programmer of the dpi database wanted boosted priority for his downloads?

Because news groups were originally forums?

No one knows.

The more important question is why the iptables rules wasn’t hitting.

Where are you getting your local port / remote port data from since iptables is rock solid.

 

[iaTa]

Because the programmer of the dpi database wanted boosted priority for his downloads?

Because news groups were originally forums?

No one knows.

The more important question is why the iptables rules wasn’t hitting.

Where are you getting your local port / remote port data from since iptables is rock solid.

Ah ok.

LiveTcpUdpWatch: https://i.imgur.com/P7Pvh6W.jpg

 

[FreshJR]

Ah ok.

LiveTcpUdpWatch: https://i.imgur.com/P7Pvh6W.jpg

That was the issue.

The built in rules are ipv4 ONLY (see table name). Your usenet traffic was ipv6.

IPv6 support is kinda tricky.

The implementation is not complicated but there is not much room in nvram to store an additional 4 rules especially with longer ipv6 addresses.

Some hacky workarounds remain at my disposal (base conversion for shorter encoding) so ipv6 support may potentially be introduced at a later date if I manage to make all the rules fit within 510chars.

(Integration with the tracked connection table for ipv6 addresses remains as another goofy issue since the table ipv6 remote and local ips truncated and incomplete)

For now, I may duplicate ipv4 rules as ipv6 rules IF they don’t contain a filtering element by local or remote ip.

 

[iaTa]

That would explain it! My ISP has only recently activated IPv6.

Thank you for your help and look forward to the IPv6 tweaks.

 

[lilstone87]

Well today I turned my PS4 on, and it had a couple game updates, and a software update to do. However during all this.. I noticed on the "Traffic classification" page of the router, it showed no data being downloaded under any of the classifications. Something doesn't seem right with that... I do have my PS4 set under the gaming rule with it's internal IP/32.

@FreshJR You have any ideal why no data is being shown actively while the PS4 is downloading updates, and such? I even paused the download, and updated from 8.5 to 8.8 version. Waited over 5 minutes for everything to set after the QoS refreshed with the script update. Still nothing showed under "Game Transferring" or any of the other classifications. So this has me curious as to what's happening.. Where's this traffic being put, and under what priority is it getting?

EDIT: It appears some of this traffic is being classified as "Unmarked". Which I'm guessing is the reason no data is being actively shown as used while some of it's being downloaded. Still I find that odd.. What exactly is the "Unmarked" rule set for?

 

[FreshJR]

Untracked goes to others.
It may or may not go to "Gaming" for the specified gaming rule devices.

If your game downloads in are appearing in wrong categories, then you can create another rule to correct this.

 

[lilstone87]

@FreshJR
That's the thing though... When this was happening, not even "others" showed data being actively passed when my PS4 was doing a couple download updates. It was as if nothing was being downloaded according to the "Traffic classification" page.