[Release] FreshJR Adaptive QOS (Improvements / Custom Rules / and Inner workings)

[Carnagerover]

I just wanted to put an update to my post above to say that I took the plunge on the AC86U and have put both the Merlin firmware and added the script for FreshJR QOS. I tried the QOS without the FreshJR script first but I found that it was putting things in the wrong place, or setting a HTTPS download like a DSL reports speedtest to Net control packets which would put it right at the top of the list and not at all what I would want.

Then I put the FreshJR QOS on, I noticed that DSL reports was then going into Web Surfing which was fine with me, I tried a Newsgroup download which is HTTPS and that went into web surfing as well which is fine, ideally it would go into File Transferring but I'm ok with it. I followed the default setup of the categories but then I moved Video and Audio Streaming above Web Surfing so things like Twitch streams and Plex streaming which I think was being registered in Others is prioritised over the download traffic.

Doing the above I was able to start a newsgroup download and three seperate twitch streams which all worked at full quality without buffering which is almost miracle like dreamstate conditions. Thank you so much @FreshJR for this QOS!

 

[FreshJR]

I just wanted to put an update to my post above to say that I took the plunge on the AC86U and have put both the Merlin firmware and added the script for FreshJR QOS. I tried the QOS without the FreshJR script first but I found that it was putting things in the wrong place, or setting a HTTPS download like a DSL reports speedtest to Net control packets which would put it right at the top of the list and not at all what I would want.

Then I put the FreshJR QOS on, I noticed that DSL reports was then going into Web Surfing which was fine with me, I tried a Newsgroup download which is HTTPS and that went into web surfing as well which is fine, ideally it would go into File Transferring but I'm ok with it. I followed the default setup of the categories but then I moved Video and Audio Streaming above Web Surfing so things like Twitch streams and Plex streaming which I think was being registered in Others is prioritised over the download traffic.

Doing the above I was able to start a newsgroup download and three seperate twitch streams which all worked at full quality without buffering which is almost miracle like dreamstate conditions. Thank you so much @FreshJR for this QOS!

Yep. I redirected all https from net control -> web surfing with the default script config. I did not agree with the net control destination either.

Since it is one Usenet server, you can look into creating a custom rule to override the Usenet https identification into downloads if you wish.
(you can detect the Usenet server traffic by filtering by its IP)

---------------------MORE DOCUMENTATION--------------------

More information is available in the header of the script.

Open the script Notepad++ and read the green header comments!
Script Link:https://github.com/FreshPr/FreshJR_QOS/archive/master.zip

It is possible to:
--Categorize traffic into QOS container to/from LAN PC by its PORT.
--Categorize traffic into QOS container to/from LAN PC by its IP
--Categorize traffic into QOS container to/from WAN SERVER by its PORT.
--Categorize traffic into QOS container to/from WAN SERVER by its IP

Custom filtering rules:

Rule Templates to achieve the above are in the header of the script.

 

[Rob Q]

Hi, I'm trying to get this to work but I have no idea what I should put for 'WAN packet overhead'. I have cable (DOCSIS) internet, 40 Mbps down/10 Mbps up.

 

[FreshJR]

Hi, I'm trying to get this to work but I have no idea what I should put for 'WAN packet overhead'. I have cable (DOCSIS) internet, 40 Mbps down/10 Mbps up.

Doesn't make much of a difference, so don't worry about it too much.

Flip a coin and choose 0 or 18. Or just go with 18.

It only makes a difference in rare circumstances where a majority of your bandwidth is used to transfer many small packets which is not very common in todays landscape.

 

[Rob Q]

Thanks. Wasn't too sure what to enter to be honest....

The main thing that we do here is Netflix, I do play online games sometimes.

 

[FreshJR]

@Rob Q

We don't really have to see all this stuff.

Also those results don't show how well QOS is working.
The instructions said to use dslreports if you want to tune for optimal results.

Good luck

 

[Rob Q]

Hey, is it supposed to slow down my internet? Not sure why the speed dropped here.

 

[Carnagerover]

Yep. I redirected all https from net control -> web surfing with the default script config. I did not agree with the net control destination either.

Since it is one Usenet server, you can look into creating a custom rule to override the Usenet https identification into downloads if you wish.
(you can detect the Usenet server traffic by filtering by its IP)

I had a go at setting up some rules tonight and had a lot of fun, I have set one up for Plex so that now the traffic goes into the video streaming category whenever someone streams from the remote server rather than Others and that is working great.

I also tried to set one up for the Newsgroups but its an address and not just one static IP so it can obviously connect to a multitude of different servers, I read up about using the CIDR calculator but then I thought I can just use port 563 as that is the default one really for Newsgroups. I also read the section about not setting up rules for ports 80 and 443 which makes sense but I think that 563 is known for being used really for just Usenet unless I am mistaken which I very well could be.

Also when I turn off QOS and turn it back on my rules seem to apply straight away but I see that it takes 5 minutes for the rest to implement, is that right?

Lastly thanks again for this FreshJR its awesome!

 

[FreshJR]

but then I thought I can just use port 563 as that is the default one really for Newsgroups.

Yup, you can create a rule for 563.

I also think that usenet shows up as "Betternet" or something else unique in app analysis so you can also redirect that specific identification as an alternative to creating a port rule.

(Both methods will yield the same results)

Since you are the 10th+ user creating a custom rule for this, I may just include it in the script as a default rule.

Glad you got it figured out.

 

[preacher65]

Since you are the 10th+ user creating a custom rule for this, I may just include it in the script as a default rule.

Some of my usenet downloads show up in Bandwidth Monitor as Lets Encrypt and some show as NNTP.
I followed an earlier discussion in this thread between yourself and @JohnSmith and moved Lets Encrypt traffic to Downloads for just that one PC. However reading the above I've now just moved port 563 instead, regardless of app analysis, since it seems to fall across different classifications.

 

[Rob Q]

If you are getting poor bufferbloat grade, reduce WebUI speeds

I got my BufferBloat rating from a B to an A just by enabling this script!

 

[Carnagerover]

Yep. I redirected all https from net control -> web surfing with the default script config. I did not agree with the net control destination either.

Since it is one Usenet server, you can look into creating a custom rule to override the Usenet https identification into downloads if you wish.
(you can detect the Usenet server traffic by filtering by its IP)

I will do that tonight, however I noticed today that when watching on plex that I lost connection to the server, I checked the log and found this;

Jan  8 14:09:33 rc_service: httpd 875:notify_rc restart_firewall
Jan  8 14:09:33 miniupnpd[14979]: shutting down MiniUPnPd
Jan  8 14:09:33 nat: apply nat rules (/tmp/nat_rules_ppp0_eth0)
Jan  8 14:09:33 custom_script: Running /jffs/scripts/firewall-start (args: ppp0)
Jan  8 14:09:33 miniupnpd[24288]: HTTP listening on port 39260
Jan  8 14:09:33 miniupnpd[24288]: Listening for NAT-PMP/PCP traffic on port 5351
Jan  8 14:09:33 adaptive QOS: Applying - Iptable Down Rules
Jan  8 14:09:33 adaptive QOS: Applying - Iptable Up   Rules (ppp0)
Jan  8 14:09:33 adaptive QOS: TC Modification Delayed Start (5min)
Jan  8 14:14:33 adaptive QOS: No modifications necessary
Jan  8 14:24:02 rc_service: httpd 875:notify_rc restart_firewall
Jan  8 14:24:02 miniupnpd[24288]: shutting down MiniUPnPd
Jan  8 14:24:02 nat: apply nat rules (/tmp/nat_rules_ppp0_eth0)
Jan  8 14:24:02 custom_script: Running /jffs/scripts/firewall-start (args: ppp0)
Jan  8 14:24:02 miniupnpd[25674]: HTTP listening on port 50221
Jan  8 14:24:02 miniupnpd[25674]: Listening for NAT-PMP/PCP traffic on port 5351
Jan  8 14:24:02 adaptive QOS: Applying - Iptable Down Rules
Jan  8 14:24:02 adaptive QOS: Applying - Iptable Up   Rules (ppp0)
Jan  8 14:24:02 adaptive QOS: TC Modification Delayed Start (5min)
Jan  8 14:29:02 adaptive QOS: No modifications necessary
Jan  8 14:42:22 rc_service: httpd 875:notify_rc restart_firewall
Jan  8 14:42:22 miniupnpd[25674]: shutting down MiniUPnPd
Jan  8 14:42:22 nat: apply nat rules (/tmp/nat_rules_ppp0_eth0)
Jan  8 14:42:22 custom_script: Running /jffs/scripts/firewall-start (args: ppp0)
Jan  8 14:42:22 miniupnpd[26290]: HTTP listening on port 37619
Jan  8 14:42:22 miniupnpd[26290]: Listening for NAT-PMP/PCP traffic on port 5351
Jan  8 14:42:22 adaptive QOS: Applying - Iptable Down Rules
Jan  8 14:42:22 adaptive QOS: Applying - Iptable Up   Rules (ppp0)
Jan  8 14:42:22 adaptive QOS: TC Modification Delayed Start (5min)
Jan  8 14:42:59 rc_service: httpd 875:notify_rc restart_firewall
Jan  8 14:42:59 miniupnpd[26290]: shutting down MiniUPnPd
Jan  8 14:42:59 nat: apply nat rules (/tmp/nat_rules_ppp0_eth0)
Jan  8 14:42:59 custom_script: Running /jffs/scripts/firewall-start (args: ppp0)
Jan  8 14:42:59 miniupnpd[26385]: HTTP listening on port 44047
Jan  8 14:42:59 miniupnpd[26385]: Listening for NAT-PMP/PCP traffic on port 5351
Jan  8 14:42:59 adaptive QOS: Delayed Start Canceled
Jan  8 14:42:59 adaptive QOS: Applying - Iptable Down Rules
Jan  8 14:42:59 adaptive QOS: Applying - Iptable Up   Rules (ppp0)
Jan  8 14:42:59 adaptive QOS: TC Modification Delayed Start (5min)
Jan  8 14:47:59 adaptive QOS: No modifications necessary
Jan  8 15:11:58 dnsmasq-

I thought maybe my internet dropped but I don’t think it did, is there a way to check WAN uptime? Is the above normal? I have noticed recently that streams to BBC iPlayer just stop but I’m not sure why it happens.

 

[FreshJR]

The script messages occur because it keeps being triggered/executed by “firewall-start”

Jan  8 14:09:33 rc_service: httpd 875:notify_rc restart_firewall
Jan  8 14:09:33 miniupnpd[14979]: shutting down MiniUPnPd
Jan  8 14:09:33 nat: apply nat rules (/tmp/nat_rules_ppp0_eth0)
Jan  8 14:09:33 custom_script: Running /jffs/scripts/firewall-start (args: ppp0)

See the 1rst and 4th lines.

I can’t tell the reason why you are triggering “restart_firewall”

Could be

Connection drop or even making changes in webui. The script shouldn’t drop your internet.

Some of my usenet downloads show up in Bandwidth Monitor as Lets Encrypt and some show as NNTP.
I followed an earlier discussion in this thread between yourself and @JohnSmith and moved Lets Encrypt traffic to Downloads for just that one PC. However reading the above I've now just moved port 563 instead, regardless of app analysis, since it seems to fall across different classifications.

Thanks for the input, I wouldn’t have known since I don’t use Usenet.

Are usenet servers hosted on port 563 or is their traffic destined to a clients port 563?

 

[skeal]

@FreshJR I found that after updating to the third refresh of 384.9 alpha 1 and the new signature file from Trend, that I had to drop my download bandwidth 3mbs or my tests on dslreports sucked for quality.

 

[cubano_14]

me being an average merlin user. i always like to read the patch note changes but i don't even understand github and how to read things in there. I have never run QoS or anything extra on the router or any scripts. My gaming on ps4 and all that has been good and i don't believe i've needed the QoS to improve it. My question is will the script improve things in general and not just relating to QoS. Without using it, i get an A or A+ across the board on the speedtest on dslreports. would it make internet any speedier in any way? Would it make the wireless any better on the 5ghz band? I had read the first page and i tried my best to understand but i had a hard time. If it can make things even better for me where now i think things are pretty solid (i'm using the 86U) then i would love to give it a try. I've been using merlin since my first asus i had the 66u. Currently i have the 2nd alpha version of 384.9.

thanks

 

[JohnSmith]

Thanks for the input, I wouldn’t have known since I don’t use Usenet.

Are Usenet servers hosted on port 563 or is their traffic destined to a clients port 563?

The client uses the following for most Usenet servers:

Usenet unencrypted is usually port 119
Usenet encrypted SSL is usually port 563

Some Usenet servers I have seen use the following ports for unencrypted:
20, 23, 25, 119, 3128, 7000, 8000 and 9000

Some Usenet servers use the following ports for encrypted SSL:
563, 443 or 8080

The code that worked to push that single computer download, into the Downloads category for me was the following:

iptables -D POSTROUTING -t mangle -o br0 -d <IP Address>/32 -m mark --mark 0x801400c2/0xc03fffff -p tcp --sport 563 -j MARK --set-mark ${Downloads_mark_down} &> /dev/null
iptables -A POSTROUTING -t mangle -o br0 -d <IP Address>/32 -m mark --mark 0x801400c2/0xc03fffff -p tcp --sport 563 -j MARK --set-mark ${Downloads_mark_down}

iptables -D POSTROUTING -t mangle -o $wan -s <IP Address>/32 -m mark --mark 0x401400c2/0xc03fffff -p tcp --dport 563 -j MARK --set-mark ${Downloads_mark_up} &> /dev/null
iptables -A POSTROUTING -t mangle -o $wan -s <IP Address>/32 -m mark --mark 0x401400c2/0xc03fffff -p tcp --dport 563 -j MARK --set-mark ${Downloads_mark_up}

Just make sure to replace the <IP Address> with the Static or DHCP Reserved IP address X.X.X.X of the computer you are trying to have its Usenet traffic go into the Downloads category.

 

[preacher65]

Are usenet servers hosted on port 563 or is their traffic destined to a clients port 563?

Usenet unencrypted is usually port 119
Usenet encrypted SSL is usually port 563

It's the port the server listens on. The above two - 119 and 563 - are the only IANA assigned ports for nntp and nntp over tls/ssl, so in theory at least 119 and 563 are supposed to be reserved for usenet. All of the servers I use offer the standard ports, plus some others as @JohnSmith has indicated, I assume these are usually offered as ways to get around firewall restrictions.

As mentioned before I've stopped matching any marks in my rules and just match on port, since about 10% of my traffic is correctly showing up as NNTP, not Lets Encrypt, so it has a 0x80050004/0xc03fffff mark. Bizarrely the appdb puts NNTP in the "VoIP" category which of course isn't where I want it.

 

[FreshJR]

@skeal we’lll cross that bridge when we get there

@cubano_14 sounds like your modem has sqm/qos built in. You don’t need a second form implemented. It would probably make the results worse since you are already getting A’s

@JohnSmith I’ll just include all traffic from server side 563 for now

@preacher65 i agree. I will only match the port and not the port+mark to get more traffic included.

 

[Kingp1n]

I got my BufferBloat rating from a B to an A just by enabling this script!

By how much and are u guys testing while wire connected or mobile wireless? It's wired testing the way to test using DLS reports speedtest? Just curious....I tried using wireless with mobile phone and always get B rating bufferbloat. Maybe I should test with wired laptop.

 

[FreshJR]

By how much and are u guys testing while wire connected or mobile wireless? It's wired testing the way to test using DLS reports speedtest? Just curious....I tried using wireless with mobile phone and always get B rating bufferbloat. Maybe I should test with wired laptop.

If your network speeds exceed the wireless throughout you will get bloat on that specific wireless link.

The wired connections and other unsaturated wireless link (2.4 or 5g) will be unbloated.

(The headaches of fast connections)