[Release] FreshJR Adaptive QOS (Improvements / Custom Rules / and Inner workings)

[ColinTaylor]

Steams guide is out dated

Why is it outdated? Just because they list all the ports that may be used doesn't mean all of them always will be used.

they should use a port other than 80 for downloading data,

The reason organisations use port 80 for downloads is because it works with just about every device on the internet (because it's just like a web browser). It even works through restrictive firewalls and proxy servers. If they used a non-standard port they'd have endless support calls from people that can't download their stuff. That's why there are millions of sites that provide downloads via HTTP rather than something like FTP or SCP.

either that or asus need to update qos database.

This is indeed a problem. If you're downloading stuff using HTTP it's very difficult to distinguish it from other web traffic. That's why I have my traditional QoS rules setup so that HTTP transfers greater than 20MB are classified as "file transfer" and given a low priority.

 

[ColinTaylor]

I see so that's how multiple devices can use port 80, hmmm interesting. Thanks for that.

Or more to the point, it is how a single client can open multiple connections to multiple web servers at the same time. It is the combination of the client's IP address and port number that uniquely identifies a single connection. The client can't change their IP address, but by changing their outgoing port number they can have multiple connections, even to the same server.

This is one of the reasons myself and Merlin get so frustrated with gamers that read but don't understand documents that talk about "opening" port 80 or 443 or 53, etc. They try to forward these ports into their network when in fact they need to do nothing at all. The document is talking about the port on the remote server, not their router.

 

[Vexira]

Or more to the point, it is how a single client can open multiple connections to multiple web servers at the same time. It is the combination of the client's IP address and port number that uniquely identifies a single connection. The client can't change their IP address, but by changing their outgoing port number they can have multiple connections, even to the same server.

This is one of the reasons myself and Merlin get so frustrated with gamers that read but don't understand documents that talk about "opening" port 80 or 443 or 53, etc. They try to forward these ports into their network when in fact they need to do nothing at all. The document is talking about the port on the remote server, not their router.

i always assumed 443 and 80 were automatically opened, when i used to help people port forward cod games id only open 3074 and upward on a per game basis, ive seen isp routers and some old units which thier upnp must not have worked. But manual forwarding did.

 

[ColinTaylor]

i always assumed 443 and 80 were automatically opened,

No, they are not "automatically" opened. They are destination ports for outgoing connections. By default all outgoing ports are open all the time. Don't confuse outgoing connections with incoming connections. It is only unsolicited incoming connections that may sometimes need ports to be forwarded, and even then usually they don't.

Anyway, this has nothing to do with QoS so I'll leave the conversation here.

 

[Vexira]

No, they are not "automatically" opened. They are destination ports for outgoing connections. By default all outgoing ports are open all the time. Don't confuse outgoing connections with incoming connections. It is only unsolicited incoming connections may sometimes need ports to be forwarded, and even then usually they don't.

Anyway, this has nothing to do with QoS so I'll leave the conversation here.

but in regards to qos and steam im confused some one set up rules if you check what i linked, in the 27000 port range as steam dowonloads, its some what confusing the need to clear up wich ports are for in home streaming, i have a steam link which uses in home streaming, not sure weather or not to set it as video streaming or gaming, i suppose it is video streaming since its streaming whats on the desktop and thats encoded on the fly but gaming because it takes periferal input and you can controll the pc with it. The in home streaming ports are the same as ones used for gaming so i guess id just protise the port it keeps complaining about, 27031. But not sure the right category.

 

[Gitsum]

The QOS engine/signatures should be able to tell the difference between content downloads from Steam or game play packets. All content downloads come from *.steamcontent.com on port 80, right? That should be easy to add to the rules.
It's what I did with my Untangle box because the ASUS router would put game downloads in the GAMING category and kill everything else.

 

[FreshJR]

@Arsenal Let's review some things over PM first.

The QOS engine/signatures should be able to tell the difference between content downloads from Steam or game play packets. All content downloads come from *.steamcontent.com on port 80, right? That should be easy to add to the rules.
It's what I did with my Untangle box because the ASUS router would put game downloads in the GAMING category and kill everything else.

I wish it was that simple.

Both iptables and tc works on sorting packets by their packet headers. The packet header only contains the server ip that the dns lookup initially resolved and does not retain the user input domain name anywhere. The user input domain name was only used to perform that initial DNS lookup.

It would be great if I could intercept all dns lookups and then make a temporary iptables or tc rule for ip returned from that DNS result, if that initial DNS query matched a rule like yours (half of this is possible in ip tables to my knowledge, just not a new rule based on DNS result)

Besides that, another option would be running a whois on all *.steamcontent.com domains and compiling a list of all used steam server IP's. This method is messy since steam's public IP ranges could change overtime. Also it probably wouldn't work as the content is most likely coming from a CDN mirror instead steam server.

I'm all ears if there is already a way to temporary mark all packets matching results from a returned IP from a DNS query, then deleting that rule upon connection expiration.

Wonder how your other box has it implemented for QOS. I will look into that brand if I need a new router.

Just thinking outloud, the URL keyword blacklist filter does a iptables string match in a chain that only contains DNS queries. It then drops any dns query that matches that blacklisted string match.

If I were to apply a connection mark and accept for the upon a DNS string match instead of a drop, would the connmark include subsequent traffic from the IP from that DNS request? My guess is no.

Not sure if client <-> dns server then client <-> requested server treated as one connection or two different connections in the connection tracker if I were to perform and filter on a connmark. I should really read more about ip tables or do some experimenting!!

Logically it sounds like two connections in connection tracker so a conmark on a matvhing DNS query won't help exhibit the behavior I want.

Any other ideas?

Thanks @FreshJR for the great script and your helpful support. I would like to extend my thanks to @Vexira and @Jack Yaz for their amazing work with me to help get the script working.

I have a question regarding Mssfix on Astrill Applet client, should I enable it as (1400 the default)" with the script and QoS working, or should I disable it with value (1500)?

Mssfix value has nothing to do with QOS. Use a value that works for your clients. I am not knowledgeable in that area.

 

[Vexira]

The QOS engine/signatures should be able to tell the difference between content downloads from Steam or game play packets. All content downloads come from *.steamcontent.com on port 80, right? That should be easy to add to the rules.
It's what I did with my Untangle box because the ASUS router would put game downloads in the GAMING category and kill everything else.

its akami cdn they use I think, asus needs to update it to pick up xbox and ps4 downloads as well, cousin had to keep net work testing to get the ps4 to consistently download at full rate.
strangely ive been having issues getting YouTube on my tablet to default to highest quality, on my Samsung galaxy tab s2, minor annoyance.

 

[posthumous]

Might anybody be so kind as to help me with creating some rules please? My knowledge and understanding of networking and scripting is minimal. Only just a week ago I acquired an AC68U with f/ware 380.68

The issue I'm seeing pre and post installation of FreshJR's script is most of streaming traffic going through the router is incorrectly identified.

• Youtube traffic (both from browser and Android app) is categorized as Web Surfing
• iflix (similar service as Netflix) is identified as Web Surfing
• Streaming off Spotify as Web Surfing
• Stream traffic off Google Play Music does not seem to be categorized in any of the 8 classes in the router's QoS Statistic page

The only stream that gets classified correctly is Netflix.

 

[Jack Yaz]

Might anybody be so kind as to help me with creating some rules please? My knowledge and understanding of networking and scripting is minimal. Only just a week ago I acquired an AC68U with f/ware 380.68

The issue I'm seeing pre and post installation of FreshJR's script is most of streaming traffic going through the router is incorrectly identified.

• Youtube traffic (both from browser and Android app) is categorized as Web Surfing
• iflix (similar service as Netflix) is identified as Web Surfing
• Streaming off Spotify as Web Surfing
• Stream traffic off Google Play Music does not seem to be categorized in any of the 8 classes in the router's QoS Statistic page

The only stream that gets classified correctly is Netflix.

A lot of traffic is shifting to being served over HTTP/HTTPS these days, so its getting harder for QoS to be able to classify the traffic that is coming through

 

[Csection]

I have a question about the 1_9 ver. of this script.

I am currently using 380.66_4.
Can the 1_9 script be used with this version of firmware or do I need to wait till I upgrade the firmware?

I d/led the 1_9 script, but I set it to not run at present.
I'm planning on upgrading to 380.68, but I was going to wait until it was not beta anymore.

 

[Vexira]

I have a question about the 1_9 ver. of this script.

I am currently using 380.66_4.
Can the 1_9 script be used with this version of firmware or do I need to wait till I upgrade the firmware?

I d/led the 1_9 script, but I set it to not run at present.
I'm planning on upgrading to 380.68, but I was going to wait until it was not beta anymore.

it will work and 380.68 has been out of beta for a while now

 

[FreshJR]

The newest script is backwards compatible with the current all previous FW versions.
Use it with whatever firmware you want.
It survives firmware updates aswell.

The old v1.4 script MAY have issues with firmware 380.67 or higher.

The beta in comments just means it was tested on the beta merlin firmware at the time, since the stable version of merlin has not been released yet.
The script itself was NOT in beta.

 

[Csection]

The newest script is backwards compatible with the current all previous FW versions.
Use it with whatever firmware you want.
It survives firmware updates aswell.

The old v1.4 script MAY have issues with firmware 380.67 or higher.

The beta in comments just means it was tested on the beta merlin firmware at the time, since the stable version of merlin has not been released yet.
The script itself was NOT in beta.

Thank you for those clarifications!
I will enable your script as soon as I read the forum posts and understand how to set it up correctly. I don't want to make mistakes others have already gone through.

 

[jpclarke]

Just an update on my setup. I am running this script and it is making a huge difference to my internet usage!

I also combined into the mix the up and down bandwidth figures being pushed by a raspberry pi speedtest once a day.

Basically a script sets the required qos figures to unlimited and the rest to minimal bandwidth which ensures the speedtest gets everything it needs. This same script then remotely calls the pi script via ssh key remote logon to populate a text file for up and down speed. Lastly these figures are committed to nvram and then FreshJR script runs setting the new qos figures.

 

[Vexira]

Just an update on my setup. I am running this script and it is making a huge difference to my internet usage!

I also combined into the mix the up and down bandwidth figures being pushed by a raspberry pi speedtest once a day.

Basically a script sets the required qos figures to unlimited and the rest to minimal bandwidth which ensures the speedtest gets everything it needs. This same script then remotely calls the pi script via ssh key remote logon to populate a text file for up and down speed. Lastly these figures are committed to nvram and then FreshJR script runs setting the new qos figures.

Would be interesting if it was done directly by the router, when the network is idle, if a script could be configured that way.

 

[jpclarke]

Would work if you could work out the speedtest installation on the router

 

[Vexira]

ne

Would work if you could work out the speedtest installation on the router

netgear has it built into qos and ive seen a script for it i think, but adaptive qos is closed source so I'm not sure howto do it.

 

[MarCoMLXXV]

If only upload and download speeds matter (latency is not reliable), check out speedtest-cli. Maybe, with some modification, it could suit your purposes. It's a python script that performs a speed test against the closest speedtest.net server based on ping results. You need entware installed, to install python, create a (persistent) dictory for the script, cd into that directory and then enter:

wget -O speedtest-cli https://raw.githubusercontent.com/sivel/speedtest-cli/master/speedtest.py
chmod +x speedtest-cli

See the developer's page linked above for more info, including options to export the results.

 

[jpclarke]

That is the script I run on the raspberry pi and write the results out to a text file for the router to ingest.