[Release] FreshJR Adaptive QOS (Improvements / Custom Rules / and Inner workings)

[tmcb82]

I use PlayStation Vue (psvue.com) for my television service but instead of classifying it as streaming it puts the traffic under "File Transfer" on the QOS Statistics page and "Web File Transfer" under app analysis. My hope was to make PlayStation Vue the highest priority to prevent any buffering while watching TV but because it's not classified correctly it messes up my QOS. Is there any way to fix this to make this traffic classified as streaming using this script?

I REALLY appreciate any help!

 

[FreshJR]

I use PlayStation Vue (psvue.com) for my television service but instead of classifying it as streaming it puts the traffic under "File Transfer" on the QOS Statistics page and "Web File Transfer" under app analysis. My hope was to make PlayStation Vue the highest priority to prevent any buffering while watching TV but because it's not classified correctly it messes up my QOS. Is there any way to fix this to make this traffic classified as streaming using this script?

I REALLY appreciate any help!

Currently, the only thing you can do is if you are streaming on dedicated devices such as FireStick / AppleTV /etc, you can stick ALL traffic from those devices into video streaming. This would be a super generic catch all rule.

In the (distant) future, something else that could be implemented would be to have dnsmasq create iptable entries upon dns lookup for DNS based qos rules. This feature has not been implemented and is not an immediate priority for future releases.

What device are you using to stream ps vue ?

If you are using a semi dedicated device, such as your playstation, you can make a similar rule that states

If PS4 traffic && is File Downloads --> Video Streaming

 

[tmcb82]

I use PlayStation Vue (psvue.com) for my television service but instead of classifying it as streaming it puts the traffic under "File Transfer" on the QOS Statistics page and "Web File Transfer" under app analysis. My hope was to make PlayStation Vue the highest priority to prevent any buffering while watching TV but because it's not classified correctly it messes up my QOS. Is there any way to fix this to make this traffic classified as streaming using this script?

I'm using a RT-AC68U on 384.

I REALLY appreciate any help!

Currently, the only thing you can do is if you are streaming on a dedicated devices such as FireStick / AppleTV /etc, you can stick ALL traffic from those devices into video streaming. This would be a super generic catch all rule.

In the future, something else that could be done would be to have dnsmasq create iptable entries for QOS traffic routing upon dns lookup. This feature has not been implemented and is not an immediate priority for future releases (yet).

What device are you using to stream ps vue ?

If you are using a semi dedicated device, such as your playstation, you can make a similar rule that states

If PS4 traffic && is File Downloads --> Video Streaming

That would actually work. We only use PS Vue on a Fire TV, Fire Stick and a Roku. As I sure you can guess everything we also do from the devices is also streaming. Would you mind giving me an idea how I might go about doing that?

 

[FreshJR]

That would actually work. We only use PS Vue on a Fire TV, Fire Stick and a Roku. As I sure you can guess everything we also do from the devices is also streaming. Would you mind giving me an idea how I might go about doing that?

The creation of custom rules was ALREADY explained in great detail within the first posts. Go back and re-read the first posts closely.

I will be happy to answer any SPECIFIC questions only regarding those instructions and any issues that arise from following the instructions.

 

[tmcb82]

Would this be correct?

INBOUND:
iptables -D POSTROUTING -t mangle -o br0 -d 192.168.1.XXX/32 -j MARK --set-mark ${Streaming_mark_down} &> /dev/null
iptables -A POSTROUTING -t mangle -o br0 -d 192.168.1.XXX/32 -j MARK --set-mark ${Streaming_mark_down}

OUTBOUND:
iptables -D POSTROUTING -t mangle -o br0 -d 192.168.1.XXX/32 -j MARK --set-mark ${Streaming_mark_up} &> /dev/null
iptables -A POSTROUTING -t mangle -o br0 -d 192.168.1.XXX/32 -j MARK --set-mark ${Streaming_mark_up}

Where XXX is the rest of the Fire TVs IP.

 

[FreshJR]

Would this be correct?

nope. Look closely at each section.

You can't just copy the download rules twice.

 

[tmcb82]

nope. Look closely at each section.

You can't just copy the download rules twice.

Hopefully now it's correct

INBOUND:
iptables -D POSTROUTING -t mangle -o br0 -d 192.168.1.XXX/32 -j MARK --set-mark ${Streaming_mark_down} &> /dev/null
iptables -A POSTROUTING -t mangle -o br0 -d 192.168.1.XXX/32 -j MARK --set-mark ${Streaming_mark_down}

OUTBOUND:
iptables -D POSTROUTING -t mangle -o $wan -s 192.168.1.XXX/32 -j MARK --set-mark ${Streaming_mark_up} &> /dev/null
iptables -A POSTROUTING -t mangle -o $wan -s 192.168.1.XXX/32 -j MARK --set-mark ${Streaming_mark_up}

 

[tmcb82]

I actually just tested it and it works! I'm soo thrilled. Thank you FreshJR!

Just one last question: how can I modify the rule to include multiple devices? Right now I have my 3 streaming devices on 192.168.1.102 103 and 104.

 

[FreshJR]

I actually just tested it and it works! I'm soo thrilled. Thank you FreshJR!

Just one last question: how can I modify the rule to include multiple devices? Right now I have my 3 streaming devices on 192.168.1.102 103 and 104.

See CIDR calculator. You really gotta read the first post ......

 

[tmcb82]

See CIDR calculator. You really gotta read the first post ......

Sorry about my thick head... I really appropriate your help and hard work again

 

[JohnSmith]

Yes I repurposed defaults for game download traffic some time ago (it was unused for a long time with my script). The "Game Transferring/Defaults" category also swaps priority with the users last defined category as shown on the page.

To be more clear. The "Game Transfering" category under the hood is actually still "Defaults" with a different label slapped on it within the statistics page. If you create iptables ${Defaults_Mark} rules, they WILL go into "Game Transferring"

I’ll look into creating filter functionality for that page.

Thanks for the info and looking into this filter functionality.

So I have my AC3100 at Merlin's Firmware V384.9_alpha3-g90685d684, and using your FreshJR QOS Stats page, and it appears to be working. Just wondering if you could change the color scheme and layout to match Merlin's Class info as yours and his looks like this:

FreshJR QOS Stats Page:

Merlin's QOS Stats Page:

Merlin's is easier to read, and the colors then stand out, as your Game Transferring and File Transferring are too similar in color.

Thanks for listening and appreciate all your efforts.

 

[FreshJR]

Merlin's is easier to read, and the colors then stand out, as your Game Transferring and File Transferring are too similar in color.

Thanks for listening and appreciate all your efforts.

"Game transferring" and "File transferring" are almost identical in the sense that they are both unimportant "Bulk Data Transfers", so I made them almost the same color. I think of them as being the SAME category.

The only reason I did NOT create a rule to put "Game Data" into "File Transfering" is that I wanted "Game Bulk Data" to get a priority for using excess bandwidth. (Also so that the lobby data erroneously caught within "GameDownloads" could be separate from the congested FileDownloads class).

A smaller reason for me changing the color scheme was since I always got mixed up which shade of "Blue" was what when looking at the pie chart as a quick glance.

Getting rid of one of the shades also had that bonus for me Maybe I was getting confused since I always reference a rainbow gradient in my head and I was subconsciously thinking of light blue as the priority level BEFORE dark blue instead of last priority. (Kinda stupid I know).

Upon release, I will show you which portion of code you can change easily to revert the color scheme back to normal.

EDIT: If its an aesthetics thing. Painting "Game Transferring" as blue will still make your pie chart have large grey portions. Anyway, I will show how to paint whatever category whatever color you want.

EDIT2: In the mean time you can paste any of these lines in chromes console (right click -> inspect) and change the colors temporarily to see which one you like the best. I am still undecided.

      //original reversed
      //             Net=Red   Cat1=Org   Cat2=Yel   Cat3=Grn   Cat4=DBlu   Cat5=Prp  Cat6=LBlu  Cat7=DGry
      var color = ["#B3645B", "#B98F53", "#C6B36A", "#849E75", "#2B6692", "#7C637A", "#4C8FC0", "#6C604F"];

      //original
      //             Net=Red   Cat1=Org   Cat2=Yel   Cat3=Grn   Cat4=DBlu   Cat5=Prp  Cat6=DGry   Cat7=LBlu
      var color = ["#B3645B", "#B98F53", "#C6B36A", "#849E75", "#2B6692", "#7C637A", "#6C604F", "#4C8FC0"];

      //blue swapped position + grey reversed
      //             Net=Red   Cat1=Org   Cat2=Yel   Cat3=Grn    Cat4=LBlu   Cat5=Prp   Cat6=Gry     Cat7=DBlu
      var color = ["#B3645B", "#B98F53", "#C6B36A", "#849E75", "#4C8FC0",  "#7C637A", "#2B6692",  "#6C604F"];
 

      //double blue
      //             Net=Red   Cat1=Org   Cat2=Yel   Cat3=Grn    Cat4=LBlu   Cat5=Prp   Cat6=DBlu  Cat7=DBlu
      var color = ["#B3645B", "#B98F53", "#C6B36A", "#849E75", "#4C8FC0",  "#7C637A", "#2B6692",  "#2B6692"];
 
      //double grey
      //             Net=Red   Cat1=Org   Cat2=Yel   Cat3=Grn   Cat4=DBlu   Cat5=Prp   Cat6=Gry  Cat7=DGry
      var color = ["#B3645B", "#B98F53", "#C6B36A", "#849E75", "#2B6692", "#7C637A", "#796B58", "#6C604F"];
 
      //roy-g-biv
     //             Net=Red   Cat1=Org   Cat2=Yel   Cat3=Grn    Cat4=LBlu   Cat5=DBlu   Cat6=Prp   Cat7=Gry
      var color = ["#B3645B", "#B98F53", "#C6B36A", "#849E75", "#4C8FC0",  "#2B6692", "#7C637A", "#796B58"];

I may now be partial to the (blue swapped position + grey reversed) layout since
1) I can simply assume ALL dark colors (Purple, Dark Blue, Grey) are bulk data
2) light blue isn't causing me that weird confusion upon quick glance!

 

[Vexira]

Currently, the only thing you can do is if you are streaming on dedicated devices such as FireStick / AppleTV /etc, you can stick ALL traffic from those devices into video streaming. This would be a super generic catch all rule.

In the (distant) future, something else that could be implemented would be to have dnsmasq create iptable entries upon dns lookup for DNS based qos rules. This feature has not been implemented and is not an immediate priority for future releases.

What device are you using to stream ps vue ?

If you are using a semi dedicated device, such as your playstation, you can make a similar rule that states

If PS4 traffic && is File Downloads --> Video Streaming

I wish asus would hire you to fix their qos, also im exited for when you decide to implement the dns based qos rules, again thanks for all the hard work.

 

[heysoundude]

@FreshJR I added your script to my new router yesterday, and HooBOY what a difference! Thank you!!
I can't seem to find a way to send you a financial token of appreciation to encourage your continued work on this...or did I miss it?

 

[lilstone87]

I wish asus would hire you to fix their qos, also im exited for when you decide to implement the dns based qos rules, again thanks for all the hard work.

I totally agree. Their QoS could be very good.. Instead they give minimal effort on their QoS. The DPI they're using is sh*t as well. What's the point of having signature updates to it, yet it improves hardly anything? For example.. You have a very popular game like Call of Duty, yet when actually playing it. Its gaming traffic gets tagged as general. Even if this year's Call of Duty changed ports, and such it uses. At this point, its traffic should be getting sorted correctly. As the game has been out for over 3 months now.

Even with FreshJR's hard work on improving the way QoS is running. When it comes to gaming traffic overall, with the DPI not doing its job correctly. It would require a lot of manual work to get everyone's games to be properly classified. That's on a per game basis as well. I know FreshJR has done things to improve how traffic is handled, and such. Still when it comes to gaming type traffic, the DPI system need's to work a hell of a lot better then it currently does. Having to give every device manual LAN IP's, and figuring out which port's are being used to pass gaming traffic, is a ton of work for a single person to do, and ultimately would require others help.

Still for a game like COD, it's beyond embarrassing to see the DPI system ASUS is using can't properly sort its traffic at this point. Because I bet money right now with Black Ops 4, the way it passes traffic, and the ports being used. Are the same as it has been for many years now.. Yet I load this game up on my PS4, jump in a multiplayer game, and guess what? Under my PS4 within the QoS section, it shows the data being passed while in a game, classified as general traffic. All around piss poor.

Lastly to be frank.. These last couple years, many of these router companies have pushed out routers promoting as a gaming router.. But guess what? When it comes to the QoS system they include which is supposed to help keep your gaming as the top priority, well more times then not.. your gaming traffic isn't even being properly classified. The QoS system classifying all your console traffic as gaming, is horrible. Especially when often times these consoles have games, and updates being downloaded while you're playing. So having all this traffic classified as gaming traffic, ultimately helps no one. With all this said.. DPI can be very good. We just need one of these companies to put effort into it. Have a team who actually works to get traffic classified properly. Because whoever is currently working the DPI that ASUS uses, doesn't care, and it shows with the way traffic is being classified.

 

[RMerlin]

DPI can be very good.

Complete DPI is also very CPU intensive. Trend Micro has to take shortcuts to ensure that you can run DPI on a 1.4 GHz CPU.

 

[FreshJR]

@lilstone87

On a positive note, since only your game traffic is “unidentified” then in essence its in its own “Gaming” QOS category under a different label.

Also you are one custom rule away to achieve the following:

If from gaming device && unidentified —> assume it’s gaming

Most of the important traffic in my household is caught correctly. The only important bit for most use cases is to ensure video streaming doesn’t choke everything else out. So as long as Netflix, YouTube, and the like are identified, I can’t complain.

 

[Sinner]

@lilstone87

On a positive note, since only your game traffic is “unidentified” then in essence its in its own “Gaming” QOS category under a different label.

Also you are one custom rule away to achieve the following:

If from gaming device && unidentified —> assume it’s gaming

Most of the important traffic in my household is caught correctly. The only important bit for most use cases is to ensure video streaming doesn’t choke everything else out. So as long as Netflix, YouTube, and the like are identified, I can’t complain.

I'd have to agree with you there. Streaming media, downloads and torrents are the real network hogs.. as long as everything else is above them in priority everything just works smoothly all the time. Any other tweaking for the most part only achieves minor gains. Im still using an old script back from 384.5 days and quite happy with it. I think its the one or two updates before you fixed the umm dpi search function and added a few feature I wasn't into. I believe my search can only be 1 word still or something if I recall. But its just rock solid for me otherwise.

 

[NGRhodes]

It is frustrating seeing my Xbox downloads being treated as game traffic and plex streaming being treated as downloads. I have put some specific rules in to fix, but does highlight the fact that failure mode of Adaptive QoS (by mis-categorisation) is often worse than QoS being turned off.

I do wish we could have Cake (I have another router running OpenWRT that I currently use for experimenting with) - out of the box scirpts avoid most of these side-effects of Adaptive QoS by allowing per-device fairness (simple gui config to turn on) and then doing fair queuing for the flows in each devices separately. By keeping it simple you loose the fine grained advantages of DPI classification, but you gain by not having to worry about mis-categorisation (or lesser extend no-categorisation).

I had a look at implementing per-device fairness and its very difficult to do alongside Adaptive QoS as requires a complete restructure of how the classes are implemented and becomes very fragile. It could be done with a completely custom script avoiding adaptive QoS, but would be hard to make generic and reusable (where FreshJR does a good job of hooking into Adaptive QoS).

 

[Sinner]

It is frustrating seeing my Xbox downloads being treated as game traffic and plex streaming being treated as downloads. I have put some specific rules in to fix, but does highlight the fact that failure mode of Adaptive QoS (by mis-categorisation) is often worse than QoS being turned off.

I do wish we could have Cake (I have another router running OpenWRT that I currently use for experimenting with) - out of the box scirpts avoid most of these side-effects of Adaptive QoS by allowing per-device fairness (simple gui config to turn on) and then doing fair queuing for the flows in each devices separately. By keeping it simple you loose the fine grained advantages of DPI classification, but you gain by not having to worry about mis-categorisation (or lesser extend no-categorisation).

I had a look at implementing per-device fairness and its very difficult to do alongside Adaptive QoS as requires a complete restructure of how the classes are implemented and becomes very fragile. It could be done with a completely custom script avoiding adaptive QoS, but would be hard to make generic and reusable (where FreshJR does a good job of hooking into Adaptive QoS).

im not really familiar with plex but if its external to your home network then downloads or video is a good place for it at a lower priority. That allows you to still browse a webpage or use skype teamspeak discord online games etc with little issue since they are very much ping sensitive. if plex is anything like watching Netflix it will use your entire bandwidth if it can at least during the initial buffering lagging out everything else until its caught up unless it at a lower priority.