FlexQoS FlexQoS 1.2.5 - Flexible QoS Enhancement Script for Adaptive QoS

[maghuro]

Here it goes.
Same IP, same exit port, different internal port.

Different classifications - HTTP and Speedtest.net.

Signatures?

 

[Kingp1n]

@dave14305

Just wanted to say, THANK YOU for all you do...the latest development version is working flawlessly even with my sucky comcast speeds (paying for 50/5 mbps) & all while using a VPN.

It's funny/crazy that I tested comcast's 400/10 mbps (also tested 200/5, 100/5) speeds and realized all you need is the proper router & script to route internet traffic accordingly. I didn't notice any difference and finally sticked with 50/5 and everything (I mean everthing) runs smoothly to include some 1080p/4K streaming content, a couple wired & wireless PS4/Xbox gaming devices playing multi-player online games with no lag.

I appreciate all you've done with FlexQoS.

 

[albinoman887]

is the bandwith not being messured when using a vpn still a thing ?

 

[BikeHelmet]

I think I stumbled across a bug. I rearranged my priority levels to better fit my network:

However, I then discovered that Learn-From-Home (Green) was using all the bandwidth. I dug into it, and... Disney+, YouTube, Netflix, FaceBook (videos on wall) were all in the Green category - Learn-From-Home - which by default is Video and Audio Streaming. I manually created rules to shuffle them into the proper purple category, which has vastly improved the internet experience, but before that it got bogged down and choked out websites from loading.

I use the Streaming category for latency non-sensitive streaming, and Work-From-Home for latency-sensitive streaming. YouTube and Netflix have 60+ second buffers, so a little bit of ping doesn't matter. Zoom calls though, or Discord, or VOIP - very important to be in the right category.

Just now I spotted regular websites in there too...

I am wondering if I should just reset FlexQOS and reconfigure, leaving the order of everything default? I assumed that if I shuffled priorities of Streaming lower, that the categorized stuff would go lower, not drop into the category that takes its place. It appears I'll have to rethink my %'s, as I had streaming stuff towards the bottom with high minimum bandwidth guarantees, and other stuff towards the top with lower guarantees. But if nothing can be moved without adding to the limited 32 rule AppDB area, my current arrangement isn't going to do the trick.

Any thoughts on how I should proceed?

Cheers,

 

[dave14305]

Any thoughts on how I should proceed?

Don’t think of Learn-From-Home as a useful category. Think of it like the human appendix — doesn’t serve a useful purpose anymore, but can sometimes lead to trouble, and should be removed.

Learn-From-Home consists of all Streaming and some Web Surfing. So your experience is 100% expected when Learn-From-Home is higher priority than Streaming and Web Surfing.

I have Learn-From-Home at the bottom and I don’t think about it at all.

 

[BikeHelmet]

Alright, so that's the expected behaviour - I'm curious though, what exactly ends up in Video and Audio Streaming if all the common video and audio streaming is in Learn From Home? Zoom and VOIP are in Work From Home... Streaming does very little? This is just a quirk of how QOS/TrendMicro do it, right? Got to learn their categorisation quirks and roll with it? It almost seems that the way they use categories, we could get away with only having ~5 buckets instead of 8.

I will move Learn From Home into the same area as File Transfers, now that I better understand what is in it. (All the high bandwidth stuff!)

I noticed you have Web Surfing under Others - is there a reason for that? What are your minimums/maximums %'s for each category? Back when I did Tomato QOS, I used to have Web Surfing (under 1024KB for web ports) very high up, then after it exceeded that it dropped into the category right above video streams. (YouTube, Netflix, etc.) That way downloads quickly got deprioritized without interfering with maximum website responsiveness. This was a good compromise considering that many other things use those ports - Steam game downloads, some XBox stuff, and even some game traffic itself. (Talk about annoying.) As such, it can't be deprioritized to the point that it's in the trash heap, or you end up with problems.

With that in mind, I'd like to understand why you have Web Surfing where it is, with Others above it.

Cheers,

 

[dave14305]

what exactly ends up in Video and Audio Streaming if all the common video and audio streaming is in Learn From Home?

Nothing will, unless you create iptables rules to force traffic to Streaming. All the traffic will end up in the higher priority bucket and nothing will reach Streaming.

This is just a quirk of how QOS/TrendMicro do it, right? Got to learn their categorisation quirks and roll with it? It almost seems that the way they use categories, we could get away with only having ~5 buckets instead of 8.

Yes, it’s been a head scratcher ever since they announced the new categories. Essentially, Learn-From-Home = Streaming + ( Web Surfing - Social Networks ). So all that would end up in Web Surfing would be social media traffic.

I only need 6 of 8 categories for my usage patterns.

I noticed you have Web Surfing under Others - is there a reason for that?

Some of my work VPN traffic ends up Untracked, so I want it higher than general web surfing, but lower than traffic like Teams or streaming. I take the approach that if it’s not general web surfing or file transfers, it gets higher priority. Others is not a high-volume category for me, so it’s not going to impact the lower priority categories too much.

 

[BikeHelmet]

Well, I am officially confused. I had created some rules sending Disney+, YouTube, YouTube Kids, Netflix, Snapchat and a few others into Streaming. I deleted them afterwards and moved Learn-From-Home to the bottom... only kept a couple rules like Snapchat and TikTok. Yet... they're still classifying as Streaming rather than Learn-From-Home, now? I feel like I'm missing some nuance or piece of info that will help me understand this stuff? The rules are gone - why did they stay in the specified categories?

 

[dave14305]

Well, I am officially confused. I had created some rules sending Disney+, YouTube, YouTube Kids, Netflix, Snapchat and a few others into Streaming. I deleted them afterwards and moved Learn-From-Home to the bottom... only kept a couple rules like Snapchat and TikTok. Yet... they're still classifying as Streaming rather than Learn-From-Home, now? I feel like I'm missing some nuance or piece of info that will help me understand this stuff? The rules are gone - why did they stay in the specified categories?

Any traffic previously landing in Learn-From-Home (when it was higher priority) will now be “claimed” first by Web Surfing or Streaming since both are evaluated first in the priority rules. Learn-From-Home is inert once it is lower priority than those other 2 categories.

I am confused by your screenshot showing Snapchat in Streaming from an AppDB rule (denoted by the ~), but your AppDB screenshot shows no such rule.

And the 2 rules moving TikTok and Crunchyroll would now be superfluous since they are in Streaming by default.

 

[BikeHelmet]

Ahh, I see! I think I misunderstood how classification worked. I was imagining a list of rules (some customizable, some proprietary and unknown) that get evaluated and push connections into each category/bucket. But instead it seems, connections could go in any of the buckets, and rules for each one get evaluated in order of the categories. So some could be Web Browsing, or could be Streaming, or could be Learn-From-Home, changing the order of them around may result in it changing where it appears? This is rather different from my Tomato days, which had one sequentially evaluated list. You stuck your heavy hitters like torrent traffic at the top of the list, since you'd be classifying hundreds of those per second, and then worked downward from there.

Oh, that might be a copy paste error. I had about 20 screenshots in Paint.net as I was piecing together what was happening. That one may have been before I scrapped Snapchat, as I was removing them one by one and retesting where things landed. If I had grabbed the screenshot to the right of it, probably no squiggly.

And the 2 rules moving TikTok and Crunchyroll would now be superfluous since they are in Streaming by default.

Cheers. This interaction is helping me wrap my head around the differences. I didn't imagine that re-ordering categories would change where connections end up.

 

[Vexira]

I'm wondering if there is a way to get adaptive to see rumble and bitchute as video streaming, Im thinking rules wise.

 

[BikeHelmet]

This might be answered elsewhere, but can you tell me how UDP traffic is handled in FlexQOS? Back in the Tomato QOS days, you had the option to exclude incoming UDP from traffic shaping. That was great since many UDP protocols did not have a retransmit mechanism like say, uTP does - so for example, games might desync and VOIP might crackle if packets were dropped - but then you needed to set your maximums low enough to have some spare room for the incoming undroppable UDP data, or you'd end up with bufferbloat at your ISP. I believe this was later improved to just throttle TCP connections more and keep UDP+TCP within each limit?

I noticed a bunch of UDP traffic getting classified automatically, so I assume that AppDB magic/identification is at work, and anything classified is accounted for? Does FlexQOS do anything special to decide what to drop and what not to? I assume, drop all that junk at the bottom first... Learn-From-Home, File Transfers, Streaming, etc.; but what would happen if it made its way to higher up UDP protocols that are packet loss intolerant? Does it chop away at the speed available to TCP connections first and foremost?

This question really only applies when you're running redline (fully maxed out internet connections) for long periods of time, as then you're likely to hit edge cases throughout the day.

Cheers,

 

[dave14305]

But instead it seems, connections could go in any of the buckets, and rules for each one get evaluated in order of the categories. So some could be Web Browsing, or could be Streaming, or could be Learn-From-Home, changing the order of them around may result in it changing where it appears?

In general, every identifiable application should only end up in one category. But logic was defied when Learn-From-Home was introduced last year because they duplicated existing categories that were perfectly fine already. So you won’t see this behavior with any other categories than these already mentioned.

can you tell me how UDP traffic is handled in FlexQOS? Back in the Tomato QOS days, you had the option to exclude incoming UDP from traffic shaping.

Adaptive QoS doesn’t seem to differentiate at all. Therefore, neither does FlexQoS.

I assume that AppDB magic/identification is at work, and anything classified is accounted for?

Also anything unclassified is still shaped and scheduled.

Does FlexQOS do anything important to decide what to drop and what not to? I assume, drop all that junk at the bottom first... Learn-From-Home, File Transfers, Streaming, etc.; but what would happen if it made its way to higher up UDP protocols that are packet loss intolerant? Does it chop away at the speed available to TCP connections first and foremost?

Dropping is a passive function in the sense that we don’t do anything to explicitly drop packets. If a flow queue builds up within a fq_codel qdisc and exceeds the target time (waiting in the queue) or the total queue length, then dropping starts to happen. And since traffic is sent in descending priority of categories (top to bottom) it could be that the lowest categories could have more delayed packets and therefore build up a longer queue, or a slower queue and drop more. But quantum prevents starvation of lower priority classes and competing flows within fq_codel.

So in general, in my learnings, drops should be rare under normal circumstances. I find that for people with low bandwidth (e.g. below 2.5Mbit) the default 5ms target used for fq_codel is insufficient and could lead to unnecessary dropping. But FlexQoS accounts for that. Other QoS options in the firmware do not (but CAKE does).

 

[dave14305]

I'm wondering if there is a way to get adaptive to see rumble and bitchute as video streaming, Im thinking rules wise.

I don’t use either, so what is unique about those connections?

I’ve thought about adding the same ipset functionality I just added to CakeQos-Merlin to FlexQoS, but I know it would not work nicely with the Tracked Connections table (no awareness in the WebUI which remote IPs are members of an ipset).

 

[BikeHelmet]

Ahh, I see! Very, very helpful.

I think I hit an edge case then. Back before Learn-From-Home was moved down below, when the connection was maxed and it was interfering with everything... just check out this packet loss from connmon. People got home at 6PM and went to sleep at 10:30.

It is much improved with the categorisation changes and small adjustments to minimums/maximums: (You'd barely notice the full connection use.)

 

[dave14305]

I'm wondering if there is a way to get adaptive to see rumble and bitchute as video streaming, Im thinking rules wise.

OK, so I did a little research by watching some Dinesh D'Souza videos on Rumble and you could do a manual classification outside of FlexQoS if you were so inclined. The important DNS name seems to be sp.rmbl.ws.

Create an ipset for Streaming:

ipset create streaming_4 hash:ip timeout 86400
ipset create streaming_6 hash:ip family inet6 timeout 86400

Add it to dnsmasq.conf.add:

max-cache-ttl=86400
ipset=/sp.rmbl.ws/bitchute.com/streaming_4,streaming_6

Create iptables rules to match on those ipsets:

iptables -t mangle -A POSTROUTING -o eth0 -m set --match-set streaming_4 dst -j MARK --set-mark 0x4004ffff/0xc03fffff
iptables -t mangle -A POSTROUTING -o br0 -m set --match-set streaming_4 src -j MARK --set-mark 0x8004ffff/0xc03fffff
ip6tables -t mangle -A POSTROUTING -o eth0 -m set --match-set streaming_6 dst -j MARK --set-mark 0x4004ffff/0xc03fffff
ip6tables -t mangle -A POSTROUTING -o br0 -m set --match-set streaming_6 src -j MARK --set-mark 0x8004ffff/0xc03fffff

 

[Vexira]

I don’t use either, so what is unique about those connections?

I’ve thought about adding the same ipset functionality I just added to CakeQos-Merlin to FlexQoS, but I know it would not work nicely with the Tracked Connections table (no awareness in the WebUI which remote IPs are members of an ipset).

the seem to be or it could be me with bad eyes not falling into the steaming category, im going to half guess that ill have to wait for ASUS to update classification, they are alternative platforms to YouTube, unfortunately some creators i watch seem to be moving to these platforms vs YouTube.

 

[BikeHelmet]

Is that a feature that could be added to FlexQOS? Tomato QOS allowed both remote IPs and domain names. I used that with DDNS to prioritize game traffic to friends, way way back. It re-registered at the same time as the Firewall scripts section, if I remember right. Occasionally after someone's IP changed, the service would have to be restarted for it to pick up a new IP.

In FlexQOS remote IPs can only be IPs. Just wondering if this is a feature that could be added as a third section, similar to the AppDB and IPTables sections. It's sometimes very handy to easily categorize by domain name. (Especially in combination with other details like port numbers.)

If too much work, no worries - just bringing up the idea.

 

[dave14305]

Is that a feature that could be added to FlexQOS? Tomato QOS allowed both remote IPs and domain names. I used that with DDNS to prioritize game traffic to friends, way way back. It re-registered at the same time as the Firewall scripts section, if I remember right. Occasionally after someone's IP changed, the service would have to be restarted for it to pick up a new IP.

In FlexQOS remote IPs can only be IPs. Just wondering if this is a feature that could be added as a third section, similar to the AppDB and IPTables sections. It's sometimes very handy to easily categorize by domain name. (Especially in combination with other details like port numbers.)

What variant of Tomato are people generally referring to when they say “Tomato” in 2021? I ask because I want to see what their QoS code looks like to support domains. Shibby, Fresh, Advanced, blah blah blah. Maybe they’re all the same. I haven’t used Tomato since the Linksys WRT54G.

As I mentioned a couple posts above, if I add any domain name/ipset based features to FlexQoS, traffic that gets modified by such an iptables rule won’t show up in the correct priority class in the list of Tracked Connections. That tends to freak people out when they think the traffic should show up in one class (e.g. Gaming) but it shows up in another (e.g. Web Surfing).

 

[BikeHelmet]

FreshTomato is the currently supported/developed project, but I had Tomato mostly during the Original, then Toastman and Shibby days.

To my knowledge they all had that capability going right back to builds a decade and a half back. I have some QOS notes in my Router stuff folder from 2009, which clearly list some rules by domain name. My earliest retained firmware is from 2007 (no source code, sorry) - and I have screenshots from 2012 that include rules that used domain names, not IPs. So as far as I know, it's a core capability that has been around right from the start - just like VLANs, which came in very early too.

Can't do much to solve people freaking out, except maybe put a disclaimer on that section saying that it won't show up in the right category, with a giant *BETA* tag.

Hehe, WRT54GL user here too. And then RT-N16, RT-AC3200, RT-AX56U - the last two are still in active use.