FlexQoS FlexQoS 1.0 - Flexible QoS Enhancement Script for Adaptive QoS

[rlj2]

I just started using flex, and like it quite well. But I was messing with youtube on my desktop (i seldom use youtube). Walked away, came back later and it had used 23 gig but was marked in the "OTHERS" category?

 

[brummygit]

Thank you for overestimating my skills.


Right now, it’s going to use hard-coded Upload and Download bandwidth from the QoS settings while I see how hard it is to calculate per-class ceilings on the fly since it hasn’t been needed in the webui before.

I don’t think an additive bar will convey much meaning in the context of QoS, since it’s focused on limiting a particular class within its ceiling (no class will ever exceed its ceiling by definition). In your example, file transferring can exceed its rate (guaranteed bw) and borrow bandwidth up to its ceiling.

What I find so far is that because tc’s rate measurement is smoothed out, you may not feel like the meters reflect what you see in real-time. Even so, I find it more interesting than packets per second, which it replaces.

Is the develop branch in a state where I can sample the new layout in relative safety?

 

[dave14305]

Is the develop branch in a state where I can sample the new layout in relative safety?

Now it is.

 

[dave14305]

I just started using flex, and like it quite well. But I was messing with youtube on my desktop (i seldom use youtube). Walked away, came back later and it had used 23 gig but was marked in the "OTHERS" category?

Was YouTube using QUIC on port 443/udp?

 

[Morris]

Was YouTube using QUIC on port 443/udp?

Sounds like you are watching Youtube in Chrome Browser. You need to add two rules to the "iptables Rules" to move it to streaming:

1)
Protocol = TCP
Remote Port = 443
Mark = 13005E
Class = Sreaming

2)
Protocol = UDP
Remote Port = 443
Mark = 000000
Class = Sreaming

Enjoy,

Morris

 

[brummygit]

Now it is.

And it looks very nice too

My first reaction was to wonder whether you can repurpose the coloured Class bars to display the utilisation bar by overlaying it's description on top, and maybe put a matched coloured dot first to identify the colour coding. However I don't know how difficult that would be from a code point of view.

 

[Phil Outram]

My AppDB rules are working. Can't comment on yours without seeing some data.

1. flexqos debug
2. tc -s filter show dev br0 | grep -i x800800 -B1
3. Screenprint of the connection list showing Origin and Steam connections colored as File Downloads

Thanks Dave

I've managed to fix the issue sort of, now both Origin and Steam show as game downloads, however when you actually start a game download it's still under file downloads. I suspect therefore that the trend DB just doesn't have this stuff categorised correctly. Since this isn't an issue with your script I'll try and figure things out.

 

[dave14305]

Thanks Dave

I've managed to fix the issue sort of, now both Origin and Steam show as game downloads, however when you actually start a game download it's still under file downloads. I suspect therefore that the trend DB just doesn't have this stuff categorised correctly. Since this isn't an issue with your script I'll try and figure things out.

Do you see other Application names (besides Steam or Origin) colored as File Downloads while the download is going? I just want to make sure it’s not a bug.

 

[dave14305]

And it looks very nice too

My first reaction was to wonder whether you can repurpose the coloured Class bars to display the utilisation bar by overlaying it's description on top, and maybe put a matched coloured dot first to identify the colour coding. However I don't know how difficult that would be from a code point of view.

I’m less enamored with this experiment since there is such a lag in updates to the data and meters. My speed tests are done before the meters really get going. Won’t be in 1.0 for sure.

 

[pattiri]

I have a problem
First here is my setup;
- I have 100 mbps down 5 mbps upload speed.
- I'm using my AC88u as VPN client and my nvidia sheld tv is routed to VPN client.

When I set Bandwidth Setting as auto; my nvidia shield TV can get max of 50 mbits but VPN client traffic is detected wrong by router. here is the proof;

If I set Bandwidth Setting as Manual and set it as 100 down and 5 up, my shield TV can only get 5 mbps;

and here debug

Spoiler: FlexQoS Debug

FlexQoS v0.9.6 released 2020-07-28

Debug:

Log date: 2020-08-05 15:25:30+0300
Router Model: RT-AC88U
Firmware Ver: 384.18_0
tc WAN iface: eth0
Undf Prio: 2
Undf FlowID: 1:14
Classes Present: 8
Down Band: 102400
Up Band  : 5120
***********
Net Control = 1:10
Work-From-Home = 1:15
Gaming = 1:12
Others = 1:14
Web Surfing = 1:13
Streaming = 1:11
Downloads = 1:16
Defaults = 1:17
***********
Downrates -- 5120, 30720, 15360, 10240, 10240, 20480, 5120, 5120
Downceils -- 102400, 102400, 102400, 102400, 102400, 102400, 102400, 102400
Downbursts -- 6400b, 25Kb, 12796b, 6400b, 4797b, 3194b, 3198b, 3200b
DownCbursts -- 125Kb, 125Kb, 125Kb, 125Kb, 125Kb, 125Kb, 125Kb, 125Kb
DownQuantums -- default, 384000, default, default, default, 256000, default, default
***********
Uprates -- 256, 512, 768, 512, 1536, 1024, 256, 256
Upceils -- 5120, 5120, 5120, 5120, 5120, 5120, 5120, 5120
Upbursts -- 3200b, 3200b, 3199b, 3200b, 3199b, 3198b, 3198b, 3198b
UpCbursts -- 6400b, 6400b, 6400b, 6400b, 6400b, 6400b, 6400b, 6400b
UpQuantums -- default, default, default, default, default, default, default, default
iptables settings: <>>udp>>500,4500>>3<>>udp>16384:16415>>>3<>>tcp>>119,563>>5<>>tcp>>80,443>08****>7
iptables -D POSTROUTING -t mangle -o br0   -p udp  -m multiport  --sports 500,4500  -j MARK --set-mark 0x80060001 >/dev/null 2>&1
iptables -A POSTROUTING -t mangle -o br0   -p udp  -m multiport  --sports 500,4500  -j MARK --set-mark 0x80060001
iptables -D POSTROUTING -t mangle -o eth0   -p udp  -m multiport  --dports 500,4500  -j MARK --set-mark 0x40060001 >/dev/null 2>&1
iptables -A POSTROUTING -t mangle -o eth0   -p udp  -m multiport  --dports 500,4500  -j MARK --set-mark 0x40060001
iptables -D POSTROUTING -t mangle -o br0   -p udp  --dport 16384:16415   -j MARK --set-mark 0x80060001 >/dev/null 2>&1
iptables -A POSTROUTING -t mangle -o br0   -p udp  --dport 16384:16415   -j MARK --set-mark 0x80060001
iptables -D POSTROUTING -t mangle -o eth0   -p udp  --sport 16384:16415   -j MARK --set-mark 0x40060001 >/dev/null 2>&1
iptables -A POSTROUTING -t mangle -o eth0   -p udp  --sport 16384:16415   -j MARK --set-mark 0x40060001
iptables -D POSTROUTING -t mangle -o br0   -p tcp  -m multiport  --sports 119,563  -j MARK --set-mark 0x80030001 >/dev/null 2>&1
iptables -A POSTROUTING -t mangle -o br0   -p tcp  -m multiport  --sports 119,563  -j MARK --set-mark 0x80030001
iptables -D POSTROUTING -t mangle -o eth0   -p tcp  -m multiport  --dports 119,563  -j MARK --set-mark 0x40030001 >/dev/null 2>&1
iptables -A POSTROUTING -t mangle -o eth0   -p tcp  -m multiport  --dports 119,563  -j MARK --set-mark 0x40030001
iptables -D POSTROUTING -t mangle -o br0   -p tcp  -m multiport  --sports 80,443 -m mark --mark 0x80080000/0xc03f0000 -j MARK --set-mark 0x803f0001 >/dev/null 2>&1
iptables -A POSTROUTING -t mangle -o br0   -p tcp  -m multiport  --sports 80,443 -m mark --mark 0x80080000/0xc03f0000 -j MARK --set-mark 0x803f0001
iptables -D POSTROUTING -t mangle -o eth0   -p tcp  -m multiport  --dports 80,443 -m mark --mark 0x40080000/0xc03f0000 -j MARK --set-mark 0x403f0001 >/dev/null 2>&1
iptables -A POSTROUTING -t mangle -o eth0   -p tcp  -m multiport  --dports 80,443 -m mark --mark 0x40080000/0xc03f0000 -j MARK --set-mark 0x403f0001
appdb rules: <000000>6<00006B>6<0D0007>5<0D0086>5<0D00A0>5<12003F>4<13****>4<14****>4<1A****>5
realtc filter change dev br0 prio 2 protocol all handle 827::800 u32 flowid 1:14
realtc filter change dev eth0 prio 2 protocol all handle 827::800 u32 flowid 1:14
realtc filter add dev br0 protocol all prio 2 u32 match mark 0x8000006B 0xc03fffff flowid 1:14
realtc filter add dev eth0 protocol all prio 2 u32 match mark 0x4000006B 0xc03fffff flowid 1:14
realtc filter add dev br0 protocol all prio 15 u32 match mark 0x800D0007 0xc03fffff flowid 1:16
realtc filter add dev eth0 protocol all prio 15 u32 match mark 0x400D0007 0xc03fffff flowid 1:16
realtc filter add dev br0 protocol all prio 15 u32 match mark 0x800D0086 0xc03fffff flowid 1:16
realtc filter add dev eth0 protocol all prio 15 u32 match mark 0x400D0086 0xc03fffff flowid 1:16
realtc filter add dev br0 protocol all prio 15 u32 match mark 0x800D00A0 0xc03fffff flowid 1:16
realtc filter add dev eth0 protocol all prio 15 u32 match mark 0x400D00A0 0xc03fffff flowid 1:16
realtc filter add dev br0 protocol all prio 20 u32 match mark 0x8012003F 0xc03fffff flowid 1:13
realtc filter add dev eth0 protocol all prio 20 u32 match mark 0x4012003F 0xc03fffff flowid 1:13
realtc filter change dev br0 prio 22 protocol all handle 802::800 u32 flowid 1:13
realtc filter change dev eth0 prio 22 protocol all handle 802::800 u32 flowid 1:13
realtc filter change dev br0 prio 23 protocol all handle 804::800 u32 flowid 1:13
realtc filter change dev eth0 prio 23 protocol all handle 804::800 u32 flowid 1:13
realtc filter change dev br0 prio 2 protocol all handle 827::802 u32 flowid 1:16
realtc filter change dev eth0 prio 2 protocol all handle 827::802 u32 flowid 1:16
realtc class change dev br0 parent 1:1 classid 1:10 htb overhead 18 linklayer ethernet prio 0 rate 5120Kbit ceil 102400Kbit burst 6400b cburst 125Kb
realtc class change dev br0 parent 1:1 classid 1:11 htb overhead 18 linklayer ethernet prio 1 rate 30720Kbit ceil 102400Kbit burst 25Kb cburst 125Kb quantum 384000
realtc class change dev br0 parent 1:1 classid 1:12 htb overhead 18 linklayer ethernet prio 2 rate 15360Kbit ceil 102400Kbit burst 12796b cburst 125Kb
realtc class change dev br0 parent 1:1 classid 1:13 htb overhead 18 linklayer ethernet prio 3 rate 10240Kbit ceil 102400Kbit burst 6400b cburst 125Kb
realtc class change dev br0 parent 1:1 classid 1:14 htb overhead 18 linklayer ethernet prio 4 rate 10240Kbit ceil 102400Kbit burst 4797b cburst 125Kb
realtc class change dev br0 parent 1:1 classid 1:15 htb overhead 18 linklayer ethernet prio 5 rate 20480Kbit ceil 102400Kbit burst 3194b cburst 125Kb quantum 256000
realtc class change dev br0 parent 1:1 classid 1:16 htb overhead 18 linklayer ethernet prio 6 rate 5120Kbit ceil 102400Kbit burst 3198b cburst 125Kb
realtc class change dev br0 parent 1:1 classid 1:17 htb overhead 18 linklayer ethernet prio 7 rate 5120Kbit ceil 102400Kbit burst 3200b cburst 125Kb
realtc class change dev eth0 parent 1:1 classid 1:10 htb overhead 18 linklayer ethernet prio 0 rate 256Kbit ceil 5120Kbit burst 3200b cburst 6400b
realtc class change dev eth0 parent 1:1 classid 1:11 htb overhead 18 linklayer ethernet prio 1 rate 512Kbit ceil 5120Kbit burst 3200b cburst 6400b
realtc class change dev eth0 parent 1:1 classid 1:12 htb overhead 18 linklayer ethernet prio 2 rate 768Kbit ceil 5120Kbit burst 3199b cburst 6400b
realtc class change dev eth0 parent 1:1 classid 1:13 htb overhead 18 linklayer ethernet prio 3 rate 512Kbit ceil 5120Kbit burst 3200b cburst 6400b
realtc class change dev eth0 parent 1:1 classid 1:14 htb overhead 18 linklayer ethernet prio 4 rate 1536Kbit ceil 5120Kbit burst 3199b cburst 6400b
realtc class change dev eth0 parent 1:1 classid 1:15 htb overhead 18 linklayer ethernet prio 5 rate 1024Kbit ceil 5120Kbit burst 3198b cburst 6400b
realtc class change dev eth0 parent 1:1 classid 1:16 htb overhead 18 linklayer ethernet prio 6 rate 256Kbit ceil 5120Kbit burst 3198b cburst 6400b
realtc class change dev eth0 parent 1:1 classid 1:17 htb overhead 18 linklayer ethernet prio 7 rate 256Kbit ceil 5120Kbit burst 3198b cburst 6400b

 

[Milan]

Sounds like you are watching Youtube in Chrome Browser. You need to add two rules to the "iptables Rules" to move it to streaming:

1)
Protocol = TCP
Remote Port = 443
Mark = 13005E
Class = Sreaming

2)
Protocol = UDP
Remote Port = 443
Mark = 000000
Class = Sreaming

Enjoy,

Morris

same i on latest firefox. seems youtube is encapsulating traffic to HTTP over TLS SSL.

your rules will move all HTTP over TLS SSL to streaming, not only youtube.

EDIT: to be precise, only web player is affected. if you use some sort of native app then it is categorized properly.

 

[brummygit]

I’m less enamored with this experiment since there is such a lag in updates to the data and meters. My speed tests are done before the meters really get going. Won’t be in 1.0 for sure.

Yes it is a bit laggy, and it could just be in my head but I think the rest of the UI might be a little slower as a result. But there is a lot of potential for visualising what's going on a bit better, however it's your project so I will be happy with whatever you decide.

 

[Vexira]

Thanks Dave

I've managed to fix the issue sort of, now both Origin and Steam show as game downloads, however when you actually start a game download it's still under file downloads. I suspect therefore that the trend DB just doesn't have this stuff categorised correctly. Since this isn't an issue with your script I'll try and figure things out.

How about Uplay and battle.net traffic?

I'm curious about what rules you have used to fix the classification of steam and origin of you don't mind me asking.

 

[JohnSmith]

@dave14305 , as I currently turn FlexQOS ON and OFF during testing and sometimes run it live depending on network conditions, if QOS is turned OFF, but I happen to go check for FlexQOS updates, it deletes the FlexQOS script. Can you add to your update script that if QOS is OFF, but FlexQOS is present, that either you give the option to delete it, or it is just left there, and you have to run the menu or command line in order to uninstall the script?

 

[dave14305]

I have a problem
First here is my setup;
- I have 100 mbps down 5 mbps upload speed.
- I'm using my AC88u as VPN client and my nvidia sheld tv is routed to VPN client.

When I set Bandwidth Setting as auto; my nvidia shield TV can get max of 50 mbits but VPN client traffic is detected wrong by router. here is the proof;

View attachment 25176View attachment 25175



If I set Bandwidth Setting as Manual and set it as 100 down and 5 up, my shield TV can only get 5 mbps;

View attachment 25178

View attachment 25179

Comments:

1. When Automatic Bandwidth is selected, the script skips any rate modifications since it no longer knows your upload/download rates (they are set to 0 in the GUI).
2. Most of the VPN traffic classification as upload/dowload was fixed in Merlin a while back. Check for the existence of a /etc/openvpn/fw/client-qos.sh script or similar and see what's inside.
3. Post the output of iptables -t mangle -nvL POSTROUTING

 

[dave14305]

@dave14305 , as I currently turn FlexQOS ON and OFF during testing and sometimes run it live depending on network conditions, if QOS is turned OFF, but I happen to go check for FlexQOS updates, it deletes the FlexQOS script. Can you add to your update script that if QOS is OFF, but FlexQOS is present, that either you give the option to delete it, or it is just left there, and you have to run the menu or command line in order to uninstall the script?

Deleting the script during a failed installation is necessary for integration with amtm. I consider QoS being disabled a failed pre-requisite for installing FlexQoS. An update uses the same install function as a fresh install. So I'll have to think about this one, but in the meantime, don't do that.

 

[pattiri]

Comments:

1. When Automatic Bandwidth is selected, the script skips any rate modifications since it no longer knows your upload/download rates (they are set to 0 in the GUI).
2. Most of the VPN traffic classification as upload/dowload was fixed in Merlin a while back. Check for the existence of a /etc/openvpn/fw/client-qos.sh script or similar and see what's inside.
3. Post the output of iptables -t mangle -nvL POSTROUTING

1. got it.
2. There is client1-fw.sh and server1-fw.sh. how can I find this "client-qos.sh"?
3.

admin@FaTiii:/tmp/home/root# iptables -t mangle -nvL POSTROUTING
Chain POSTROUTING (policy ACCEPT 2048K packets, 350M bytes)
 pkts bytes target     prot opt in     out     source               destination         
   44  8796 MARK       udp  --  *      br0     0.0.0.0/0            0.0.0.0/0            multiport sports 500,4500 MARK set 0x80060001
   45 11626 MARK       udp  --  *      eth0    0.0.0.0/0            0.0.0.0/0            multiport dports 500,4500 MARK set 0x40060001
    0     0 MARK       udp  --  *      br0     0.0.0.0/0            0.0.0.0/0            udp dpts:16384:16415 MARK set 0x80060001
    0     0 MARK       udp  --  *      eth0    0.0.0.0/0            0.0.0.0/0            udp spts:16384:16415 MARK set 0x40060001
    0     0 MARK       tcp  --  *      br0     0.0.0.0/0            0.0.0.0/0            multiport sports 119,563 MARK set 0x80030001
    0     0 MARK       tcp  --  *      eth0    0.0.0.0/0            0.0.0.0/0            multiport dports 119,563 MARK set 0x40030001
 6918 6989K MARK       tcp  --  *      br0     0.0.0.0/0            0.0.0.0/0            multiport sports 80,443 mark match 0x80080000/0xc03f0000 MARK set 0x803f0001
 5169  864K MARK       tcp  --  *      eth0    0.0.0.0/0            0.0.0.0/0            multiport dports 80,443 mark match 0x40080000/0xc03f0000 MARK set 0x403f0001

 

[pattiri]

Check for the existence of a /etc/openvpn/fw/client-qos.sh script or similar and see what's inside.

172.24.5.0/255.255.255.0 is my LAN subnet.

#!/bin/sh
iptables -I OVPN -i tun11 -j DROP
iptables -t mangle -I PREROUTING -i tun11 -j MARK --set-mark 0x01/0x7
iptables -t nat -I POSTROUTING -s 172.24.5.0/255.255.255.0 -o tun11 -j MASQUERADE
for i in /proc/sys/net/ipv4/conf/*/rp_filter ; do
echo 0 > $i
done

 

[dave14305]

There is client1-fw.sh and server1-fw.sh. how can I find this "client-qos.sh"?

I don't use VPN clients so I have no idea. But it's part of the whole updown-client.sh feature on 384.18.

 

[pattiri]

I don't use VPN clients so I have no idea. But it's part of the whole updown-client.sh feature on 384.18.

I've updated to 384.19 beta1 and now it's OK. Thanks