FlexQoS FlexQoS 1.0 - Flexible QoS Enhancement Script for Adaptive QoS

[Gatsu]

Firstly:
@dave14305 thanks for your great work

Secondly:
Is there a way to reapply qos_ibw and recalculate coresponding values without restarting QOS/Firewall ? My main connection is LTE and download speed vary from 70Mbps to 8Mbps. I frankenstein-ed small script that calculate new download speed based on current ping, optimal ping and last speed and use part of @squidbilly AutoBW script to apply it, but recently it stopped working, and when it was working it make my connection sluggish after some time. So I'm open to every sugestion

 

[dave14305]

Firstly:
@dave14305 thanks for your great work

Secondly:
Is there a way to reapply qos_ibw and recalculate coresponding values without restarting QOS/Firewall ? My main connection is LTE and download speed vary from 70Mbps to 8Mbps. I frankenstein-ed small script that calculate new download speed based on current ping, optlimal ping and last speed and use part od @squidbilly AutoBW script to apply it, but recently it stopped working, and when it was working it make my connection sluggish after some time. So I'm open to every sugestion

Restarting QoS is the only way to force the Trend engine to regenerate the other class rates and ceilings that FlexQoS does not change (we change classes 1:10-1:17, but not classes 10:1-17:256, for example).

And restarting QoS wipes out the iptables mangle table, requiring a firewall restart.

Why do you think the script stopped working?

 

[Gatsu]

Restarting QoS is the only way to force the Trend engine to regenerate the other class rates and ceilings that FlexQoS does not change (we change classes 1:10-1:17, but not classes 10:1-17:256, for example).

And restarting QoS wipes out the iptables mangle table, requiring a firewall restart.

Why do you think the script stopped working?

I rewrited my script from the ground and it is working now (with @squidbilly AutoBW part , will see about sluggish connection in few hours).

So if I go with restart_qos approach I should restart_firewall also and I assume that restarting those 2 every 1-3 minutes isn't optimal because of other scripts (etc. Skynet). Damm LTE

 

[dave14305]

So if I go with restart_qos approach I should restart_firewall also and I assume that restarting those 2 every 1-3 minutes isn't optimal because of other scripts (etc. Skynet).

Restarting both takes a long time for me on an AC68U due to the Skynet ipset processing. You could just modify the classes directly instead of restarting qos if you intend to adjust it so frequently. I think it might be OCD to do it every few minutes, but I don’t know your script nor your setup. No harm in trying!

 

[Gatsu]

I rewrited my script to more "other human readable" form:
https://pastebin.com/mdz9jCTw
or

Spoiler: AutoBWPingModFQE

#!/bin/sh

readonly SCRIPT_NAME="AutoBWPingModFQE"

#Settings
pingC=5 #Ping quantity
pingZ=120 #Optimal ping
pingIP=8.8.8.8 #Ping IP
WspolGora=133 #Ping factor upper limit in %
WspolDol=50 #Ping factor lower limit in %
download_lower_limit=4 #Download lower limit in Mbps
download_upper_limit=120 #Download upper limit in Mbps
transfactor=33 #Transmission speed in %

#Constants
pingS=0 #Starting value for average ping, currently not used
counter=0 #Counter

ping_based_speed() {

echo Ping IP: $pingIP
while [ "$counter" -lt $pingC ] ; do
pingtest=$(ping -c 1 $pingIP | grep 'bytes from' | awk '{print $7}' | sed s/time=//)
pingtest=${pingtest%.*}
pingS=$((pingS+$pingtest))
counter=$((counter+1))
echo Ping: $pingtest ms
sleep 1
done

pingS=$((pingS / $pingC))
echo Average from $pingC pings $pingS ms

pingZ=$(echo " scale=2; $pingZ / $pingS * 100 " | bc -l)
pingZ=${pingZ%.*}

if [ $pingZ -lt $WspolDol ]; then
pingZ=$WspolDol
echo Limiting ping factor to $(echo " scale=2; $WspolDol / 100 " | bc -l) "(lower limit)"
elif [ $pingZ -gt $WspolGora ]; then
pingZ=$WspolGora
echo Limiting ping factor to $(echo " scale=2; $WspolGora / 100 " | bc -l) "(upper limit)"
fi

pingZ=$(echo " scale=2; $pingZ / 100 " | bc -l)
echo Ping factor $pingZ

old_Kbps_down=$(nvram get qos_ibw)
echo QOS Download old $old_Kbps_down Kbps
Kbps_down=$(echo " $old_Kbps_down * $pingZ " | bc -l)
Kbps_down=${Kbps_down%.*}
echo QOS Download new $Kbps_down Kbps
}

update_via_TC_down() {
#read parameters for tc
PARMS=""
OVERHEAD=$(nvram get qos_overhead)
if [ ! -z "$OVERHEAD" ] && [ "$OVERHEAD" -gt "0" ] ; then
ATM=$(nvram get qos_atm)
if [ "$ATM" == "1" ] ; then
PARMS="overhead $OVERHEAD linklayer atm "
else
PARMS="overhead $OVERHEAD linklayer ethernet "
fi
fi

#Calculate scale factor based on new/old speeds. This factor will be used to scale class rates & ceils
#If old speeds are 0, just set the scale factor to 1 (cannot divide by 0)
# if [ "$old_Kbps_up" -eq "0" ]; then
# uscale_factor=1.0
# else
# uscale_factor=$( echo "$Kbps_up/$old_Kbps_up" | bc -l )
# fi
if [ "$old_Kbps_down" -eq "0" ]; then
dscale_factor=1.0
else
dscale_factor=$( echo "$Kbps_down/$old_Kbps_down" | bc -l )
fi

printf "%s %8.3f %8.3f\n" "Scaling by" $dscale_factor

echo
echo " ====================================== DOWNLOAD ======================================"
echo " ----------------- RATE ----------------- ----------------- CEIL -----------------"
echo "Class Previous (Kbps) New (Kbps) Previous (Kbps) New (Kbps) "
echo "----- ------------------- ------------------- ------------------- -------------------"
while read -r line;
do
class=$( echo ${line} | sed -n -e 's/.*class htb 1://p' | cut -d' ' -f1 )
prio=$( echo ${class} | cut -c 2 )

prevRate=$( echo ${line} | awk -F"rate |K?M?G?bit" '{print $2}' )
prevCeil=$( echo ${line} | awk -F"ceil |K?M?G?bit" '{print $3}' )
rateUnit=$( echo ${line} | cut -d' ' -f11)
rateUnit=${rateUnit//[0-9]/}
ceilUnit=$( echo ${line} | cut -d' ' -f15)
ceilUnit=${ceilUnit//[0-9]/}
if [ "$rateUnit" = "bit" ]; then
prevRate=$( echo "$prevRate*0.001" | bc -l)
fi
if [ "$rateUnit" = "Mbit" ]; then
prevRate=$( echo "$prevRate*1000" | bc -l)
fi
if [ "$rateUnit" = "Gbit" ]; then
prevRate=$( echo "$prevRate*1000000" | bc -l)
fi

if [ "$ceilUnit" = "bit" ]; then
prevCeil=$( echo "$prevCeil*0.001" | bc -l)
fi
if [ "$ceilUnit" = "Mbit" ]; then
prevCeil=$( echo "$prevCeil*1000" | bc -l)
fi
if [ "$ceilUnit" = "Gbit" ]; then
prevCeil=$( echo "$prevCeil*1000000" | bc -l)
fi

Burst=$( echo ${line} | sed -n -e 's/.* burst \([a-zA-Z0-9]*\).*/\1/p' )
Cburst=$( echo ${line} | sed -n -e 's/.*cburst \([a-zA-Z0-9]*\).*/\1/p' )

Rate=$(echo "$prevRate*$dscale_factor" | bc -l)
Ceil=$(echo "$prevCeil*$dscale_factor" | bc -l)

printf "%2.0f %10.1f %10.1f %10.1f %10.1f\n" $class $prevRate $Rate $prevCeil $Ceil

#Debug
#echo -e "$line\n class=$class\n prio=$prio\n Rate=$Rate\n Ceil=$Ceil\n Burst=$Burst\n Cburst=$Cburst\n"

#echo -e "tc class change dev br0 parent 1:1 classid 1:${class} htb ${PARMS}prio ${prio} rate ${Rate}Kbit ceil ${Ceil}Kbit burst ${Burst} cburst ${Cburst}"
Rate=$(echo $Rate | cut -d. -f1)
Ceil=$(echo $Ceil | cut -d. -f1)

tc class change dev br0 parent 1:1 classid 1:${class} htb ${PARMS}prio ${prio} rate ${Rate}Kbit ceil ${Ceil}Kbit burst ${Burst} cburst ${Cburst}

done <<EOF
$( tc class show dev br0 | grep "parent 1:1 " )
EOF
}

update_via_restart_qos_flexqos_check(){
service restart_qos >/dev/null 2>&1
/jffs/addons/flexqos/flexqos.sh -check &
}

checking_and_applying_limits() {

download_lower_limit=$(echo "$download_lower_limit*1024" | bc -l)
download_upper_limit=$(echo "$download_upper_limit*1024" | bc -l)

if [ $Kbps_down -lt $download_upper_limit ]; then
if [ $Kbps_down -lt $download_lower_limit ]; then
nvram set qos_ibw=$download_lower_limit
nvram commit
update_via_TC_down # for testing change to update_via_restart_qos_flexqos_check
logger -t $SCRIPT_NAME QOS Download set to $download_lower_limit Kbps "(LOW LIMIT)", average from $pingC pings $pingS ms
echo $SCRIPT_NAME 1 QOS Download set to $download_lower_limit Kbps "(LOW LIMIT!!!)", average from $pingC pings $pingS ms
else
nvram set qos_ibw=$Kbps_down
nvram commit
update_via_TC_down # for testing change to update_via_restart_qos_flexqos_check
logger -t $SCRIPT_NAME QOS Download set to $Kbps_down Kbps, average from $pingC pings $pingS ms
echo $SCRIPT_NAME 2 QOS Download set to $Kbps_down Kbps, average from $pingC pings $pingS ms
fi
else
logger -t $SCRIPT_NAME "QOS Download doesn't change, average from" $pingC pings $pingS ms
echo $SCRIPT_NAME 3 "QOS Download doesn't change, average from" $pingC pings $pingS ms
fi
}

#Limiting Transmission Download
translimit() {
transfactor=$(echo " 100 / $transfactor " | bc -l)
transfactor=${transfactor%.*}
Kbps_downtrans=$(echo " $Kbps_down / $transfactor / 8 " | bc -l)
Kbps_downtrans=${Kbps_downtrans%.*}
echo Limit transmission: $Kbps_downtrans
transmission-remote -d $Kbps_downtrans > /dev/null 2>&1
}

#Saving recent download limit to file
saving_to_file() {
QOSD=$(nvram get qos_ibw)
QOSD=$(echo "scale=0; $QOSD / 1024;" | bc -l)
echo QOSD: "$QOSD" Mbps > /mnt/Trans/Finished/qos.txt
}

########### START HERE ###########

ping_based_speed
checking_and_applying_limits
translimit

I'm also using AC86U (full setup in signature) and this script is executed via cron every minute.

 

[dave14305]

I rewrited my script to more "other human readable" form:
https://pastebin.com/mdz9jCTw

I'm also using AC86U (full setup in signature) and this script is executed via cron every minute.

This seems like a lot of work to try to dynamically adjust QoS. Have you looked at CAKE to see if it behaves better for you without extra intervention?

CakeQOS - CakeQOS-Merlin

I am pleased to announce the release of CakeQOS-Merlin! Discussions (due to thread closure): https://github.com/ttgapers/cakeqos-merlin/discussions Current Version: 1.0.7 (Changelog) CakeQOS-Merlin is a custom add-on for supported Asus routers with Merlin firmware that introduces CAKE QoS as...

www.snbforums.com

 

[Gatsu]

I tried early version (pre 1.0) of cakeQOS but it was worse than Adaptive+FlexQoS. The main reason is my dynamic download speed (LTE connection, upload is somewhat constant at 15-17 Mbps). Maybe I give it another shot.

 

[dave14305]

Version 1.0.5 (25-Oct-2020)

NEW

• Order bandwidth table classes by user-defined QoS priorities (was a fixed order)
• Show effective rates after making bandwidth min/max changes

CHANGED

• Combine separate Download and Upload bandwidth tables into one
• Better handling of dual-WAN setups when determining WAN interface name
• Add better error handling when custom appdb or rate rules fail
• Disable certain UI features when QoS Bandwidth is Automatic (Bandwidth meters and effective BW rates)

FIXED

• Avoid jumping header during update check

Anyone that had been experiencing errors during a reboot with FlexQoS startup, please report if this release fixes those errors or not. If not, please share the content of /tmp/flexqos_tcrules.err if it exists.

Update via the WebUI "Check for Update" button, amtm or by running flexqos update

This will be the last release posted to the original master thread. Subsequent releases will be in their own dedicated thread. Most documentation and user instructions will move from post #1 to a Github wiki for a single source of truth.

 

[killahgoat]

Hi dave, is it possible to see the names of those connected to a repeater like the one in the classification tab. I remember being able to see the names during version 1.03 or earlier.

 

[dave14305]

Hi dave, is it possible to see the names of those connected to a repeater like the one in the classification tab. I remember being able to see the names during version 1.03 or earlier.

You're saying the Classification tab shows it correctly, but FlexQoS is incorrect? I don't have a repeater setup to experiment with, so you'll have to elaborate more on the issue. It's quite possible it's changed since I stopped fetching names from the DHCP page, but I should think it brings me closer to what happens on the Classification page (but maybe not, haven't looked that closely yet).

 

[killahgoat]

You're saying the Classification tab shows it correctly, but FlexQoS is incorrect? I don't have a repeater setup to experiment with, so you'll have to elaborate more on the issue. It's quite possible it's changed since I stopped fetching names from the DHCP page, but I should think it brings me closer to what happens on the Classification page (but maybe not, haven't looked that closely yet).

Yes, the classification tab shows it correctly, but flexQOS is incorrect.

Under the Classification tab, it shows the name of the devices connected to the repeater whereas in the FlexQOS tab they just all fall under one name RTN18U(repeater)

Also, for some reason I have been getting a lot of ping spikes and instability while playing games with version 1.05 compared to version 1.03 (I skipped version 1.04).

 

[dave14305]

Yes, the classification tab shows it correctly, but flexQOS is incorrect.

Under the Classification tab, it shows the name of the devices connected to the repeater whereas in the FlexQOS tab they just all fall under one name RTN18U(repeater)

OK Killah, I've pushed a hotfix that should restore the previous behavior. Let me know if it does or not for your setup.

Also, for some reason I have been getting a lot of ping spikes and instability while playing games with version 1.05 compared to version 1.03 (I skipped version 1.04).

I don't think any of the underlying existing tc setup changed from 1.0.3 to 1.0.5. You can add negated mark rules, but no rate, quantum, burst changes have been made. Can't offer much help there.

 

[killahgoat]

OK Killah, I've pushed a hotfix that should restore the previous behavior. Let me know if it does or not for your setup.

I don't think any of the underlying existing tc setup changed from 1.0.3 to 1.0.5. You can add negated mark rules, but no rate, quantum, burst changes have been made. Can't offer much help there.

The hotfix worked! I can now see the device name connected to the repeater. Thank you!

With regard to the ping spikes, maybe my ISP is just acting up.

 

[GitzJoey]

Hi,
I want to understand a little bit here, since i want to make sure my gaming session is uninterruptible
my router is mainly for streaming and gaming, but i want to make sure gaming is the main priority

is switching the minimum in download (gaming become 30 and streaming become 15) good enough?

 

[dave14305]

Hi,
I want to understand a little bit here, since i want to make sure my gaming session is uninterruptible
my router is mainly for streaming and gaming, but i want to make sure gaming is the main priority

is switching the minimum in download (gaming become 30 and streaming become 15) good enough?

View attachment 27226

You should switch from Automatic to Manual bandwidth mode on the QoS tab and define your Upload and Download bandwidth so it is properly limited by QoS. This script doesn't really cope with Automatic Bandwidth mode very well.

Nothing will be uninterruptible, even with this script. Gaming will be given priority in your setup, but it's not exclusive priority. Everything gets a turn to send data, the question is how much at once, and in what order.

 

[killahgoat]

Hi Dave, flexqos seems to run smoother and the lag spikes are gone after the most recent hotfix. However, I noticed that the the Gaming rate doesn't reflect with the graphs despite being classified properly. It just stays on 0 the whole time. no performance issues though.

Debug:

Log date: 2020-10-28 16:10:54+0800
Router Model: RT-AC68U
Firmware Ver: 384.19_0
tc WAN iface: eth0
Undf Prio: 2
Undf FlowID: 1:17
Down Band: 46080
Up Band : 15360
***********
Net Control: 1:10
Work-From-Home: 1:16
Gaming: 1:11
Others: 1:12
Web Surfing: 1:14
Streaming: 1:15
File Downloads: 1:13
Game Downloads: 1:17
***********
Downrates: 6912, 460, 460, 460, 460, 460, 460, 460
Downceils: 43776, 43776, 43776, 43776, 43776, 43776, 43776, 43776
Downbursts: 8000, 3200, 3200, 3200, 3200, 3200, 3200, 3200
DownCbursts: 54400, 54400, 54400, 54400, 54400, 54400, 54400, 54400
DownQuantums: 86400, 5750, 5750, 5750, 5750, 5750, 5750, 5750
***********
Uprates: 2304, 153, 153, 153, 153, 153, 153, 153
Upceils: 9216, 9216, 9216, 9216, 9216, 9216, 9216, 9216
Upbursts: 3200, 3200, 3200, 3200, 3200, 3200, 3200, 3200
UpCbursts: 11200, 11200, 11200, 11200, 11200, 11200, 11200, 11200
UpQuantums: 28800, 1912, 1912, 1912, 1912, 1912, 1912, 1912
***********
iptables settings: <>>udp>>!80,443,8080>!08****>7<>>tcp>>!80,443,8080>!08****>7<>>both>>80,443,8080>>7<192.168.1.128/30>>udp>>!80,443,8080>000000>1<>>udp>>8801:8810>>3<>>udp>>53,8053,32100,4500,500,123>>7<>>tcp>>1119,3074>000000>1
-o br0 -p udp -m multiport ! --sports 80,443,8080 -m mark ! --mark 0x80080000/0xc03f0000 -j MARK --set-mark 0x803fffff/0x3fffff
-o eth0 -p udp -m multiport ! --dports 80,443,8080 -m mark ! --mark 0x40080000/0xc03f0000 -j MARK --set-mark 0x403fffff/0x3fffff
-o br0 -p tcp -m multiport ! --sports 80,443,8080 -m mark ! --mark 0x80080000/0xc03f0000 -j MARK --set-mark 0x803fffff/0x3fffff
-o eth0 -p tcp -m multiport ! --dports 80,443,8080 -m mark ! --mark 0x40080000/0xc03f0000 -j MARK --set-mark 0x403fffff/0x3fffff
-o br0 -p tcp -m multiport --sports 80,443,8080 -j MARK --set-mark 0x803fffff/0x3fffff
-o br0 -p udp -m multiport --sports 80,443,8080 -j MARK --set-mark 0x803fffff/0x3fffff
-o eth0 -p tcp -m multiport --dports 80,443,8080 -j MARK --set-mark 0x403fffff/0x3fffff
-o eth0 -p udp -m multiport --dports 80,443,8080 -j MARK --set-mark 0x403fffff/0x3fffff
-o br0 -d 192.168.1.128/30 -p udp -m multiport ! --sports 80,443,8080 -m mark --mark 0x80000000/0xc03f0000 -j MARK --set-mark 0x8008ffff/0x3fffff
-o eth0 -s 192.168.1.128/30 -p udp -m multiport ! --dports 80,443,8080 -m mark --mark 0x40000000/0xc03f0000 -j MARK --set-mark 0x4008ffff/0x3fffff
-o br0 -p udp --sport 8801:8810 -j MARK --set-mark 0x8006ffff/0x3fffff
-o eth0 -p udp --dport 8801:8810 -j MARK --set-mark 0x4006ffff/0x3fffff
-o br0 -p udp -m multiport --sports 53,8053,32100,4500,500,123 -j MARK --set-mark 0x803fffff/0x3fffff
-o eth0 -p udp -m multiport --dports 53,8053,32100,4500,500,123 -j MARK --set-mark 0x403fffff/0x3fffff
-o br0 -p tcp -m multiport --sports 1119,3074 -m mark --mark 0x80000000/0xc03f0000 -j MARK --set-mark 0x8008ffff/0x3fffff
-o eth0 -p tcp -m multiport --dports 1119,3074 -m mark --mark 0x40000000/0xc03f0000 -j MARK --set-mark 0x4008ffff/0x3fffff
***********
appdb rules: <000000>7<04****>7<00****>7
filter change dev br0 prio 2 protocol all handle 828::800 u32 flowid 1:17
filter change dev eth0 prio 2 protocol all handle 828::800 u32 flowid 1:17
filter change dev br0 prio 7 protocol all handle 821::800 u32 flowid 1:17
filter change dev eth0 prio 7 protocol all handle 821::800 u32 flowid 1:17
filter change dev br0 prio 3 protocol all handle 822::800 u32 flowid 1:17
filter change dev eth0 prio 3 protocol all handle 822::800 u32 flowid 1:17
[/CODE][/SPOILER]

 

[dave14305]

However, I noticed that the the Gaming rate doesn't reflect with the graphs despite being classified properly. It just stays on 0 the whole time.

What does the Classification tab report?

 

[killahgoat]

What does the Classification tab report?

Removing the 1st rule <>>udp>>!80,443,8080>!08****>seems to solve the problem.

 

[dave14305]

Removing the 1st rule <>>udp>>!80,443,8080>!08****>seems to solve the problem.

It doesn’t look much better to me, assuming there’s actually traffic on your network right now. Can you post the output of this command:

realtc -s class show dev br0 parent 1:

Also check your syslog to see if QoS keeps restarting or not.

 

[killahgoat]

It doesn’t look much better to me, assuming there’s actually traffic on your network right now. Can you post the output of this command:

realtc -s class show dev br0 parent 1:

Also check your syslog to see if QoS keeps restarting or not.

No sign of QoS restarting in syslog.

ASUSWRT-Merlin RT-AC68U 384.19_0 Fri Aug 14 19:17:44 UTC 2020
admin@RT-AC68U-E990:/tmp/home/root# realtc -s class show dev br0 parent 1:
class htb 1:11 parent 1:1 leaf 11: prio 1 rate 4608Kbit ceil 43776Kbit burst 4799b cburst 54391b
Sent 3359 bytes 11 pkt (dropped 0, overlimits 0 requeues 0)
rate 0bit 0pps backlog 0b 0p requeues 0
lended: 11 borrowed: 0 giants: 0
tokens: 128688 ctokens: 155172

class htb 1:10 parent 1:1 leaf 10: prio 0 rate 460Kbit ceil 43776Kbit burst 3199b cburst 54391b
Sent 2001 bytes 20 pkt (dropped 0, overlimits 0 requeues 0)
rate 16bit 0pps backlog 0b 0p requeues 0
lended: 20 borrowed: 0 giants: 0
tokens: 841312 ctokens: 155032

class htb 1:1 root rate 46080Kbit ceil 46080Kbit burst 57600b cburst 57600b
Sent 242677408 bytes 195322 pkt (dropped 0, overlimits 0 requeues 0)
rate 1118Kbit 138pps backlog 0b 0p requeues 0
lended: 135122 borrowed: 0 giants: 0
tokens: 154657 ctokens: 154657

class htb 1:2 root leaf 2: prio 0 rate 1Gbit ceil 1Gbit burst 1249375b cburst 1249375b
Sent 1750540 bytes 1683 pkt (dropped 0, overlimits 0 requeues 0)
rate 328bit 1pps backlog 0b 0p requeues 0
lended: 1384 borrowed: 0 giants: 0
tokens: 156187 ctokens: 156187

class htb 1:13 parent 1:1 leaf 13: prio 3 rate 460Kbit ceil 43776Kbit burst 3199b cburst 54391b
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)