Adaptive CAKE QoS

[dave14305]

EDIT: After a couple days of playing with it, I'm shutting this experiment down. Thanks for engaging with me.

I've been reviewing old discussions and PMs about CAKE when @RMerlin was adding it to the firmware earlier this year. One idea RMerlin had was to see if it was technically feasible to use CAKE in combination with Adaptive QoS marks. Since I've had a renewed interest in CAKE this week, I decided to see what could be done.

What I have below is a script that you can run after setting up Adaptive QoS on your router (not CAKE). This script will remove the htb+sfq setup of Adaptive QoS and replace it with CAKE using diffserv4. Streaming will be sent to Video. File Transfers will be sent to Bulk. IM, Gaming and Video Conferencing will be sent to Voice. Download shaping happens on the br0 bridge. Router to LAN traffic is excluded from CAKE.

I do not disable runner or flow cache. I can get CAKE to shape at roughly 385Mbit on my router, even though it's configured for 470Mbit. That's considerably more than the ~220Mbit I might expect with the default CAKE setup on an AC86U.

This might be a big turd, or it might be the start of something cool. If you're brave, you can copy/paste it onto your router and run it after plain Adaptive QoS is running.

If anyone else can reproduce my results (i.e. better performance and tinning than default CAKE or even CakeQOS-Merlin), then we'll see where we go from here. This might become a feature of FlexQoS 3.0. All the Adaptive QoS marks are available to use, but I only included the ones I thought should not be sent to besteffort.

Spoiler: flexcake.sh

#!/bin/sh

DL="$(nvram get qos_ibw)"
UL="$(nvram get qos_obw)"
OH="overhead $(nvram get qos_overhead)"
ATM="$(nvram get qos_atm)"
case ${ATM} in
    1) OH="${OH} atm" ;;
    2) OH="${OH} ptm" ;;
esac
OH="${OH} mpu $(nvram get qos_mpu)"
WAN=eth0

fc disable
fc flush

# Setup download
tc qdisc del dev br0 root
tc qdisc add dev br0 root handle 1: htb default 1
tc class add dev br0 parent 1: classid 1:2 htb prio 0 rate 10Gbit ceil 10Gbit
tc filter add dev br0 parent 1: protocol all prio 1 u32 match mark 0x0000 0xc0000000 flowid 1:2
tc class add dev br0 parent 1: classid 1:1 htb prio 0 rate 10Gbit ceil 10Gbit
tc qdisc add dev br0 parent 1:1 handle 110: cake bandwidth ${DL}kbit diffserv4 dual-dsthost ingress wash raw no-split-gso ${OH}

# Bulk :1
tc filter add dev br0 parent 110: protocol all prio 1 u32 match mark 0x80010000 0xc03f0000 action skbedit priority 110:1  # P2P
tc filter add dev br0 parent 110: protocol all prio 2 u32 match mark 0x80030000 0xc03f0000 action skbedit priority 110:1  # File Sharing
tc filter add dev br0 parent 110: protocol all prio 3 u32 match mark 0x800e0000 0xc03f0000 action skbedit priority 110:1  # Update downloads
# Video :3
tc filter add dev br0 parent 110: protocol all prio 4 u32 match mark 0x80160000 0xc03f0000 action skbedit priority 110:3  # Adult
tc filter add dev br0 parent 110: protocol all prio 5 u32 match mark 0x80040000 0xc03f0000 action skbedit priority 110:3  # Media Streaming
tc filter add dev br0 parent 110: protocol all prio 5 u32 match mark 0x801400ac 0xc03fffff action skbedit priority 110:3  # STUN
# Voice :4
tc filter add dev br0 parent 110: protocol all prio 6 u32 match mark 0x80060000 0xc03f0000 action skbedit priority 110:4  # Video Chat
tc filter add dev br0 parent 110: protocol all prio 7 u32 match mark 0x80080000 0xc03f0000 action skbedit priority 110:4  # Gaming
tc filter add dev br0 parent 110: protocol all prio 8 u32 match mark 0x80110000 0xc03f0000 action skbedit priority 110:4  # Video Conferencing
# Besteffort :2
tc filter add dev br0 parent 110: protocol all prio 99 matchall action skbedit priority 110:2

# Setup upload
tc qdisc del dev ${WAN} root
tc qdisc add dev ${WAN} root handle 120: cake bandwidth ${UL}kbit diffserv4 dual-srchost nat raw no-split-gso ${OH}

# Bulk :1
tc filter add dev ${WAN} parent 120: protocol all prio 1 u32 match mark 0x40010000 0xc03f0000 action skbedit priority 120:1  # P2P
tc filter add dev ${WAN} parent 120: protocol all prio 2 u32 match mark 0x40030000 0xc03f0000 action skbedit priority 120:1  # File Sharing
tc filter add dev ${WAN} parent 120: protocol all prio 3 u32 match mark 0x400e0000 0xc03f0000 action skbedit priority 120:1  # Update downloads
# Video :3
tc filter add dev ${WAN} parent 120: protocol all prio 4 u32 match mark 0x40160000 0xc03f0000 action skbedit priority 120:3  # Adult
tc filter add dev ${WAN} parent 120: protocol all prio 5 u32 match mark 0x40040000 0xc03f0000 action skbedit priority 120:3  # Media Streaming
tc filter add dev ${WAN} parent 120: protocol all prio 5 u32 match mark 0x401400ac 0xc03fffff action skbedit priority 120:3  # STUN

# Voice :4
tc filter add dev ${WAN} parent 120: protocol all prio 6 u32 match mark 0x40060000 0xc03f0000 action skbedit priority 120:4  # Video Chat
tc filter add dev ${WAN} parent 120: protocol all prio 7 u32 match mark 0x40080000 0xc03f0000 action skbedit priority 120:4  # Gaming
tc filter add dev ${WAN} parent 120: protocol all prio 8 u32 match mark 0x40110000 0xc03f0000 action skbedit priority 120:4  # Video Conferencing
# Besteffort :2
# tc filter add dev ${WAN} parent 120: protocol all prio 99 matchall action skbedit priority 120:2

Copy this script to a new file. Upload it to the router (e.g. /jffs/addons/flexcake.sh). Fix line endings (dos2unix /jffs/addons/flexcake.sh). Change permissions (chmod 755 /jffs/addons/flexcake.sh). Run the script once Adaptive QoS is running (sh /jffs/addons/flexcake.sh). Run a fast.com speedtest and see if the traffic ends up in the Video tin on br0 (tc -s qdisc show dev br0)

 

[dave14305]

With CAKE download configured to 470Mbit, I can achieve 446Mbit goodput (which is about right after taking overhead into account) when I use the CAKE options raw docsis no-split-gso.

I find it very compelling.

 

[rlj2]

tc -s qdisc show dev br0

I trust you enough I did this test while my wife was netflixing. My life was literally in your hands. So far im still around. As per our previous convos once I disabled flowcache on the AX86, flex-qos started working perfect. Ill have to play around with that. Not going to push it to much tonight.
Edit: looks like flex needs to be disabled. i think the rule check changed back to regular adaptive after a few minutes

 

[dave14305]

looks like flex needs to be disabled. i think the rule check changed back to regular adaptive after a few minutes

Yes, this test is meant to run with no qos addons active, to avoid such conflicts. You will still be able to see Tracked Connections on the Classification tab of the GUI, but without the pie charts for now (since the old categories are destroyed by the script).

 

[dave14305]

Interested in anyone with more than 500Mbit bandwidth to see how high you can speedtest with this experimental setup. My raw ISP speed is 480Mbit.

 

[rlj2]

Yes, this tet is meant to run with no qos addons active, to avoid such conflicts. You will still be able to see Tracked Connections on the Classification tab of the GUI, but without the pie charts for now (since the old categories are destroyed by the script).

ill play around. I can seldom use the classification tab. Always says disable to many connections.

 

[RMerlin]

I do not disable runner or flow cache. I can get CAKE to shape at roughly 385Mbit on my router, even though it's configured for 470Mbit. That's considerably more than the ~220Mbit I might expect with the default CAKE setup on an AC86U.

Based on some quick exchanges with one of the Cake devs at the time, hardware acceleration needs to be disabled because otherwise flows can get cached by the Broadcom (or QCA) blackbox, which means Cake won't be able to properly track them. Cake would need some API to be able to access the info related to those cached flows.

 

[Vexira]

So far im finding the results impressive, i still fail to understand why ASUS/Trend don't use the diffserve number that correlates to the adaptive categories, and set up something natively like this it would be absolutely brilliant, much better than the default sfq.
And as always, thank you @RMerlin and @dave14305

 

[albinoman887]

well i'll be dammed. at first i had some bufferbloat but i disabled hw accel and now i have A+ 0ms increase under load!

 

[dave14305]

Based on some quick exchanges with one of the Cake devs at the time, hardware acceleration needs to be disabled because otherwise flows can get cached by the Broadcom (or QCA) blackbox, which means Cake won't be able to properly track them. Cake would need some API to be able to access the info related to those cached flows.

I hear you, but I’m not yet seeing any ill effects of this experiment. The router thinks Adaptive QoS is running, so it’s doing the usual marking. How would fq_codel have kept track of flows? Was it a modified version with API access?

I’m not doing anything in iptables with the traffic, where I know HW acceleration could cause issues.

 

[Milan]

How to check that it is working - fast com is showing 940 Mbps on 1 Gbps line with AX88U

 

[dave14305]

How to check that it is working - fast com is showing 940 Mbps on 1 Gbps line with AX88U

First let’s see if CAKE recognized all the packets:

tc -s qdisc | grep -A29 cake
tc -s filter show dev br0 parent 11:
tc -s filter show dev eth0 parent 12:

 

[Milan]

seems yes:

tc -s qdisc
qdisc cake 12: dev eth0 root refcnt 2 bandwidth 102400Kbit diffserv4 dual-srchost nat nowash no-ack-filter no-split-gso rtt 100ms raw overhead 4
Sent 1111867963 bytes 899956 pkt (dropped 198, overlimits 779706 requeues 0)
backlog 0b 0p requeues 0
memory used: 1594880b of 5000Kb
capacity estimate: 102400Kbit
min/max network layer size: 42 / 1514
min/max overhead-adjusted size: 46 / 1518
average network hdr offset: 58

Bulk Best Effort Video Voice
thresh 6400Kbit 102400Kbit 51200Kbit 25600Kbit
target 5ms 5ms 5ms 5ms
interval 100ms 100ms 100ms 100ms
pk_delay 9.75ms 443us 3.21ms 293us
av_delay 3.99ms 25us 2.21ms 7us
sp_delay 3us 2us 3us 2us
backlog 0b 0b 0b 0b
pkts 609174 9624 281207 149
bytes 802050468 2211252 307875955 21665
way_inds 77490 25 16 0
way_miss 1516 1535 104 28
way_cols 0 0 0 0
drops 101 1 96 0
marks 0 0 0 0
ack_drop 0 0 0 0
sp_flows 7 1 1 1
bk_flows 1 0 0 0
un_flows 0 0 0 0
max_len 1494 1514 1514 1494
quantum 300 1514 1514 781

qdisc pfifo_fast 0: dev eth1 root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
Sent 543872430 bytes 1989874 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: dev eth2 root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
Sent 797845085 bytes 2576668 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: dev eth3 root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: dev eth4 root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
Sent 739884980 bytes 2500005 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: dev eth5 root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
Sent 56536828334 bytes 39629466 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: dev spu_us_dummy root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: dev spu_ds_dummy root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: dev eth6 root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
Sent 975346153 bytes 3246563 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: dev eth7 root refcnt 2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
Sent 105582393688 bytes 85792122 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc htb 1: dev br0 root refcnt 2 r2q 10 default 0x1 direct_packets_stat 0 direct_qlen 2
Sent 2015349638 bytes 1372313 pkt (dropped 25, overlimits 1212027 requeues 0)
backlog 0b 0p requeues 0
qdisc cake 11: dev br0 parent 1:1 bandwidth 1024Mbit diffserv4 dual-dsthost nonat wash ingress no-ack-filter no-split-gso rtt 100ms raw overhead 4
Sent 1952683577 bytes 1317397 pkt (dropped 0, overlimits 1255339 requeues 0)
backlog 0b 0p requeues 0
memory used: 392832b of 15140Kb
capacity estimate: 1024Mbit
min/max network layer size: 54 / 1514
min/max overhead-adjusted size: 58 / 1518
average network hdr offset: 37805

Bulk Best Effort Video Voice
thresh 64Mbit 1024Mbit 512Mbit 256Mbit
target 5ms 5ms 5ms 5ms
interval 100ms 100ms 100ms 100ms
pk_delay 15us 277us 869us 17us
av_delay 3us 37us 75us 3us
sp_delay 2us 2us 3us 1us
backlog 0b 0b 0b 0b
pkts 329 21429 1295497 142
bytes 184991 23028871 1929439527 30188
way_inds 0 62 0 0
way_miss 31 557 104 28
way_cols 0 0 0 0
drops 0 0 0 0
marks 0 0 0 0
ack_drop 0 0 0 0
sp_flows 2 1 0 0
bk_flows 0 1 0 0
un_flows 0 0 0 0
max_len 1514 1514 1514 1514
quantum 1514 1514 1514 1514

 

[Milan]

part 2:

tc -s filter show dev br0
filter parent 1: protocol all pref 1 u32
filter parent 1: protocol all pref 1 u32 fh 800: ht divisor 1
filter parent 1: protocol all pref 1 u32 fh 800::800 order 2048 key ht 800 bkt 0 flowid 1:2 (rule hit 1343582 success 25412)
mark 0x0000 0xc0000000 (success 25412)

tc -s filter show dev br0
filter parent 1: protocol all pref 1 u32
filter parent 1: protocol all pref 1 u32 fh 800: ht divisor 1
filter parent 1: protocol all pref 1 u32 fh 800::800 order 2048 key ht 800 bkt 0 flowid 1:2 (rule hit 1343582 success 25412)
mark 0x0000 0xc0000000 (success 25412)
admin@RT-AX88U-F810:/tmp/home/root#
admin@RT-AX88U-F810:/tmp/home/root# tc -s filter show dev eth0
filter parent 12: protocol all pref 1 u32
filter parent 12: protocol all pref 1 u32 fh 800: ht divisor 1
filter parent 12: protocol all pref 1 u32 fh 800::800 order 2048 key ht 800 bkt 0 terminal flowid ??? (rule hit 1026313 success 0)
mark 0x40010000 0xc03f0000 (success 0)
action order 1: skbedit priority 12:1 pipe
index 10 ref 1 bind 1 installed 909 sec used 909 sec firstused 558351646 sec
Action statistics:
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0

filter parent 12: protocol all pref 2 u32
filter parent 12: protocol all pref 2 u32 fh 801: ht divisor 1
filter parent 12: protocol all pref 2 u32 fh 801::800 order 2048 key ht 801 bkt 0 terminal flowid ??? (rule hit 1026312 success 734418)
mark 0x40030000 0xc03f0000 (success 734418)
action order 1: skbedit priority 12:1 pipe
index 11 ref 1 bind 1 installed 909 sec firstused 515401973 sec
Action statistics:
Sent 969128201 bytes 734418 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0

filter parent 12: protocol all pref 3 u32
filter parent 12: protocol all pref 3 u32 fh 802: ht divisor 1
filter parent 12: protocol all pref 3 u32 fh 802::800 order 2048 key ht 802 bkt 0 terminal flowid ??? (rule hit 291894 success 6)
mark 0x400e0000 0xc03f0000 (success 6)
action order 1: skbedit priority 12:1 pipe
index 12 ref 1 bind 1 installed 909 sec used 675 sec firstused 4037275156 sec
Action statistics:
Sent 822 bytes 6 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0

filter parent 12: protocol all pref 4 u32
filter parent 12: protocol all pref 4 u32 fh 803: ht divisor 1
filter parent 12: protocol all pref 4 u32 fh 803::800 order 2048 key ht 803 bkt 0 terminal flowid ??? (rule hit 291888 success 0)
mark 0x40160000 0xc03f0000 (success 0)
action order 1: skbedit priority 12:3 pipe
index 13 ref 1 bind 1 installed 909 sec used 909 sec firstused 3779577118 sec
Action statistics:
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0

filter parent 12: protocol all pref 5 u32
filter parent 12: protocol all pref 5 u32 fh 804: ht divisor 1
filter parent 12: protocol all pref 5 u32 fh 804::800 order 2048 key ht 804 bkt 0 terminal flowid ??? (rule hit 291888 success 281211)
mark 0x40040000 0xc03f0000 (success 281211)
action order 1: skbedit priority 12:3 pipe
index 14 ref 1 bind 1 installed 909 sec used 100 sec firstused 3779577118 sec
Action statistics:
Sent 307876226 bytes 281211 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0

filter parent 12: protocol all pref 6 u32
filter parent 12: protocol all pref 6 u32 fh 805: ht divisor 1
filter parent 12: protocol all pref 6 u32 fh 805::800 order 2048 key ht 805 bkt 0 terminal flowid ??? (rule hit 10677 success 152)
mark 0x40060000 0xc03f0000 (success 152)
action order 1: skbedit priority 12:4 pipe
index 15 ref 1 bind 1 installed 909 sec used 8 sec firstused 3307130716 sec
Action statistics:
Sent 32987 bytes 152 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0

filter parent 12: protocol all pref 7 u32
filter parent 12: protocol all pref 7 u32 fh 806: ht divisor 1
filter parent 12: protocol all pref 7 u32 fh 806::800 order 2048 key ht 806 bkt 0 terminal flowid ??? (rule hit 10525 success 25)
mark 0x40080000 0xc03f0000 (success 25)
action order 1: skbedit priority 12:4 pipe
index 16 ref 1 bind 1 installed 909 sec used 51 sec firstused 1073747722 sec
Action statistics:
Sent 1650 bytes 25 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0

filter parent 12: protocol all pref 8 u32
filter parent 12: protocol all pref 8 u32 fh 807: ht divisor 1
filter parent 12: protocol all pref 8 u32 fh 807::800 order 2048 key ht 807 bkt 0 terminal flowid ??? (rule hit 10500 success 0)
mark 0x40110000 0xc03f0000 (success 0)
action order 1: skbedit priority 12:4 pipe
index 17 ref 1 bind 1 installed 909 sec used 909 sec firstused 5898 sec
Action statistics:
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0

 

[rlj2]

Yes, this tet is meant to run with no qos addons active, to avoid such conflicts. You will still be able to see Tracked Connections on the Classification tab of the GUI, but without the pie charts for now (since the old categories are destroyed by the script).

seems to work as intended, even on the AX.

 

[dave14305]

seems to work as intended, even on the AX.

What speeds?

 

[Milan]

Mine is 1Gbps down 100Mbps up

 

[rlj2]

What speeds?

slooooow, i was just testing tins. only 25Mbit here

 

[dave14305]

Just made a small update to avoid using the same qdisc handles as Adaptive QoS (11: and 12: ). I found A.QoS had added extra classes and filters when another device came online overnight. Forgot about that behavior in normal A.QoS.

 

[rlj2]

Just made a small update to avoid using the same qdisc handles as Adaptive QoS (11: and 12: ). I found A.QoS had added extra classes and filters when another device came online overnight. Forgot about that behavior in normal A.QoS.

Ive got to do some more testing on the AX. noticed a weird thing playing with your script this morning. If FC was enabled , speedtests were only showing around 15MBit, the router speedtest showed the full 25. But when I disabled FC. speedtests jumped right up. Wth would cause that with the rules. Normal cake is fine. I swear this AX is a bastard router.
I hooked up a AC86u to the modem also I can play around with. Curious where your able to take this.