QoS-Cake Throttling USB?

[Lynx]

Why does:

qdisc cake 801a: dev br0 root refcnt 2 bandwidth 30720Kbit besteffort dual-dsthost nonat wash ingress no-ack-filter split-gso rtt 100ms noatm overhead 91
qdisc cake 8019: dev tun11 root refcnt 2 bandwidth 30720Kbit diffserv3 dual-srchost nat nowash no-ack-filter split-gso rtt 100ms noatm overhead 91

result in transfer with respect to the USB samba share being throttled? It seems cake on br0 throttles downloading from my USB samba share. If I set the download bandwidth to unlimited, the throttling goes away.
smb.conf includes the line:

interfaces = lo br0 192.168.1.1/255.255.255.0

QoS is setup using:

admin@RT-AX86U-4168:/jffs/scripts# cat /jffs/configs/cake-qos.conf.add
DLIF='br0'
DLOPTIONS='wash dual-dsthost ingress'
ULIF_TUN='tun11'
TUN_PACKET_OVERHEAD='53'

and also:

admin@RT-AX86U-4168:/jffs/scripts# cat qos-start
#!/bin/sh

(
cat <<'ADDTEXT'
#!/bin/sh
source /etc/cake-qos.conf

OVERHEAD_TUN=$(echo $OVERHEAD | awk '$2=$2+'$TUN_PACKET_OVERHEAD)

case "$1" in
start)
        if [[ -d /sys/class/net/"$ULIF_TUN" ]]; then
                tc qdisc add dev $ULIF_TUN root cake $ULPRIOQUEUE $ULOPTIONS $ULBW $OVERHEAD_TUN $FRAMING 2>/dev/null
                tc qdisc add dev $DLIF root cake $DLPRIOQUEUE $DLOPTIONS $DLBW $OVERHEAD_TUN $FRAMING 2>/dev/null
        else
                tc qdisc add dev $ULIF root cake $ULPRIOQUEUE $ULOPTIONS $ULBW $OVERHEAD $FRAMING 2>/dev/null
                tc qdisc add dev $DLIF root cake $DLPRIOQUEUE $DLOPTIONS $DLBW $OVERHEAD $FRAMING 2>/dev/null
        fi
        ;;
stop)
        tc qdisc del dev $ULIF root 2>/dev/null
        tc qdisc del dev $ULIF_TUN root 2>/dev/null
        tc qdisc del dev $DLIF root 2>/dev/null
        ;;
*)
esac
ADDTEXT
) > /tmp/qos

Loosely based on:

[Cake test] Cake experiments for 386.2 Beta

Hi folks, If you are currently testing Cake on 386.2 Beta, could you guys test the following customization, and see if there is any measurable improvement or issues? Make sure you have custom JFFS scripts enabled (Administration -> System). Then, create /jffs/configs/cake-qos.conf.add with...

www.snbforums.com

But altered to handle VPN active or not active.
How do I overcome this problem?

 

[TonyK132]

You may want to move this message to the Merlin AddOns Forum that covers cake-qos.

 

[Lynx]

I don't understand why samba share traffic is getting throttled. Is that really 'ingress' traffic on br0? Is the workaround the use of a 'filter' or something else here? This stuff gets quickly rather complicated, so hopefully somebody very bright has an idea here. Ideally with an explanation of why this problem occurs in the first place.

 

[ColinTaylor]

Generally speaking QoS has little or no control over the rate of incoming traffic. But it can shape outgoing traffic. Therefore to throttle "downloads" you are really controlling the rate at which traffic is forwarded through the LAN interface (br0). In the reverse situation, to shape "uploads" you control the rate of traffic sent through the WAN interface (eth0).

So this probably explains why the "download" rate is effecting router-to-LAN as well as WAN-to-LAN rates. I don't know how it's done in
QoS-Cake, but in John's implementation of Traditional QoS he had a specific priority level dedicated to router-to-LAN traffic so that it always operated without restrictions.

 

[dave14305]

This is one of several reasons the use of br0 didn't become a standard with the CAKE implementation in the firmware.

Samba traffic is treated as egress traffic on br0.

 

[Lynx]

OpenWRT allows you to set the interface to which shaping is applied in SQM config and then it creates the appropriate IFB for that selected interface. I am supposing such flexibility is not presently handled in the Asus-Merlin cake implementation?
So to overcome the above-identified problem with using 'br0', I assume that I would need to manually add the appropriate IFB for 'tun11' using the recipe here:

ip link add name ifb4eth2 type ifb
tc qdisc del dev eth2 ingress
tc qdisc add dev eth2 handle ffff: ingress
tc qdisc del dev ifb4eth2 root
tc qdisc add dev ifb4eth2 root cake bandwidth 110000kbit besteffort
ip link set ifb4eth2 up # if you don't bring the device up your connection will lock up on the next step.
tc filter add dev eth2 parent ffff: matchall action mirred egress redirect dev ifb4eth2

Cake - Bufferbloat.net

www.bufferbloat.net

Only in my case rather than 'eth2', I would use 'tun11'?
Since I need to deal with the 'stop' case in my 'qos-start' script, what would be the appropriate commands to delete the above I wonder?

 

[dave14305]

You made this trade-off when you wanted Cake to work with your traffic through a VPN tunnel. Cake won’t apply its fairness algorithms accurately to Download flows if you apply the ingress shaping on eth0 or tun11. That’s why you’re using br0 now.

 

[Lynx]

Thank you for the comments. I am hopeful that there is a solution here, in terms of QoS + VPN + unthrottled USB transfer - one way or another.
Option 1: using 'tun11' + ifb
I totally get that shaping on 'eth0' can never work in a VPN context because CAKE is operating entirely blind. And indeed switching to 'tun11'/'br0' made the QoS work very well for me. For me it makes sense on a theoretical level too, and it is what I see others using OpenWRT doing.
In any case, why can cake not work on 'tun11' in respect of downloads via an appropriate ifb? I have not tried this yet. Isn't both upload and download traffic on 'tun11' visible - as in, it is at a point in the flow outside the VPN encapsulation that otherwise prevents cake from working properly?
From my understanding of how the OpenWRT SQM works, it always has you set the interface and then creates the appropriate ifb. Looking over the QoS scripts, the implementation seems rather elegantly coded to me.

ingress() {
    sqm_debug "ingress"

    SILENT=1 $TC qdisc del dev $IFACE handle ffff: ingress
    $TC qdisc add dev $IFACE handle ffff: ingress
    SILENT=1 $TC qdisc del dev $DEV root

    [ "$ZERO_DSCP_INGRESS" -eq "1" ] && INGRESS_CAKE_OPTS="$INGRESS_CAKE_OPTS wash"

    $TC qdisc add dev $DEV root $( get_stab_string ) cake \
        bandwidth ${DOWNLINK}kbit $( get_cake_lla_string ) ${INGRESS_CAKE_OPTS} ${IQDISC_OPTS}

    $IP link set dev $DEV up

    # redirect all IP packets arriving in $IFACE to ifb0

    $TC filter add dev $IFACE parent ffff: protocol all prio 10 u32 \
    match u32 0 0 flowid 1:1 action mirred egress redirect dev $DEV
}

GitHub - tohojo/sqm-scripts: SQM scripts traffic shaper

SQM scripts traffic shaper. Contribute to tohojo/sqm-scripts development by creating an account on GitHub.

github.com

sqm-scripts/src/layer_cake.qos at master · tohojo/sqm-scripts

SQM scripts traffic shaper. Contribute to tohojo/sqm-scripts development by creating an account on GitHub.

github.com

So if using 'tun11' for uploads, doesn't it make sense to use 'tun11' via an appropriate ifb for downloads? I believe this is what would get set up in OpenWRT if I were to set the interface as the 'tun11' equivalent. This is what someone advised to do on their IRC channel anyway - they indicated that an SQM implementation should always set up an 'ifb' for the chosen interface.
Option 2: using 'br0' + tc filter
As a further alternative, am I not right in thinking that I can just set up a tc filter on 'br0' to handle only WAN traffic? Anyone particularly familiar with this approach that could make a suggestion?

 

[dave14305]

In any case, why can cake not work on 'tun11' in respect of downloads via an appropriate ifb? I have not tried this yet. Isn't both upload and download traffic on 'tun11' visible - as in, it is at a point in the flow outside the VPN encapsulation that otherwise prevents cake from working properly?

I’m pretty sure there will be another trade off in that the true Remote IP will be known, but the local IP will still be seen as the local IP of the VPN tunnel on the router, so you would lose any ability to apply fairness based on local LAN IPs. In other words, dual-dsthost won’t work well as an option for ifb4tun11. It is kind of the flip side of the remote side only being seen as the remote VPN server IP when shaping on eth0.

 

[Lynx]

I’m pretty sure there will be another trade off in that the true Remote IP will be known, but the local IP will still be seen as the local IP of the VPN tunnel on the router, so you would lose any ability to apply fairness based on local LAN IPs. In other words, dual-dsthost won’t work well as an option for ifb4tun11. It is kind of the flip side of the remote side only being seen as the remote VPN server IP when shaping on eth0.

Makes sense I think, providing local IP is truly obscured. Dumb question really but can't nat in this situation reveal the local LAN IP? I will try having a look at what 'tcpdump -vpni tun11' shows. From memory I could see the LAN IPs?
What about my 'option 2' above?

 

[dave14305]

Option 2: using 'br0' + tc filter
As a further alternative, am I not right in thinking that I can just set up a tc filter on 'br0' to handle only WAN traffic? Anyone particularly familiar with this approach that could make a suggestion?

What about my 'option 2' above?

I don’t know where you think you’d be filtering the traffic to. There aren’t options like there are with htb. What did you read that suggested it?

Dumb question really but can't nat in this situation reveal the local LAN IP?

I don’t believe it’s a NAT issue but a VPN issue at that point. I’m no VPN expert though.

 

[Lynx]

See e.g.:

So recent cake can actually be address from tc filters, so onr option would be to still use sqm-scripts with layer_cake and set up two filters to place torrents into the bulk tin and game traffic into the high pririty tin. Not as nifty as your example, but if all that is required is to sort stuff into three priority classes that should be fine

But how to do this?

first run "tc -s qdisc" and figure out the major number of the cake instance in question:
qdisc cake 801b: dev pppoe-wan root refcnt 2 bandwidth 9545Kbit diffserv3 dual-srchost nat split-gso rtt 100.0ms noatm overhead 34 mpu 64
In this example 801b for interfave pppoe-wan, and remember that number

Then look at the tins for diffserv3:

verage network hdr offset: 0

Bulk Best Effort Voice
thresh 596560bit 9545Kbit 2386Kbit
target 30.5ms 5.0ms 7.6ms
interval 125.5ms 100.0ms 102.6ms


1 being Bulk or Background and remember this as minor number.
Then all you need to do is to add a filter for the torrent port:

tc filter add dev pppoe-wan parent 801b: protocol ip u32 match ip dport 60435 0xffff action skbedit priority 801b:1

Mind you I have not tested that myself, but this was reported to work on the cake mailing list. @ldir can you spot an error of my copy of your command above

SQM, cake and piece_of_cake.qos - High CPU usage

So recent cake can actually be address from tc filters, so onr option would be to still use sqm-scripts with layer_cake and set up two filters to place torrents into the bulk tin and game traffic into the high pririty tin. Not as nifty as your example, but if all that is required is to sort...

forum.openwrt.org

And also:

OVERRIDING CLASSIFICATION WITH TC FILTERS​

CAKE supports overriding of its internal classification of packets through the tc filter mechanism. Packets can be assigned to different priority tins by setting the priority field on the skb, and the flow hashing can be overridden by setting the classid parameter.

Tin override

To assign a priority tin, the major number of the priority field needs to match the qdisc handle of the cake instance; if it does, the minor number will be interpreted as the tin index. For example, to classify all ICMP packets as 'bulk', the following filter can be used:

# tc qdisc replace dev eth0 handle 1: root cake diffserv3
# tc filter add dev eth0 parent 1: protocol ip prio 1 \
u32 match icmp type 0 0 action skbedit priority 1:1

Flow hash override

To override flow hashing, the classid can be set. CAKE will interpret the major number of the classid as the host hash used in host isolation mode, and the minor number as the flow hash used for flow-based queueing. One or both of those can be set, and will be used if the relevant flow isolation parameter is set (i.e., the major number will be ignored if CAKE is not configured in hosts mode, and the minor number will be ignored if CAKE is not configured in flows mode).


This example will assign all ICMP packets to the first queue:

# tc qdisc replace dev eth0 handle 1: root cake
# tc filter add dev eth0 parent 1: protocol ip prio 1 \
u32 match icmp type 0 0 classid 0:1

If only one of the host and flow overrides is set, CAKE will compute the other hash from the packet as normal. Note, however, that the host isolation mode works by assigning a host ID to the flow queue; so if overriding both host and flow, the same flow cannot have more than one host assigned. In addition, it is not possible to assign different source and destination host IDs through the override mechanism; if a host ID is assigned, it will be used as both source and destination host.

Man page of CAKE

Option 3: create ifb from 'br0' that mirrors only matched WAN traffic
Or as a still further alternative, can I not just match 'br0' traffic of interest and create 'ifb' with that matched traffic. Then cake on that?
I am sure there is a good way forward here, if not one of the above options.

 

[dave14305]

Option 3: create ifb from 'br0' that mirrors only matched WAN traffic
Or as a still further alternative, can I not just match 'br0' traffic of interest and create 'ifb' with that matched traffic. Then cake on that?
I am sure there is a good way forward here, if not one of the above options.

You’re going to have to start testing yourself because you’re in uncharted territory, as far as I can tell.

 

[Lynx]

Exploring 'Option 3', does the following make sense to you?
To strip out router originating traffic on 'br0' from the cake:

tc filter add dev $DLIF parent ffff: protocol ip prio 1 u32 match ip src 192.168.1.1 action pass
tc filter add dev $DLIF parent ffff: protocol ip prio 2 matchall action mirred egress redirect dev $DLIF_IFB

That is to say, when a packet is encountered on 'br0' if it originates from the router then pass, else mirror egress to the DLIF_IFB, and so cake correctly operates on DLIF_IFB with everything except router-originating traffic? So this should take out samba share traffic originating from the router on download?
Making the new QoS script to handle VPN active / not-active (and pass on router-originating traffic):

admin@RT-AX86U-4168:/jffs/scripts# cat qos-start
#!/bin/sh

(
cat <<'ADDTEXT'
#!/bin/sh
source /etc/cake-qos.conf

OVERHEAD_TUN=$(echo $OVERHEAD | awk '$2=$2+'$TUN_PACKET_OVERHEAD)
DLIF_IFB=ifb4$DLIF

case "$1" in
start)
    ip link add name $DLIF_IFB type ifb
    tc qdisc add dev $DLIF handle ffff: ingress
    if [[ -d /sys/class/net/"$ULIF_TUN" ]]; then
                tc qdisc add dev $ULIF_TUN root cake $ULPRIOQUEUE $ULOPTIONS $ULBW $OVERHEAD_TUN $FRAMING 2>/dev/null
                tc qdisc add dev $DLIF_IFB root cake $DLPRIOQUEUE $DLOPTIONS $DLBW $OVERHEAD_TUN $FRAMING 2>/dev/null
    else
                tc qdisc add dev $ULIF root cake $ULPRIOQUEUE $ULOPTIONS $ULBW $OVERHEAD $FRAMING 2>/dev/null
                tc qdisc add dev $DLIF_IFB root cake $DLPRIOQUEUE $DLOPTIONS $DLBW $OVERHEAD $FRAMING 2>/dev/null
    fi
    ip link set $DLIF_IFB up
    tc filter add dev $DLIF parent ffff: protocol ip prio 1 u32 match ip src 192.168.1.1 action pass
    tc filter add dev $DLIF parent ffff: protocol ip prio 2 matchall action mirred egress redirect dev $DLIF_IFB
     ;;
stop)
    tc qdisc del dev $ULIF root 2>/dev/null
    tc qdisc del dev $ULIF_TUN root 2>/dev/null
    tc qdisc del dev $DLIF root 2>/dev/null
    ip link set dev $DLIF_IFB down
    ip link delete $DLIF_IFB type ifb
    ;;
*)
esac
ADDTEXT
) > /tmp/qos

n.b. I ought to use an 'nvram' for the router LAN to set this properly.

 

[Lynx]

I am also curious: why does LAN-LAN not count as 'ingress' but USB samba share download does count as 'ingress'? Could I just modify the smb.conf in some way to ensure the USB samba share download is not counted as 'ingress' on 'br0'? Or might there be another workaround?

 

[dave14305]

I am also curious: why does LAN-LAN not count as 'ingress' but USB samba share download does count as 'ingress'? Could I just modify the smb.conf in some way to ensure the USB samba share download is not counted as 'ingress' on 'br0'? Or might there be another workaround?

It would be egress on br0, not ingress. Ingress would be traffic FROM the LAN. br0 Egress would be traffic TO the LAN from the Router or Internet.

 

[heysoundude]

This may or may not still be the case, but I seem to recall that at one point cake was to be placed/run on eth0 via config, which is/was the WAN port? I'm probably misremembering...but it would be worth a browse through the original threads, unless @ttgapers happens to see this and respond.

 

[Lynx]

It would be egress on br0, not ingress. Ingress would be traffic FROM the LAN. br0 Egress would be traffic TO the LAN from the Router or Internet.

I remain at a loss as to why samba download traffic is caught by the below (which I tend to think of as the 'download' aspect of cake):

qdisc cake 801a: dev br0 root refcnt 2 bandwidth 30720Kbit besteffort dual-dsthost nonat wash ingress no-ack-filter split-gso rtt 100ms noatm overhead 91

What does 'ingress' mean in the context of the command above?

 

[dave14305]

What does 'ingress' mean in the context of the command above?

It informs Cake that the traffic to be shaped has already traversed the bottleneck link (ISP), so treat it differently in terms of shaping. Once you’ve received the packet at your router it’s really too late to try to delay it on its journey since you’re not worried about inducing bufferbloat on your own LAN devices. It’s a different story when sending traffic towards the bottleneck (ISP) where you want to be sure not to overload the buffers out there (the alternative default value of egress applies in this case).

It’s discussed in some detail on the OpenWRT Wiki.

[OpenWrt Wiki] SQM Details

openwrt.org

 

[Lynx]

Since you raised the issue of terminology, how about the following.
'Ingress' from the perspective of CAKE is traffic from the Router or Internet, i.e. from the perspective of WAN traffic? That is ingress is in the direction of traffic from the ISP. Egress is in the direction of traffic to the ISP.
And so when you wrote:

It would be egress on br0, not ingress. Ingress would be traffic FROM the LAN. br0 Egress would be traffic TO the LAN from the Router or Internet.

This is from the perspective of LAN traffic.

Ingress vs Egress Network Traffic

What is ingress and Egress Network traffic?

www.safdar.com

So from a LAN perspective, samba share traffic destined for my local computer is egress from 'br0', but from the perspective of CAKE (in which ingress and egress is based on the WAN direction) this falls under the category of ingress because this is in the direction from ISP to my local computer?
This would seem consistent with the SQM wiki that you linked:

With the 'ingress' keyword, the above example for incoming packets from the internet would be nat dual-dsthost ingress

This note assumes that you have an Internet facing interface, usually eth0 and will call traffic leaving that interface TO the ISP egress traffic. Traffic received on that interface FROM the ISP is called ingress traffic. This interface is usually connected to an ISP's modem.

[OpenWrt Wiki] SQM Details

openwrt.org

And explains why the samba traffic is captured by:

qdisc cake 801a: dev br0 root refcnt 2 bandwidth 30720Kbit besteffort dual-dsthost nonat wash ingress no-ack-filter split-gso rtt 100ms noatm overhead 91

But this only explains the directionality aspect and addresses the WAN/LAN direction flip.

Say I hosted a samba share on another LAN device, would that also be picked up as 'ingress' from the perspective of CAKE? If not, why not? Why would a LAN share be treated differently than share from the router?