CakeQOS CakeQOS-Merlin

[randomName]

Still investigating my PL with a particular game. I did happen to change my tscsyncpolicy to enhanced and it seemed to help but that shouldn't be the solution when traditional Qos worked without PL so. I noticed that not setting a cap on my 50/3 connection returns values that are exactly 95% of my speeds according to spdMerlin

 Latency:    21.76 ms   (0.10 ms jitter)
   Download:    47.56 Mbps (data used: 48.0 MB)
     Upload:     2.83 Mbps (data used: 2.7 MB)
Packet Loss:     0.0%

My download general settings:

qdisc cake 8006: root refcnt 2 bandwidth 50Mbit besteffort triple-isolate nat w
 Sent 523942552 bytes 367546 pkt (dropped 366, overlimits 347168 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 228Kb of 4Mb
 capacity estimate: 50Mbit
 min/max network layer size:           46 /    1500
 min/max overhead-adjusted size:       64 /    1518
 average network hdr offset:           14

                  Tin 0
  thresh         50Mbit
  target          5.0ms
  interval      100.0ms
  pk_delay        6.2ms
  av_delay        205us
  sp_delay          0us
  backlog            0b
  pkts           367912
  bytes       524496676
  way_inds            1
  way_miss          395
  way_cols            0
  marks               0
  drops             366
  ack_drop            0
  sp_flows            0
  bk_flows            1
  un_flows            0
  max_len          1514
  quantum          1514

Upload general settings:

qdisc cake 8005: root refcnt 2 bandwidth 3Mbit besteffort triple-isolate nat no
 Sent 45390535 bytes 211360 pkt (dropped 1535, overlimits 135931 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 117312b of 4Mb
 capacity estimate: 3Mbit
 min/max network layer size:           28 /    1500
 min/max overhead-adjusted size:       64 /    1518
 average network hdr offset:           14

                  Tin 0
  thresh          3Mbit
  target          6.1ms
  interval      101.1ms
  pk_delay       16.7ms
  av_delay        5.3ms
  sp_delay         65us
  backlog            0b
  pkts           212895
  bytes        47710318
  way_inds           38
  way_miss          361
  way_cols            0
  marks               1
  drops            1535
  ack_drop            0
  sp_flows            0
  bk_flows            1
  un_flows            0
  max_len          1514
  quantum           300

qdisc ingress ffff: parent ffff:fff1 ----------------
 Sent 519515604 bytes 370016 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0

 

[heysoundude]

Still investigating my PL with a particular game. I did happen to change my tscsyncpolicy to enhanced and it seemed to help but that shouldn't be the solution when traditional Qos worked without PL so. I noticed that not setting a cap on my 50/3 connection returns values that are exactly 95% of my speeds according to spdMerlin

Latency:    21.76 ms   (0.10 ms jitter)
   Download:    47.56 Mbps (data used: 48.0 MB)
     Upload:     2.83 Mbps (data used: 2.7 MB)
Packet Loss:     0.0%

My download general settings:

qdisc cake 8006: root refcnt 2 bandwidth 50Mbit besteffort triple-isolate nat w
Sent 523942552 bytes 367546 pkt (dropped 366, overlimits 347168 requeues 0)
backlog 0b 0p requeues 0
memory used: 228Kb of 4Mb
capacity estimate: 50Mbit
min/max network layer size:           46 /    1500
min/max overhead-adjusted size:       64 /    1518
average network hdr offset:           14

                  Tin 0
  thresh         50Mbit
  target          5.0ms
  interval      100.0ms
  pk_delay        6.2ms
  av_delay        205us
  sp_delay          0us
  backlog            0b
  pkts           367912
  bytes       524496676
  way_inds            1
  way_miss          395
  way_cols            0
  marks               0
  drops             366
  ack_drop            0
  sp_flows            0
  bk_flows            1
  un_flows            0
  max_len          1514
  quantum          1514

Upload general settings:

qdisc cake 8005: root refcnt 2 bandwidth 3Mbit besteffort triple-isolate nat no
Sent 45390535 bytes 211360 pkt (dropped 1535, overlimits 135931 requeues 0)
backlog 0b 0p requeues 0
memory used: 117312b of 4Mb
capacity estimate: 3Mbit
min/max network layer size:           28 /    1500
min/max overhead-adjusted size:       64 /    1518
average network hdr offset:           14

                  Tin 0
  thresh          3Mbit
  target          6.1ms
  interval      101.1ms
  pk_delay       16.7ms
  av_delay        5.3ms
  sp_delay         65us
  backlog            0b
  pkts           212895
  bytes        47710318
  way_inds           38
  way_miss          361
  way_cols            0
  marks               1
  drops            1535
  ack_drop            0
  sp_flows            0
  bk_flows            1
  un_flows            0
  max_len          1514
  quantum           300

qdisc ingress ffff: parent ffff:fff1 ----------------
Sent 519515604 bytes 370016 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0

Your av_delay is 0.8ms below your target. sp_delay is 0.065ms (65us is MICROseconds)
is max_len the same as MTU? shorten that up (max overhead-adjusted size?) and see what what happens...try 1495-1498 maybe? is it a matter of crowding necessary system data out of the user data stream in the name of speed? greedy bites choking the system? IF Packet Loss is the system spitting out what it can't fit in its mouth/chew/swallow...take smaller bites more often, right? go to a shorter duty cycle, if that makes sense.

 

[randomName]

Your av_delay is 0.8ms below your target. sp_delay is 0.065ms (65us is MICROseconds)
is max_len the same as MTU? shorten that up (max overhead-adjusted size?) and see what what happens...try 1495-1498 maybe? is it a matter of crowding necessary system data out of the user data stream in the name of speed? greedy bites choking the system? IF Packet Loss is the system spitting out what it can't fit in its mouth/chew/swallow...take smaller bites more often, right? go to a shorter duty cycle, if that makes sense.

In Merlin, WAN MTU is 1500

How do I set that? I've removed docsis and tried overhead 4 mpu 64 noatm for both up and down and max_len still shows 1514.

 

[heysoundude]

In Merlin, WAN MTU is 1500

How do I set that? I've removed docsis and tried overhead 4 mpu 64 noatm for both up and down and max_len still shows 1514.

Advanced Settings: WAN/Account Settings for me (AC86, 50/10 vDSL, PPPoE on my router (bridged gateway)). I have both MTU and MRU

Do your homework on this before changing anything - when I looked into it, the wikipedia page was a big help to get clear on things - then talk to people who know their stuff in this dept at your ISP.
(I went down the Jumbo frames rabbit hole, which is what I seem to recall makes me aware of this MTU stuff (1514 or 1518 IS the max IIRC, so with 14-18 system overhead bits, 1500 could be right for the ROUTER...but that doesn't account for what might make Cake work best, and I figure there are maybe a dozen bits it needs to do what we're asking) You need to leave a few spaces for system/timing messages to be included in your packets, but when you do, even though the packets may contain "smaller payload", they come flying at you at their best speed => AS I UNDERSTAND THINGS.
This might be rather important "router config for optimal internet connection stuff," or I might well be barking up the wrong tree here, but in my head/logic, packets are getting dropped not because of a speed/software/firmware issue, but because of a slight mis-configuration of our routers/firmware/scripts we run....and remember, we have to allow for the slop of errors in the system. technology isn't perfect, just like humans, but we CAN work out how to make it function optimally

I hope that's correct and I'm making sense explaining it to you

TL;DR- if Merlin's MTU is 1500, your Cake's max_len is larger than that. try making cake's max_len to be 1498, 1495, 1490 even, until your packets stop getting dropped when you have cake running - those "missing bits" might just give enough room for ins/outs to continue smoothly without interruption, and lower the latency introduced when error correction kicks in...that's optimal QoS as far as I'm concerned. And there's gotta be a way to make it happen in cake's setup/config...find it, fix it and you'll be golden.

Maybe now some of the things here make more sense?

But wait -there's more: https://www.linksys.com/us/support-article?articleNum=134914

 

[randomName]

Hi thanks again for the reply

I ran ping www/google/com -f -l 1472 and results were good. So +28 then the max mtu is 1500. I tried 1473 and it fragmented.

Cake-qos doesn't understand max_len when adding it to the extra dl/up options

 

[heysoundude]

Hi thanks again for the reply

I ran ping www/google/com -f -l 1472 and results were good. So +28 then the max mtu is 1500. I tried 1473 and it fragmented.

Cake-qos doesn't understand max_len when adding it to the extra dl/up options

I think you missed some things in the first link in that reply. There is plenty of wiggle room in overhead compensation; the trick is to test and select the correct method for YOU.
I suspect you'll want to try raw, which may resolve the PL issue. (my insufficient coffee consumption so far today may be skewing my judgement, so tread carefully)

 

[Morris]

Hi,

For those of you that don't know me I'm a retired network engineer and manager and a long time user of Robert Merlin's great code. Thank you for all the effort getting cake-qos working. I just installed it on my RT-86U. I had one bump along the way when the install script asked me to enter additional download and upload options. I entered ? and it was accepted yet cake wold not start. I did more reading and figured out that "ethernet" is valid for my setup (Verizon FIOS). I tried to changed the option and that failed. Simple fix, uninstall and reinstall and I'm up and running. I chose diffserv4 for queue priority as it's most similar to my Cisco training for a network with voice and video on the network. I've beat on my network with large downloads at the same time as a few 4k video streams and experienced no stuttering or buffering delays and good utilization of the link staying close to 100%.

I've run Asus Adaptive QOS which lacks in classifying traffic. I progressed to Fresh_Jr_QOS and later Flex_QOS both of which help yet they rely on the foundation of Adaptiv_ QOS. I became tied of trying to make up for Asus not calcifying common traffic properly such as Youtube when run under Chrome Browser and this is what lead me to trying Cake. I'll be monitoring and trying to break cake more yet now that I have it set up I expect that other than normal updates there will be little to do.

Suggestion: In the install script, replace the free form additional options with a menu. As there are probably only a few of the options that make sense for most people the menu could be limited and have an option for other (free form entry).

Thank you for all the great work,

Morris

 

[heysoundude]

Hi,

For those of you that don't know me I'm a retired network engineer and manager and a long time user of Robert Merlin's great code. Thank you for all the effort getting cake-qos working. I just installed it on my RT-86U. I had one bump along the way when the install script asked me to enter additional download and upload options. I entered ? and it was accepted yet cake wold not start. I did more reading and figured out that "ethernet" is valid for my setup (Verizon FIOS). I tried to changed the option and that failed. Simple fix, uninstall and reinstall and I'm up and running. I chose diffserv4 for queue priority as it's most similar to my Cisco training for a network with voice and video on the network. I've beat on my network with large downloads at the same time as a few 4k video streams and experienced no stuttering or buffering delays and good utilization of the link staying close to 100%.

I've run Asus Adaptive QOS which lacks in classifying traffic. I progressed to Fresh_Jr_QOS and later Flex_QOS both of which help yet they rely on the foundation of Adaptiv_ QOS. I became tied of trying to make up for Asus not calcifying common traffic properly such as Youtube when run under Chrome Browser and this is what lead me to trying Cake. I'll be monitoring and trying to break cake more yet now that I have it set up I expect that other than normal updates there will be little to do.

Suggestion: In the install script, replace the free form additional options with a menu. As there are probably only a few of the options that make sense for most people the menu could be limited and have an option for other (free form entry).

Thank you for all the great work,

Morris

Thanks for popping by and offering your endorsement, support and encouragement. Please stop by more often to offer your experience and knowledge!

 

[heysoundude]

Is there anyone following along here familiar with the rtt parameter?
None of this is germane for people who have set besteffort, it seems, but if youre a diffservX-er like me, here's something possibly to have a gander/tweak at:

I'm questioning if the 100ms default is the best way to go when my ping, jitter and unbound numbers summed average below 100ms.
the next step down is "regional" at around 30ms....and that seems closer for me (my average being 50-60ms...ish. the unbound classification bins are a bit ridiculous IMO).
There's a sentence on the cake manpage that indicates (if I'm reading it correctly) that for the fairness logic to work best, sparse flows reduce throughput through the "tins" and there needs to be a bit of what they call "back pressure".

Or is this all counterproductive to having a fast and stable WAN connection for all network devices and don't poke the bear because it's all beyond me how it really works? lol.

 

[Morris]

Is there anyone following along here familiar with the rtt parameter?
None of this is germane for people who have set besteffort, it seems, but if youre a diffservX-er like me, here's something possibly to have a gander/tweak at:

I'm questioning if the 100ms default is the best way to go when my ping, jitter and unbound numbers summed average below 100ms.
the next step down is "regional" at around 30ms....and that seems closer for me (my average being 50-60ms...ish. the unbound classification bins are a bit ridiculous IMO).
There's a sentence on the cake manpage that indicates (if I'm reading it correctly) that for the fairness logic to work best, sparse flows reduce throughput through the "tins" and there needs to be a bit of what they call "back pressure".

Or is this all counterproductive to having a fast and stable WAN connection for all network devices and don't poke the bear because it's all beyond me how it really works? lol.

I suspect that's just a starting point and cake will self tune. I have not set the distance and it's working fine for me. The documentation for cake dose not make setting it up cake. If you have the background to understand this video

You can have a bit more confidence in what you are doing.

Besteffort is designed to reduce latency and thus make real time applications such as gaming much much more responsive. The video goes into why.

 

[Wade Coxon]

I chose diffserv4 for queue priority as it's most similar to my Cisco training for a network with voice and video on the network. I've beat on my network with large downloads at the same time as a few 4k video streams and experienced no stuttering or buffering delays and good utilization of the link staying close to 100%.

If you run "cake-qos status download", you will see that cake doesn't currently identify enough traffic types to make any of the diffserv settings effective. The default "besteffort" is currently the most sensible option until we have a mechanism for tagging traffic.

 

[Morris]

If you run "cake-qos status download", you will see that cake doesn't currently identify enough traffic types to make any of the diffserv settings effective. The default "besteffort" is currently the most sensible option until we have a mechanism for tagging traffic.

Thank you Wade,

please enplane the nice decision of traffic in my stats:

qdisc cake 8016: root refcnt 2 bandwidth 105Mbit diffserv4 triple-isolate nat wash ingress no-ack-filter split-gso rtt 100.0ms noatm overhead 38 mpu 84 no-sce
Sent 31437495395 bytes 22294992 pkt (dropped 9482, overlimits 19103262 requeues 0)
backlog 0b 0p requeues 0
memory used: 1251136b of 5250000b
capacity estimate: 105Mbit
min/max network layer size: 46 / 1500
min/max overhead-adjusted size: 84 / 1538
average network hdr offset: 14

Bulk Best Effort Video Voice
thresh 6562Kbit 105Mbit 52500Kbit 26250Kbit
target 5.0ms 5.0ms 5.0ms 5.0ms
interval 100.0ms 100.0ms 100.0ms 100.0ms
pk_delay 9.9ms 16.8ms 6.2ms 13.4ms
av_delay 2.4ms 4.9ms 637us 1.0ms
sp_delay 13us 26us 5us 23us
backlog 0b 0b 0b 0b
pkts 26112 21916118 53301 308943
bytes 35303397 31145795769 54870398 215649580
way_inds 12 1093172 2690 1433
way_miss 247 48660 1047 5968
way_cols 0 0 0 0
marks 0 0 0 0
drops 24 9273 37 148
ack_drop 0 0 0 0
sp_flows 1 5 0 1
bk_flows 0 1 0 0
un_flows 0 0 0 0
max_len 1514 1514 1514 1484
quantum 300 1514 1514 801


#########################################################

 

[Wade Coxon]

Thank you Wade,

please enplane the nice decision of traffic in my stats:

Yep, that's fairly typical. The small amount of traffic that seems to be coming in as tagged at the source is being split between the bins.

You have 99% coming in on the besteffort bucket at the moment, with 1% Voice. The Bulk and Video are trace amounts.

That's very similar to what I observe on my connection, and my traffic activity should definitely be reflecting different proportions to that.

Cake needs something similar to the Asus-default Trendmicro data categorisation engine to sort traffic into the bins properly.
I posted some links to similar projects on the OpenWrt community that are using iptables to do that job, but alas my own crude efforts to copy their scripts were unsuccessful. Someone smarter than me can have a crack though .

 

[Morris]

Yep, that's fairly typical. The small amount of traffic that seems to be coming in as tagged at the source is being split between the bins.

You have 99% coming in on the besteffort bucket at the moment, with 1% Voice. The Bulk and Video are trace amounts.

That's very similar to what I observe on my connection, and my traffic activity should definitely be reflecting different proportions to that.

Cake needs something similar to the Asus-default Trendmicro data categorisation engine to sort traffic into the bins properly.
I posted some links to similar projects on the OpenWrt community that are using iptables to do that job, but alas my own crude efforts to copy their scripts were unsuccessful. Someone smarter than me can have a crack though .

Thank you Wade,

I changed to besteffort. I was considering trying this after watching this video which so I now understand the basics of how it works

Morris

 

[heysoundude]

I suspect that's just a starting point and cake will self tune. I have not set the distance and it's working fine for me. The documentation for cake dose not make setting it up cake. If you have the background to understand this video

You can have a bit more confidence in what you are doing.

Besteffort is designed to reduce latency and thus make real time applications such as gaming much much more responsive. The video goes into why.

Awesome video - thanks for finding it! besteffort it is, then...
(It's nice to know I was on the right track with an earlier reply to someone else here in the thread - I understand things better than I gave myself credit for)
https://www.networkworld.com/article/2176641/are-your-pipes-too-big-.html might explain it better than I did there

I suspect rolling up my sleeves and digging into the rtt (distance, as you call it...which I love) may still bring some benefit to me. I'll let everyone know how it turns out, either way.

 

[brummygit]

Yep, that's fairly typical. The small amount of traffic that seems to be coming in as tagged at the source is being split between the bins.

You have 99% coming in on the besteffort bucket at the moment, with 1% Voice. The Bulk and Video are trace amounts.

That's very similar to what I observe on my connection, and my traffic activity should definitely be reflecting different proportions to that.

Cake needs something similar to the Asus-default Trendmicro data categorisation engine to sort traffic into the bins properly.
I posted some links to similar projects on the OpenWrt community that are using iptables to do that job, but alas my own crude efforts to copy their scripts were unsuccessful. Someone smarter than me can have a crack though .

I've noticed that when running diffserv4 that my Ring Doorbell Pro must mark it's traffic, particularly when I use Live View, as I get my upload traffic incrementing the Voice and Video tin counters. The Ring Doorbell is very fussy (aka rubbish) about it's connection so this helps me.

If you want to use diffserv4 on the download you need to set the extra download option nowash as the default on cake-qos is to wash the DSCP marking.

I would also love to see additional traffic marking added to cake-qos, but also don't have the knowledge to do this. I have a vision that we could use the Trend Micro engine classifications just like with FlexQoS and apply these to cake as per this info https://man7.org/linux/man-pages/man8/tc-cake.8.html#OVERRIDING_CLASSIFICATION_WITH_TC_FILTERS, but sadly even with 30 years in IT Infrastructure, I don't have the knowledge to even know where to start in the code.

 

[Morris]

Awesome video - thanks for finding it! besteffort it is, then...
(It's nice to know I was on the right track with an earlier reply to someone else here in the thread - I understand things better than I gave myself credit for)
https://www.networkworld.com/article/2176641/are-your-pipes-too-big-.html might explain it better than I did there

I suspect rolling up my sleeves and digging into the rtt (distance, as you call it...which I love) may still bring some benefit to me. I'll let everyone know how it turns out, either way.

Interesting article and nice to see Network World is still alive. I had a letter to the editor published there about 25 years ago where I talked about how to fund a DIY lab in a corporate environment and reducing your maintenance cost at the same time. If you maintain spare parts by getting one of each switch and router in your network then you can use them in the lab and can get a support contract that dose ships the parts by land rather than next day air. This pays back in about a year allowing for further growth of the lab. Not much help for those of us doing home networking...

Morris

 

[Morris]

I've noticed that when running diffserv4 that my Ring Doorbell Pro must mark it's traffic, particularly when I use Live View, as I get my upload traffic incrementing the Voice and Video tin counters. The Ring Doorbell is very fussy (aka rubbish) about it's connection so this helps me.

If you want to use diffserv4 on the download you need to set the extra download option nowash as the default on cake-qos is to wash the DSCP marking.

I would also love to see additional traffic marking added to cake-qos, but also don't have the knowledge to do this. I have a vision that we could use the Trend Micro engine classifications just like with FlexQoS and apply these to cake as per this info https://man7.org/linux/man-pages/man8/tc-cake.8.html#OVERRIDING_CLASSIFICATION_WITH_TC_FILTERS, but sadly even with 30 years in IT Infrastructure, I don't have the knowledge to even know where to start in the code.

Classifying traffic is a never ending project that will exhaust you. I was so happy to see cake as I had been running fresh/flex and playing that game. Cake's use of stream timing as described in the video I posted is brilliant. It allows for simplicity, a method I've learned to love.

Morris

 

[brummygit]

Classifying traffic is a never ending project that will exhaust you. I was so happy to see cake as I had been running fresh/flex and playing that game. Cake's use of stream timing as described in the video I posted is brilliant. It allows for simplicity, a method I've learned to love.

Morris

I agree that in normal QoS situations classification is an endless task. If we had the Trend Micro engine which identifies most traffic, cake is so good with it's fair queuing that the traffic slipping through the net wouldn't be a huge issue while Trend catches up.

I started to watch the video but need to find enough time to watch it carefully end to end. However where cake is not effective is in the cases where bandwidth is oversubscribed and therefore prioritisation is needed rather than fairness. Just catching the big downloads and reducing their priority while treating video & voice higher would be enough in my own (and I assume many others) situation.

 

[Morris]

I agree that in normal QoS situations classification is an endless task. If we had the Trend Micro engine which identifies most traffic, cake is so good with it's fair queuing that the traffic slipping through the net wouldn't be a huge issue while Trend catches up.

I started to watch the video but need to find enough time to watch it carefully end to end. However where cake is not effective is in the cases where bandwidth is oversubscribed and therefore prioritisation is needed rather than fairness. Just catching the big downloads and reducing their priority while treating video & voice higher would be enough in my own (and I assume many others) situation.

Cake is not using fairness. It classify traffic by traffic pattern. Real time streams such as Voice, Video and Gaming don't consume all the bandwidth, they send traffic at set rates while a download can grow to fill the pipe. Cake will slow the applications that fill the pipe to allow the real time applications the bandwidth they need. It is explained in the video.

I just tested this on my 100-Mb symmetric FIOS link by running 4 4K videos, a 1400 byte ping and then added some downloads. I was surprised I could fit 4 4k videos yet they were working so I started task manager to watch interface utilization and was surprised to see it below at about 75-Mb. When I pegged the link at 100-Mb cake kicked in and after one of the videos experienced a buffer spinning wheel for a moment and there was one dropped packet everything started working and utilization was at 98-Mb. I might need to drop the bandwidth limits on my link a bit. It's very unusual for my network to be this busy so I'll leave it till I see real world issues rather than ones I create.

Morris