[Release] FreshJR Adaptive QOS (Improvements / Custom Rules / and Inner workings)

[HowIFix]

Thanks, I added the new VPN rules and now it works without problems! (QOS + VPN Client fix reverse)

Aug 12 23:51:01 adaptive QOS: Applying - Iptable Down Rules
Aug 12 23:51:02 adaptive QOS: Applying - Iptable Up   Rules (eth0)
Aug 12 23:51:02 adaptive QOS: TC Modification Delayed Start (5min)
Aug 12 23:56:04 adaptive QOS: Applying  TC Down Rules
Aug 12 23:56:04 adaptive QOS: Applying  TC Up   Rules
Aug 12 23:56:04 adaptive QOS: Modifying TC Class Rates

Download rules:

Upload rules:

My rules in VPN Client:

The rule with the IP 192.168.2.25, I only use it when I play on the computer, I add that static IP and when I stop playing, I change the static IP in the computer.

Other:

Spoiler: About 192.168.2.8/30

192.168.2.8/30 is IP range, it's like adding rules for 192.168.2.8 to 192.168.2.11

• Static IP in DHCP
• 1 Xbox One: 192.168.2.8
• 2 Xbox One: 192.168.2.9
• Play Station 4: 192.168.2.10
• Nintendo: 192.168.2.11

http://www.subnet-calculator.com/cidr.php

 

[shelleyevans]

@shelleyevans

Just to confirm, you have tried executing the curl command within the ssh session?

The curl command should have been executed locally on the router inside the ssh session, with zero reliance of having curl on your Mac .

Very sorry for introducing confusion with my lack of programming awareness. My early assumption was that I needed curl on the Mac, but I can see now that it's all inside the router. In spite of that, I did execute the curl command within the ssh session. I SSH'd into the router, and then cut and pasted exactly this:

curl "https://raw.githubusercontent.com/FreshPr/FreshJR_QOS/master/FreshJR_QOS.sh" -o /jffs/scripts/FreshJR_QOS --create-dirs && sh /jffs/scripts/FreshJR_QOS -install

EDIT: because I am curious and somewhat impatient, I uninstalled the script and tried again. Ran into the same problem. It's definitely not working on my setup. As I mentioned, I am running AiMesh, although I can't really see how that would muck it up. My firmware version is 3.0.0.4.384_21140. I'm attaching a screenshot of my terminal window, just in case somebody spots something I am doing wrong.

 

[skeal]

Very sorry for introducing confusion with my lack of programming awareness. My early assumption was that I needed curl on the Mac, but I can see now that it's all inside the router. In spite of that, I did execute the curl command within the ssh session. I SSH'd into the router, and then cut and pasted exactly this:

curl "https://raw.githubusercontent.com/FreshPr/FreshJR_QOS/master/FreshJR_QOS.sh" -o /jffs/scripts/FreshJR_QOS --create-dirs && sh /jffs/scripts/FreshJR_QOS -install

EDIT: because I am curious and somewhat impatient, I uninstalled the script and tried again. Ran into the same problem. It's definitely not working on my setup. As I mentioned, I am running AiMesh, although I can't really see how that would muck it up. My firmware version is 3.0.0.4.384_21140. I'm attaching a screenshot of my terminal window, just in case somebody spots something I am doing wrong.
View attachment 14066

It would appear that "curl" is not installed on the stock firmware. This is an issue @FreshJR is looking into. The install script looks to not work with the stock version.

 

[EnF70]

Is there a specific mark to exclude some traffic from QoS?
Having issues with LAG traffic on the LAN being restricted by QoS.

 

[Tekneek]

Is there a specific mark to exclude some traffic from QoS?
Having issues with LAG traffic on the LAN being restricted by QoS.

I would imagine that Net control packets always have the highest priority, so you might look to utilize those? I would wield that power very carefully, though.

 

[FreshJR]

Is there a specific mark to exclude some traffic from QoS?
Having issues with LAG traffic on the LAN being restricted by QoS.

Can you explain which LAN traffic is being picked up by QOS.

 

[FreshJR]

I'm attaching a screenshot of my terminal window, just in case somebody spots something I am doing wrong.

You did everything correctly. It seems curl is bundled with the stock firmware on an “as-needed” basis. I assumed it was present on all Asus firmware, but your router is one of the many ones that doesn't have it.

Scp is a tool to transfer files between devices.
Curl is similar, but it transfers file from a website and onto a device.

Scp is executed on the computer. Curl is executed on the remote device/router itself.

Curl had an ease of installation bonus since it created missing directories on the destination device, unlike scp.

There will be no alternate “curl” command syntax update for stock firmware as there is nothing to update since the required executable does not exist.

@shelleyevans , the first post instructions have been updated to reflect your findings.

Glad you got it figured out and learned some tools in the process.

In any case, the scp method is superior since it is required if doing custom rules. The scp method was originally the only method explained since I didn't find it complicated and it was more versatile in regards to modifications.

Curl simply was a workaround for the installation complaints I received, but I do not like it since robs users the ability to see inside the script.

 

[EnF70]

Can you explain which LAN traffic is being picked up by QOS.

I had LAG (link aggregation) enabled on 2 LAN ports using the bonding driver but I stopped using it.
Apart from issues with QoS there are also LAN traffic performance issues associated so it's not worth it.
Back to the normal setup now, thanks for your script, playing with it at the moment.

 

[TicoMan]

@HowIFix
Wrong variable used in upload rules. Use
${Downloads_mark_up}

@TicoMan
Packet drop occurs after queues fill up.
Queues fill up if the tcp connection is too slow to negotiate down to your max speed.
This can happen if your bandwidth limit is set too low from your actual atainable speeds.
Overall, the big picture is that your connection has to negotiate underneath set limits at a rate faster than the packet queue fills up to get no packet loss.



Fq-codel is amazing at queue management.

1) It drops invalid timed out packets in the middle of the queue which opens more space within the queue for fresh packets. This effectively makes the queue larger

2) Since the toxic packets are ignored faster, tcp will renegotiate the connection rate to set limits faster which is great since we are at a race to prevent to queue from filling up in order to get no drop.

Sfq not so much.

In contrast to the above, sfq keeps timed out packets, and drops the new packets since they can’t fit into the queue. Not only will the entire queue turn toxic while tcp renegotiates down, new packets are dropped since there’s is no room for them.

This is especially severe with sfq when your bandwidth limits are too low for your connection since negotiating will take even longer!

This would be seen as a poor quality grade but Dslreports reported your pocket drop/quality as an A+ so I don’t see the issue you are mentioning.

An improvement would be to switch to fq-codel, but unfortunately, only sfq is present on stock firmware. In the end, it is what it is, since that is how QOS works under the hood.

If you are curious as to the differences check out the sfq vs fq-codec animated graphic in the first post.

You can see faster nogotiation rate to max speed && higher quality with fq-codel compared to sfq with the same exact settings.

Cake is even better from reports on the net, but we don’t have that at our disposal.



Custom rules are more to get traffic into the right category while prioritization is the device priority within the category.

It is nice that it was set to highest automatically (are you sure you didn’t assign this manually) but I still am not sure if the default rules would place the traffic correctly into the VoIP category without testing it.

Anyway as a failsafe, there are default rules in the script that route most VoIP traffic into VoIP, even if it is redundant.

@FreshJR Thanks for taking the time to explain me!

So I just installed lastest Merlin firmware +yourScript. I enabled QoS on adaptive mode, with manual bandwidth, using sfq or fq-codel I alway get between 4%-8% packet drop . From Netalyzr results I see this :"All the packet loss appears to have occurred on the path from our server to your computer."

Could it be an issue with my ISP? how could I prove that?

Also, I have been having Skype quality issues when working from home. I made a test call using my cell phone and noticed that the call's traffic is being categorized as General. Is that ok? What change(s) can I apply to your rules to correct it?

 

[Sinner]

Regarding skype, i havent tested it but most others ive found to be routed as unidentified and not ness voip. Fresh does have a line in there for it putting it someplace. Check the script and make sure your categorys are where they need to be for that. Id have the akype container high if its important to you.

Regarding the packet loss, im no expert on this but i believe dropping packets is a function of qos. It would be my guess at the start of the teat when the speed ramps up it would do its thing dropping some to reduce the speed and bloat and then level off. You may want to try a much larger test with much more data. If that % of loss starts lowering on larger tests id say my theory is correst if not then well it may be another issue.

 

[Pit_g]

/jffs/scripts/FreshJR_QOS -debug


Aug 15 10:03:24 dropbear[12556]: Child connection from xxx.xx.xx..xxx:xxxxx
Aug 15 10:03:32 dropbear[12556]: Password auth succeeded for 'XXXXX' from xxx.xx.xx.xxx:xxxxx
Aug 15 10:04:05 adaptive QOS: Undf Prio: 2
Aug 15 10:04:05 adaptive QOS: Undf FlowID: 1:12
Aug 15 10:04:05 adaptive QOS: Classes Present: 8
Aug 15 10:04:05 adaptive QOS: Down Band: 465920
Aug 15 10:04:05 adaptive QOS: Up Band : 465920
Aug 15 10:04:05 adaptive QOS: ***********
Aug 15 10:04:05 adaptive QOS: Net = 1:10
Aug 15 10:04:05 adaptive QOS: VOIP = 1:13
Aug 15 10:04:05 adaptive QOS: Gaming = 1:11
Aug 15 10:04:05 adaptive QOS: Others = 1:12
Aug 15 10:04:05 adaptive QOS: Web = 1:15
Aug 15 10:04:05 adaptive QOS: Streaming = 1:14
Aug 15 10:04:05 adaptive QOS: Downloads = 1:16
Aug 15 10:04:05 adaptive QOS: Defaults = 1:17
Aug 15 10:04:05 adaptive QOS: ***********
Aug 15 10:04:05 adaptive QOS: Downrates -- 23296, 93184, 69888, 69888, 116480, 46592, 23296, 23296
Aug 15 10:04:05 adaptive QOS: Downbursts -- 28793b, 115175b, 57587b, 28793b, 22378b, 15986b, 11193b, 4798b
Aug 15 10:04:06 adaptive QOS: DownCbursts -- 582400b, 582400b, 582400b, 582400b, 582400b, 582400b, 582400b, 582400b
Aug 15 10:04:06 adaptive QOS: ***********
Aug 15 10:04:06 adaptive QOS: Uprates -- 23296, 93184, 116480, 69888, 69888, 46592, 23296, 23296
Aug 15 10:04:06 adaptive QOS: Upbursts -- 28793b, 115175b, 57570b, 28793b, 22381b, 15986b, 11193b, 4798b
Aug 15 10:04:06 adaptive QOS: UpCbursts -- 582400b, 582400b, 582400b, 582400b, 582400b, 582400b, 582400b, 582400b
Aug 15 10:04:28 dropbear[12556]: Exit (XXXXX): Exited normally


Looks fine ????'

 

[shelleyevans]

Glad you got it figured out and learned some tools in the process.

Learned a TON, with many thanks to the many patient helpers on this forum. Script now reinstalled and running beautifully, fingers crossed. @skeal spent LOTS of time in a side-bar conversation helping me tweak the script, so he gets credit for keeping me away from the ledge.

In case others without experience are struggling with QOS, the short version of my long struggle is that it really does take a lot of testing. Set your classes the way that you want them (in my case prioritizing VOIP) and then prepare to spend a LONG time (in my case several hours over two days) changing the upload/download numbers and running the speedtest until bufferbloat is fixed. I did close to a hundred tests. First I played with my upload numbers, watching the pings on the test. Interestingly, counter to popular wisdom, the closer I got to my actual upload number, the lower the pings got, and I ended up setting my upload at the actual number, 42. (This kind of makes sense to me because that number is rock solid-- over hundreds of tests.) Then I tried reducing and raising my download numbers, and NOTHING budged. A "B" for bufferbloat every time. I was about to give up and call it a day when I decided to try one last test, cutting my download speeds in half-- the download number is pretty erratic in my speed tests, both the average download, but also, as I watched the tests, jumping from 900-500-600 inside the test itself. I entered 500 as my download maximum and voila! Suddenly I have my old scores back! Hard work, but totally worth it.

 

[FreshJR]

From Netalyzr results I see this :"All the packet loss appears to have occurred on the path from our server to your computer."

Noticed that the call's traffic is being categorized as General. Is that ok? What change(s) can I apply to your rules to correct it?

When I tried netalyzr I got zero packet loss, but it did say that I have to watch out for bufferbloat on my upload, which is an erroneous result.

I have no bloat on the upload. I confirmed with dslreports, continuous pings via command prompt, and pingplotter.

With that, I am a little skeptical of the results form netalyzr.

Try

 ping google.com -n 100

and see if you get the same loss.

Pingotter will give a nice timeline view to see if the issue is intermittant.

General means that it is unidentified via QOS. With my script it will be directed into the “others” category and if you have the recommended category order in place, then Skype should be alright even though it’s unidentified.

Not much you can to get it into VOIP since Skype doesn’t have a set port it uses.


What modem do you have. Some Intel chipset modems have big issues with bloat and packet loss.

 

[TicoMan]

Try

 ping google.com -n 100

So these are the results I got:

As for my modem, it is a Arris DG1670 in BridgeMode.

Categories are: 1.VoIp 2. Gaming 3. Video and Audio 4.Others 5. WebSurfing 6.FileTransferring

FreshJR_QOS -debug shows this:

 

[FreshJR]

Yup that’s packet loss (red vertical strips), most likely due to your modem.

https://www.speedguide.net/routers/arris-touchstone-dg1670-docsis-30-dual-band-concurrent-3412

Look up Intel puma6 on google.

Never buy any Intel modem.
Not only should puma6 should be discontinued, even their cellular line inside the iPhones is trash.

http://www.dslreports.com/tools/puma6

Terrible modems all around.

Also the script modifications are not applied in your screenshot. If you are running stock firmware then most WebUI modifications knock the settings loose until a daily check at 3:30am or a system reboot. You can tell the modifications are not in effect since undf prio = 1:17 as stated in the original post. Basically it only works on stock firmware in a continuous manor once you stop messing with the router. RMerlin install doesn’t have this issue.

Also for Skype traffic you don’t really want streaming above others as explained in the first post!

 

[Pit_g]

Nice!

 

[shelleyevans]

Hello all! Not wanting to hijack this excellent thread with beginner level discussion about QoS, I started another thread with the discoveries that I made while troubleshooting QoS on my AiMesh system, and the few questions that remain for me. I haven't gotten any answers, so I thought I would post the link here: QoS Mysteries. If any experts have free time, and would be willing to take a look, I'd appreciate it. No worries if not-- perhaps this is a sign that I should leave well enough alone!

 

[FreshJR]

Not wanting to hijack this excellent thread

Feel free, this thread is open to all QOS related questions, even if I do not have a concrete answer myself!

 

[Xruptor]

General question, if I wanted to prioritize a particular IP Address or category for example Games above all else. How would I go about doing either of those?

 

[FreshJR]

General question, if I wanted to prioritize a particular IP Address or category for example Games above all else. How would I go about doing either of those?

Priority is present as

1) Traffic Type Priority
2) Device Priority

Asus first evaluates priority by "Traffic Category".
It then goes through the process again within each traffic category to evaluate "Device Priority" INSIDE each container.

Traffic Priority is assigned via "Adaptive QOS" in webUI under the "Customize List" option.
Device Priority is assigned via "Highest, High, Medium, Default, Low" via "Bandwidth Monitor"

You can drag "Games" where you please, and the default priority recommendation already has it high up in the list.
You can set a specific device to "Highest" but I DO NOT recommend using device priority at all as the Asus implementation is poor.

As a work around if you really want a device to get highest priorty, but with sane bandwidth allocation in regards to other devices, you can make a custom rule to put all traffic from a particular IP into "Net Control" which is the top container of "Traffic Priority". (This is a hacky workaround, but that device will indeed have highest priority, with better results than the alternative of Asus built in device priority.)

I was supposed to fix the cascaded device priority in the past, but changes of Asus implementation between updates had me discontinue that goal.

Read the 1rst post entirely, and you should get a good understanding of how it actually works under the hood.