[Release] FreshJR Adaptive QOS (Improvements / Custom Rules / and Inner workings)

[FreshJR]

Question regarding the wan overhead value for the following setup: Fiber router bridged with vlan to my Asus router and the internet connection has been configured from the Asus router as PPPoE with ISP logins credentials. Currently I have chosen the option Ethernet Vlan which is 4. Should I keep it this way or should I try another value for more optimized performance?

It's hard to say. This is an advanced value that is really important for the ISP to account for themselves when managing their bandwidth links.

But actually on the user side it all depends. They may whitelist that overhead traffic and not count it against you, or they may count it against you but their actual internal overhead may be different from the preset templates.

A speedtest can be designed to determine what (and if) WAN overhead a particular ISP is actually limiting your connection against client side.

I have no desire to create such a speedtest website as I do not have a sever to host it, nor do I have the time to undertake the project. (The data has actually transverse through the WAN port/internet to calculate your ISP overhead value and speedtests are a bandwidth intensive tool to host publicly, so this is a waste of time).

Dslreports has a very nice set of network tools (even tools to check modem reliability in terms of poor performing puma6 chipsets). Maybe they have something available or would be willing to cook something up if enough users request it on their forums. I hold their services in really high regard.

For example,

You need speedtest results from transfering full packet sizes (1460 bytes payload). Then you need a second speedtest with smaller packets, let's say (400 bytes payload). With some very simple math, from the differences in throughput you can calculate per packet overhead.

I have never tested for any WAN overhead so I don't have any history for reference.

If you have a high bandwidth server host connection available, I can quickly cook up the math formula and provide instructions to set a server and change its MTU (working packet size).

EDIT:
In all honesty, besides having OCD and needing to figure out the exact value, free to overshoot on the WAN overhead present by using the template present and then find a new number in the 85%-95% range that gives best network results.

A WAN overhead of 20bytes is like reducing your fixed bandwidth by 1.3%.
So it's easy to hone in on your ideal number again by increasing it slightly from the ideal value you found @ 0 overhead.

Now that you are back to ideal equivalent performance with a WAN overhead set, the WAN overhead will then dynamically throttle throughout a little bit more WHILE & IF your network is pushing many small packets.

But in reality, the varying difference in the QOS limit while dealing with small packets isn't too excessive in terms of your entire bandwidth.

As an example, when dealing with many small packets (lets say 380 bytes instead of 1500 bytes) pushing data at your ISP's limit, a WAN overhead value of 20 will dynamically reduce your bandwidth by an additional 3.4% from the limit present compared to dealing with only large packets.

That 3.4% is one of the worser case scenarios.
In reality, small packets are rarely able to consume your entire available bandwidth. As such, the WAN overhead vary the inputted limit around +- 1% depending what the network is doing.

As home connections have HIGHER and HIGHER bandwidths, small packet traffic continues to accounting for less and less of your entire bandwidth link. The smaller portion of your bandwidth that these packets become makes them for be accounted for and remain in tolerance with the crudely limited 85-95% QOS value.

The WAN overhead setting is really meant for ISP's to be able to set and QOS the bandwidth available on the cable at let's say 1000mbps, and then with the overhead parameter will make the the hub be dynamically police the rates on the hub itself dynamically vary between 900-990mbps depending on indiviudal packet sizes present at a given point in time.

This WAN overhead makes sure that afterencoding/encapsulating the data of different packet sizes, the transmission will never exceed 1000mbps on the cable. The hub will rarely have to drop down to 900mbps as 100% small packet traffic is unlikely.

For us though, or experienced range due t dma packets shouldn't be that drastic, let's say 1%. We already reduced our bandwidth to (85-95%) and our 1% dynamic packet size variance should essentially fit within tolerance.

TLDR: You can overestimate WAN overhead and the net effect will be a drop in throughput when dealing with many small packets. This drop should not be very significant.

 

[Sinner]

I think ive discovered a bug @FreshJR with the "compatible" version. I noticed when I just came home and the fam here fired up the xbox a considerable amount of data in the "default" container. I checked my logs and noticed the usual IOCtl errors at 2am. the errors are normal but suggested a 2am restart of qos instead of 3am. after further digging I found this:

2.062 Updated : 2018/03/30 02:00

the trend micro engine updated at 2am causing the restart but... there were no adaptive qos msgs.. suggesting it did not even try re-running the script..

 

[FreshJR]

I think ive discovered a bug @FreshJR with the "compatible" version. I noticed when I just came home and the fam here fired up the xbox a considerable amount of data in the "default" container. I checked my logs and noticed the usual IOCtl errors at 2am. the errors are normal but suggested a 2am restart of qos instead of 3am. after further digging I found this:

2.062 Updated : 2018/03/30 02:00

the trend micro engine updated at 2am causing the restart but... there were no adaptive qos msgs.. suggesting it did not even try re-running the script..

script should of been scheduled to re-run at 3am.
Did it pass 3am?

Show me the output of

cru l

it's 2hrs to 3am here, I will see install the compat version and see what happens on my end at that time.
[ I can trigger a manual check right now but I will let cron execute it like it would in reality]

 

[Vexira]

Question regarding the wan overhead value for the following setup: Fiber router bridged with vlan to my Asus router and the internet connection has been configured from the Asus router as PPPoE with ISP logins credentials. Currently I have chosen the option Ethernet Vlan which is 4. Should I keep it this way or should I try another value for more optimized performance?

You could try the ppoe value, but as far as I know fiber has no overhead, the only over head if it's ppoe I think it's 8 and the vlan is 4 I could be wrong so in total 12 is your over head if in right just give it a try I can look it up again on the lede forum and link you to the thread.

 

[FreshJR]

I cannot confirm or deny if there is a problem with the compatible version of the script and the scheduled cron task.

My system clock says 3AM, but cron thinks its 8AM. For some reason it is using the GMT timezone.

This means that it would actually be triggered at 10PM instead of 3AM.

I will report this to RMerlin.

I do not think there is an issue with the script.

 

[Sinner]

I cannot confirm or deny if there is a problem with the compatible version of the script and the scheduled cron task.

My system clock says 3AM, but cron thinks its 8AM. For some reason it is using the GMT timezone.

This means that it would actually be triggered at 10PM instead of 3AM.

I will report this to RMerlin.

I do not think there is an issue with the script.

Morning.. it was 6pm already long past 3am. Im gonna look at my cron.. just woke up bare with me here.

 

[Sinner]

@FreshJR this morning its approx 6:30 amd my log shows adaptive modifaction not ness at 3am correctly but last night around that database update there was nothing all the way to 6pm when i noticed it and started messing with it.

Correction the night before during the update!

Thinking about this now if there is a cron issue it may also be related to some others reports of there parental controls not working quite right as well ive noticed latly.

just logged in.. can confirm my webui date/time matches the "date" command in putty. unfortunately I didn't let it go to 10pm before messing with it to see if it would have fired up then so I cant be sure it would have at all until 3am next day with would have been last night. after messing with it and restarting qos etc it did correctly run at 3am next day.

Mar 30 19:40:50 adaptive QOS: Delayed Start Triggered (5min)
Mar 30 19:45:52 adaptive QOS: Applying - Down Rules
Mar 30 19:45:52 adaptive QOS: Applying --- Up Rules
Mar 30 19:45:52 adaptive QOS: Modifying Class Rates
Mar 30 19:52:55 kernel: htb: htb qdisc 16: is non-work-conserving?
Mar 30 20:01:43 kernel: htb: htb qdisc 15: is non-work-conserving?
Mar 30 20:05:56 kernel: htb: htb qdisc 13: is non-work-conserving?
Mar 30 21:06:07 kernel: htb: htb qdisc 13: is non-work-conserving?
Mar 30 21:21:28 kernel: htb: htb qdisc 16: is non-work-conserving?
Mar 30 22:11:50 kernel: htb: htb qdisc 15: is non-work-conserving?
Mar 30 22:15:20 kernel: htb: htb qdisc 11: is non-work-conserving?
Mar 31 03:00:00 adaptive QOS: Scheduled Persistence Check -> No modifications necessary
Mar 31 06:48:55 dropbear[22004]: Password auth succeeded for 'admin' from 192.168.10.120:57992

*update*
cru 1 doesn't report anything usefull other than istructions to use it:

ASUSWRT-Merlin RT-AC3100 384.4-2 Sat Mar 24 17:02:49 UTC 2018
admin@Asus:/tmp/home/root# date
Sat Mar 31 06:49:20 DST 2018
admin@Asus:/tmp/home/root# cru 1
Cron Utility
add: cru a <unique id> <"min hour day month week command">
delete: cru d <unique id>
list: cru l
admin@Asus:/tmp/home/root#
admin@Asus:/tmp/home/root#

 

[geobernd]

I cannot confirm or deny if there is a problem with the compatible version of the script and the scheduled cron task.

My system clock says 3AM, but cron thinks its 8AM. For some reason it is using the GMT timezone.

This means that it would actually be triggered at 10PM instead of 3AM.

Very Strange: On My RT-AC68P Cron is in sync with local time (I am in EDT) and it runs at 3am -here is the run from Thu->Fri with the reapply after the actual 2am update:

Mar 30 02:00:37 rc_service: rc 32687:notify_rc restart_wrs
Mar 30 02:00:57 kernel: ERR[parse_qos_conf:932] Can't set new QoS conf while QoS is started!
Mar 30 02:00:57 kernel: ERR[ioctl_iqos_op_config:3592] parse qos_conf error!!
Mar 30 02:00:57 kernel: ioctl_iqos_op_config() fail!
Mar 30 02:00:57 kernel: ERR[qos_start:3344] QoS is already started!
Mar 30 02:00:57 kernel: ioctl_iqos_op_switch(1) fail!
Mar 30 02:04:43 kernel: htb: htb qdisc 17: is non-work-conserving?
...
Some DHCP request logs in between
...
Mar 30 03:00:00 crond[293]: USER redacted pid 10268 cmd /jffs/scripts/FreshJR_QOS -check
Mar 30 03:00:01 adaptive QOS: Scheduled Persistence Check -> Reapplying Changes
Mar 30 03:00:03 adaptive QOS: Applying - Down Rules
Mar 30 03:00:03 adaptive QOS: Applying --- Up Rules
Mar 30 03:00:03 adaptive QOS: Modifying Class Rates
Mar 30 03:00:03 kernel: HTB: quantum of class 10011 is big. Consider r2q change.
Mar 30 03:00:04 kernel: HTB: quantum of class 10013 is big. Consider r2q change.
Mar 30 03:00:04 kernel: HTB: quantum of class 10014 is big. Consider r2q change.
Mar 30 03:04:03 kernel: htb: htb qdisc 15: is non-work-conserving?

 

[Sinner]

Very Strange: On My RT-AC68P Cron is in sync with local time (I am in EDT) and it runs at 3am -here is the run from Thu->Fri with the reapply after the actual 2am update:

Mar 30 02:00:37 rc_service: rc 32687:notify_rc restart_wrs
Mar 30 02:00:57 kernel: ERR[parse_qos_conf:932] Can't set new QoS conf while QoS is started!
Mar 30 02:00:57 kernel: ERR[ioctl_iqos_op_config:3592] parse qos_conf error!!
Mar 30 02:00:57 kernel: ioctl_iqos_op_config() fail!
Mar 30 02:00:57 kernel: ERR[qos_start:3344] QoS is already started!
Mar 30 02:00:57 kernel: ioctl_iqos_op_switch(1) fail!
Mar 30 02:04:43 kernel: htb: htb qdisc 17: is non-work-conserving?
...
Some DHCP request logs in between
...
Mar 30 03:00:00 crond[293]: USER redacted pid 10268 cmd /jffs/scripts/FreshJR_QOS -check
Mar 30 03:00:01 adaptive QOS: Scheduled Persistence Check -> Reapplying Changes
Mar 30 03:00:03 adaptive QOS: Applying - Down Rules
Mar 30 03:00:03 adaptive QOS: Applying --- Up Rules
Mar 30 03:00:03 adaptive QOS: Modifying Class Rates
Mar 30 03:00:03 kernel: HTB: quantum of class 10011 is big. Consider r2q change.
Mar 30 03:00:04 kernel: HTB: quantum of class 10013 is big. Consider r2q change.
Mar 30 03:00:04 kernel: HTB: quantum of class 10014 is big. Consider r2q change.
Mar 30 03:04:03 kernel: htb: htb qdisc 15: is non-work-conserving?

Hmmm that is reasuring somewhat. Mine time also appears ok like yours but script didnt run at all after update. It could be that i do mess with the rules and reapply or turn on off the script. Maybe somewhere in there its removing or not reapplying the cron job. But @FreshJR time is out as well which is wierd.

 

[maxsteel]

@FreshJR thanks for the detailed response. Will try to see the method explained to see what I can figure out regarding the ISP limitations.

@Vexira 12 Sounds reasonable based on your response. Looking for outcome you can find. Thanks,

 

[Spydawg]

Maybe someone here can help.

If I enable QOS (Default or FreshJR's) I can't seem to get to some Websites for example www.dslreports.com

I'm running a ASUS RT-AC3100 Firmware Version: 384.4_2

My Connection is Fiber 350 down 100 up.

I have done a factory reset and Initialize plus Format JFFS partition.

This all started after I upgraded my firmware from 384.3_0,
2 days later my Lexar USB Key Died that was in the router that had AB-solution, pixelserver.tls , and FreshJR's QOS script.
Replaced it with a Kingston DT50 3.1 USB Key and been having issues since. Tried with a few more USB Keys I had, thinking it could be the problem. But issues remain.


Any ideas?

 

[vibroverbus]

JR I had some weird timezone problem like that recently, but sadly can't remember what I had to do to fix. Sorry.

cru 1 doesn't report anything usefull other than istructions to use it:

Not "cru 1" aka 'the number one'. That's what you've typed and whats in your screen copy/paste so its pretty clear that is what you are inputting.

"cru l" aka lower-case-L as in 'List cron jobs'.

 

[FreshJR]

If you want to test for cron time mismatch you can run this via putty

 cru a test '* * * * * logger $(date)'

wait for some cron activity in system log

then remove the test with

cru d test

I am pretty sure the script is fine, but cron is living in a different timezone for some users!

When I looked at the cron schedule, I saw I had an extra space present in executing the script but that shouldn't cause any issues.

@Spydawg QOS does NOT block any web access, so I have no idea whats going on (especially if the issue persists with default QOS)

 

[Sinner]

If you want to test for cron time mismatch you can run this via putty

 cru a test '* * * * * logger $(date)'

wait for some cron activity in system log

then remove the test with

cru d test

I am pretty sure the script is fine, but cron is living in a different timezone for some users!

When I looked at the cron schedule, I saw I had an extra space present in executing the script but that shouldn't cause any issues.

@Spydawg QOS does NOT block any web access, so I have no idea whats going on (especially if the issue persists with default QOS)

yeh I think whatever I'm doing with it after changes may be related to why it didn't start which would suggest some combination of qos on/off or possibly after turning it on and then realizing I needed to adjust the ul/dl rates, adjting them and hitting apply again may have caused it.. I usually get an error handler of yours stating its canceling the previous and starting it over. I suspecting somewhere within these procedures is causing that issue.

 

[Sinner]

JR I had some weird timezone problem like that recently, but sadly can't remember what I had to do to fix. Sorry.



Not "cru 1" aka 'the number one'. That's what you've typed and whats in your screen copy/paste so its pretty clear that is what you are inputting.

"cru l" aka lower-case-L as in 'List cron jobs'.

yeh looked like a one.. my bad.. cru l(small L) reports nothing here atm. should it list the 3am restart I'm guessing?

 

[Sinner]

so after a "cru l" theres nothing.. after adding "cru a FreshJR_QOS "0 3 * * * /jffs/scripts/FreshJR_QOS -check"" and the using "cru l" its listed there now:

michael@Asus:/tmp/home/root# cru l
michael@Asus:/tmp/home/root# cru a FreshJR_QOS "0 3 * * *  /jffs/scripts/FreshJR
_QOS -check"
michael@Asus:/tmp/home/root# cru l
0 3 * * *  /jffs/scripts/FreshJR_QOS -check #FreshJR_QOS#
michael@Asus:/tmp/home/root#

whats strange is the script is running as we speak

*update*
ive found an extra space in the cru check line I posted. not sure if it matter.. corrected and testing now

 

[FreshJR]

So it seems that all CRON tasks get reset upon a reboot. (I was not aware that CRON tasks did not exhibit persistance)

This explains the why the users who posted above were missing their scheduled CRON jobs.

I will update the compatible version of the script to re-initialize CRON tasks upon reboot.

EDIT:

UPDATE TO COMPATIBLE VERSION OF SCRIPT (v369 -> v370)
-Fixed issues of missing cron task after reboot
-Additional check so FreshJR_QOS is only ran when WebUI is set to Adpative QOS and not Traditional QOS or Bandwidth limiter


Side Note:
Upon performing a router reboot, my bugged cron tasks started executing within the correct time zone so I have no idea what was happening.
Either way, I fixed the bug of the scheduled cron task clearing itself upon reboot! Worst case scenario if you happen to exhibit the timezone bug is that the script will be checked within a random period every 24hrs instead of 3am.

 

[skeal]

@FreshJR I have a question. I re-downloaded Ubuntu 17.04.1 and could not see the traffic in the stats it never showed up anywhere except when watching the bandwidth monitor. The monitor classed it as web file transfer. The traffic is recorded by device in traffic analyzer statistics also shows web file transfer. Is this normal to not see the data in QOS statistics classified as anything? It's like I didn't download anything as far as QOS statistics is concerned. Any ideas or advice you can give a "qos noob"?

 

[skeal]

@FreshJR I have a question. I re-downloaded Ubuntu 17.04.1 and could not see the traffic in the stats it never showed up anywhere except when watching the bandwidth monitor. The monitor classed it as web file transfer. The traffic is recorded by device in traffic analyzer statistics also shows web file transfer. Is this normal to not see the data in QOS statistics classified as anything? It's like I didn't download anything as far as QOS statistics is concerned. Any ideas or advice you can give a "qos noob"?

Ok it (ubuntu download running at full speed) is being classified as video streaming traffic. This allows it to interfere with my video stream which in my case is of highest of priorities. Video is what I do mainly and some computer gaming also. Thanks for the script!! Excellent work!
EDIT: I fixed this problem by re-installing the script with no modifications. Now all the traffic is classified correct. I'm the fool that didn't configure things right Can you give me a hint on how to make video traffic highest priority please? It may be now but I'm a noob to all this QOS stuff and wouldn't know.

 

[FreshJR]

Can you give me a hint on how to make video traffic highest priority please?

Two ways:

1) You can simply allocate more guarenteed bandwidth for the video via the configurable percents in the script area. This area is space & integer sensitive. Do NOT add a space before or after the equal sign. Do NOT use decimals
(Note: If you allocate video catagory more bandwidth, you then have to decrease bandwidth from other catagories so the sum of all categories is NOT greater than 100)


2) You can drag video higher up in WebUI. Once every QOS category gets its guaranteed bandwidth, the left over bandwidth is offered once again from top to bottom as set in the WebUI. Setting video higher will simply offer the left over bandwidth to videos before the catagories below it are offered any.

Feel free to combine steps 1 & 2

If making modifications to the script, do not forget the dos2unix step in the install instructions.