[Release] FreshJR Adaptive QOS (Improvements / Custom Rules / and Inner workings)

[Vexira]

Not quite sure what I'm doing wrong. I'm very sure the script is installed correctly, but I'm getting a consistent bufferbloat score of D or F.

My Settings:
Adaptive QOS
Manual Setting
fq_codel
WAN packet overhead = 0
Upload = 5
Download = 60
I selected "CUSTOMIZE" and am using the order from page 1 of this thread.

My internet is capped @ 60 down and 5 up. I consistently get speeds of 10% over those limits, so I left alone.
I've even tried lowering the upload value to 1, but still get high bufferbloat.


FWIW, I also get high bufferbloat with QOS disabled. If I change to traditional QOS (with all preset values) I get a bufferbloat score of A or A+.

You forgot to set the per packet overhead for your connection

 

[JGrana]

Good to hear I was on the right tracks
can you post the 50000-58192 lines...

and ... did you set 443 as VOIP? since it is standard HTTPS I have avodied that

Thanks

Good point, I forgot about that. In that case, I am going to remove the rule that puts it in VoIP

Gratis

 

[Andy1932]

You forgot to set the per packet overhead for your connection

Thanks. I have cable internet with WOW/Knology. Is "18" correct? ATM checked or unchecked?

 

[bbq9]

Good point, I forgot about that. In that case, I am going to remove the rule that puts it in VoIP

Gratis

I am a bit confused now. I grabbed TCPView from SysInternals to see if I could see what ports were in use.
seemed to be a jumbled mess, compared with the 443, 3478-3481, and 50000-59999...

I am going to have to think on this.

In fact, I went back to good-old-wikipedia, and ports 500, 4500, are actually used for IKE - internet key exchange... so not really VOIP at all, though one can imagine really important.

I think I am getting my sport and dport confused. need to draw this out

 

[FreshJR]

. Is using device priorities still recommended against?

I still do not recommend setting device priorities, this is still the current recommendation in the 2nd post of the thread. (The first three posts are always current).
I think Asus might of changed how they handle device priority on the recent firmwares but I am burnt from results in past firmware so I still do not trust it, nor do I want to experiment to find its quirks.

What has confused me is that FreshJR, in the new version has changed dport/sport for the VOIP rules in the custom rules section.

A connection has two ports.

The port on the server hosting the traffic.
The port on the computer receiving the traffic.

For Download traffic
dport = port of your LAN device
sport = port of the WAN server

For Upload traffic
sport = port of your LAN device
dport = port of the WAN server

For example, websites are hosted on port 80 or 443.

1) When you access a website, the server will send traffic from port 80 or 443.
2) Your PC will receive that traffic on a random port in the range 49152 - 65535 (ephemeral/dynamic port range).

As for VOIP,

iOS devices always recieve VoIP traffic on ports 500, 4500 (https://support.apple.com/en-us/HT202944)
(This is what the original rules were created to match on and were rock solid for all iOS devices).

The original rules were
Download (dport) = ports 500, 4500 on iPhone
Upload (sport) = port 500, 4500 on iPhone

After some complaints about VoIP classification on non-iOS devices, I found out that android actually receives its wifi-calling traffic on a random port in the 49152 - 65535 range.

Coincidentally I found that my VoIP servers (T-Mobile Wifi-Calling) were also hosted on ports 500, 4500

I updated the new rules are updated to be
Download (sport) = port 500,4500 on T-Mobile VoIP server
Upload (dport) = port 500,4500 on T-Mobile VoIP server.

Interesting note:
If you take a look at the rules will notice that Wifi-Calling has been switched to WAN port designation (as explained) but FaceTime is using original LAN port designation for port rules.
(I would of preferred filtering on fixed ports present on local devices but Android has its head in its butt!)

--

The original rule for iOS wifi calling is more reliable since the Apple guidelines guarantee those local port to be used.

The new rule is working on the assumption that all VoIP servers will be hosted on 500,4500 (which is not necessarily true) but at least gives Android devices a fighting chance for VoIP.

These changes have been made in a push to support non iOS devices.
I know it is possible to keep both sets of rules, but I always like to keep the amount of rules to a minimum.

----

Same line of dport/sprot thinking applies for custom rules.

The rules have to define either

1) the port the traffic is originating from (WAN device)
or
2) the port the local device is receiving traffic on (LAN device)

If you mix up the designations, don't be surprised if traffic isn't caught.

and ... did you set 443 as VOIP? since it is standard HTTPS I have avoided that

Correct, a 443 rule is a terrible idea. Remove it.

If I change to traditional QOS (with all preset values) I get a bufferbloat score of A or A+.

Try restarting the router and see if that helps.

I have recently received reports that the fast version may require a device restart after install to start working.

I will have to look into this bug report next time I reset my router (next major RMerlin firmware upgrade).

For now, I will just add instructions to restart the router after script install as a temporary work around.

 

[Kingp1n]

@FreshJR , quick question...the new scripts released have the option to make them as gaming priority correct? What im asking is we dont need a separate" gaming" script with a different setup correct?

 

[FreshJR]

@FreshJR , quick question...the new scripts released have the option to make them as gaming priority correct? What im asking is we dont need a separate" gaming" script with a different setup correct?

The optional gaming rule for incrementally better performance still exists, and is commented out by default, as in the previous versions.

 

[Kingp1n]

@FreshJR Ok, so messing around with the download/upload speeds within the original setup speeds within the script wont improve gaming itself, since gamin doesnt require more than 15%, its overkill correct? So for example wld there be any diffrence if I decrease the Others speed from your original setup of 30% to say 10% under the upload settings within the script? Just curious if this wld hurt someone with speeds of DL speeds of 175mbps and UL speeds 6mbps, sorry for so many questions man, i really do apologize, im just looking for consistent gaming without any lag or getting randomly kicked out from game lobbies

 

[Vexira]

Thanks. I have cable internet with WOW/Knology. Is "18" correct? ATM checked or unchecked?

Depends on your connection, if it's cable then yes, 18 is correct also ATM is only for delivery connections such as adsl 1/2 or any other connection that uses ATM proctol, vdsl 1/2 generally uses ptm.

 

[FreshJR]

@Kingp1n

Those categories are really there for you to mainly define how much speed each of the bulk transfers categories will get. The percents will control how much bandwidth streaming, file downloads, and game downloads will get with respect to each other, since each of those connection types would be happy to take all bandwidth offered (starving out the categories below) without guaranteed minimums .

I wouldn't really worry about the categories near the top of the list, since they are all primarily low bandwidth AND even during the times they are not, they still get first dibs on excess bandwidth. This means the upper categories will rarely get hurt, no matter how weirdly the numbers are assigned, so don't worry about them.

The percents will allow you to define that you will simultaneously have available XXmbps for streaming, XXmbps for downloads, and XXmbps for game downloads at any given time.

Out of these, the most important is streaming. I recommend users having at least 5mbps per simultaneous device streaming alloted. (A 15-20mbps guarentee should work well for a family w/ 1080p content). You will need to bump up the allotment an additional 15mbps per device that will be accessing 4K content.

If a users internet is slow that it cannot guarantee at least 5mbps for "Streaming" then they could bump "Streaming" up in the priority list and have other traffic suffer IF they are always wanting to have smooth video (there is only so much you can do if you have a limited connection, so trade offs have to be made).

I personally don't care how much bandwidth my (file downloads) and game downloads (default category) get so they are assigned the minimum recommended 5%.

My video category has ample breathing room and I decided to also pump up VoIP. The default numbers are generally good for most connections.

Lower than 5% is also a possible assignment but I don't want people to have super weird configurations so i recommend a reasonable minimum.

-----

UPDATE:

The reports stating that QOS (fast_version) doesn't work when performing a install on a fresh firmware until after a reboot were bothering me.

I reinitialized my router TWICE, and could not reproduce the issue.

May 10 04:54:59 adaptive QOS: FreshJR FakeTC Installed
May 10 04:54:59 adaptive QOS: Parsing User QOS Rates -> (Updating)

    May 10 04:55:06 kernel: Init chrdev /dev/detector with major 190
    May 10 04:55:06 kernel: tdts: tcp_conn_max = 8000
    May 10 04:55:06 kernel: tdts: tcp_conn_timeout = 300 sec
    May 10 04:55:21 kernel: SHN Release Version: 2.0.1 3529123_patch
    May 10 04:55:21 kernel: UDB Core Version: 0.2.14 r3529123
    May 10 04:55:21 kernel: Init chrdev /dev/idpfw with major 191
    May 10 04:55:21 kernel: IDPfw: IDPfw is ready
    May 10 04:55:21 kernel: sizeof forward pkt param = 192
 
May 10 09:55:22 adaptive QOS: Clearing qos environment
May 10 09:55:34 adaptive QOS: Clearing qos environment
 
    May 10 04:55:36 kernel: IDPfw: Exit IDPfw
    May 10 04:55:36 kernel: mod epilog takes 0 jiffies
    May 10 04:55:36 kernel: IDPfw: Exit IDPfw
    May 10 04:55:36 kernel: Exit chrdev /dev/idpfw with major 191
    May 10 04:55:36 kernel: Exit chrdev /dev/detector with major 190
    May 10 09:55:37 rc_service: bwdpi_check 319:notify_rc start_firewall
    May 10 04:55:37 miniupnpd[2808]: shutting down MiniUPnPd
    May 10 04:55:37 nat: apply nat rules (/tmp/nat_rules_eth0_eth0)
    May 10 04:55:38 custom_script: Running /jffs/scripts/firewall-start (args: eth0)
    May 10 04:55:38 miniupnpd[3214]: HTTP listening on port 34094
    May 10 04:55:38 miniupnpd[3214]: Listening for NAT-PMP/PCP traffic on port 5351
    May 10 04:55:49 rc_service: httpd 1090:notify_rc restart_qos;restart_firewall
 
May 10 04:55:49 adaptive QOS: Clearing qos environment
May 10 04:55:49 adaptive QOS: Parsing User QOS Rates -> (Updating)

    May 10 04:55:51 kernel: Init chrdev /dev/detector with major 190
    May 10 04:55:51 kernel: tdts: tcp_conn_max = 8000
    May 10 04:55:51 kernel: tdts: tcp_conn_timeout = 300 sec
    May 10 04:56:07 kernel: SHN Release Version: 2.0.1 3529123_patch
    May 10 04:56:07 kernel: UDB Core Version: 0.2.14 r3529123
    May 10 04:56:07 kernel: Init chrdev /dev/idpfw with major 191
    May 10 04:56:07 kernel: IDPfw: IDPfw is ready
    May 10 04:56:07 kernel: sizeof forward pkt param = 192
    May 10 04:56:07 kernel: ERR[qos_start:3356] qos_ops is not registered!
    May 10 04:56:07 kernel: ioctl_iqos_op_switch(1) fail!
    May 10 04:56:08 kernel: ERR[qos_start:3356] qos_ops is not registered!
    May 10 04:56:08 kernel: ioctl_iqos_op_switch(1) fail!

May 10 04:56:09 adaptive QOS: Clearing qos environment
May 10 04:56:09 adaptive QOS: Clearing qos environment
May 10 04:56:10 adaptive QOS: Parsing User QOS Rates -> (Updating)
May 10 04:56:12 adaptive QOS: Changing 1:10 class rate
May 10 04:56:14 adaptive QOS: Changing 1:11 class rate
May 10 04:56:16 adaptive QOS: Changing 1:12 class rate
May 10 04:56:17 adaptive QOS: Changing 1:13 class rate
May 10 04:56:21 adaptive QOS: Changing 1:14 class rate
May 10 04:56:25 adaptive QOS: Changing 1:15 class rate
May 10 04:56:26 adaptive QOS: Changing 1:16 class rate
May 10 04:56:28 adaptive QOS: Applying ------ Up Rules
May 10 04:56:28 adaptive QOS: Changing 1:17 class rate
May 10 04:56:28 adaptive QOS: Applying ---- Down Rules
May 10 04:56:40 adaptive QOS: *- Clearing Iptables -*

    May 10 04:56:43 miniupnpd[3214]: shutting down MiniUPnPd
    May 10 04:56:43 nat: apply nat rules (/tmp/nat_rules_eth0_eth0)
    May 10 04:56:43 custom_script: Running /jffs/scripts/firewall-start (args: eth0)
    May 10 04:56:43 miniupnpd[10845]: HTTP listening on port 50422
    May 10 04:56:43 miniupnpd[10845]: Listening for NAT-PMP/PCP traffic on port 5351
 
May 10 04:56:44 adaptive QOS: Applying  Iptables Rules


------- Manually check for updated Signatures --------

May 10 04:59:20 rc_service: httpd 1090:notify_rc start_sig_check
May 10 04:59:27 adaptive QOS: Clearing qos environment
May 10 04:59:28 adaptive QOS: Clearing qos environment
May 10 04:59:46 adaptive QOS: Clearing qos environment
May 10 04:59:46 adaptive QOS: Clearing qos environment
May 10 04:59:49 adaptive QOS: Changing 1:10 class rate
May 10 04:59:51 adaptive QOS: Changing 1:11 class rate
May 10 04:59:52 adaptive QOS: Changing 1:12 class rate
May 10 04:59:54 adaptive QOS: Changing 1:13 class rate
May 10 04:59:58 adaptive QOS: Changing 1:14 class rate
May 10 05:00:02 adaptive QOS: Changing 1:15 class rate
May 10 05:00:03 adaptive QOS: Changing 1:16 class rate
May 10 05:00:04 adaptive QOS: Applying ------ Up Rules
May 10 05:00:05 adaptive QOS: Changing 1:17 class rate
May 10 05:00:05 adaptive QOS: Applying ---- Down Rules
May 10 05:00:17 adaptive QOS: *- Clearing Iptables -*

On both router re-initializations the script modifications worked right away without a reboot. This was both before and after QOS definition updates.

Anyway, the fast version is experimental for a reason! If you cannot monitor system log or check the qos statistics page, then it is not for you. Instead you should definitely be using the compatible version!!

If it **does** have a serious bug like the reports and I happen to experience it, then fast will be discontinued instead of fixed. It is already hacky, and stacking more hacky patches on top of it is not an option. (Personally I like the instant results and verbose logging of the fast version)

Mine has been stable for the longest time, so idk what is going on.

 

[Andy1932]

Try restarting the router and see if that helps.

I have recently received reports that the fast version may require a device restart after install to start working.

I will have to look into this bug report next time I reset my router (next major RMerlin firmware upgrade).

For now, I will just add instructions to restart the router after script install as a temporary work around.

That doesn't seem to help. QOS seems to be working fine, though. It's just the dslreprts speed test shows a very high bufferbloat...around 400ms.

 

[FreshJR]

That doesn't seem to help. QOS seems to be working fine, though. It's just the dslreprts speed test shows a very high bufferbloat...around 400ms.

then your bandwidth limits are too high.

Did you read the 4 steps on how to properly setup QOS after it is installed?

 

[Andy1932]

then your bandwidth limits are too high.

Did you read the 4 steps on how to properly setup QOS after it is installed?

Yes...I think I did all correct...
Here's a quote from my earlier post in this thread.

I've since changed my wan overhead to 18, since I have WOW cable internet.
Thank you for the help.

Not quite sure what I'm doing wrong. I'm very sure the script is installed correctly, but I'm getting a consistent bufferbloat score of D or F.

My Settings:
Adaptive QOS
Manual Setting
fq_codel
WAN packet overhead = 0
Upload = 5
Download = 60
I selected "CUSTOMIZE" and am using the order from page 1 of this thread.

My internet is capped @ 60 down and 5 up. I consistently get speeds of 10% over those limits, so I left alone.
I've even tried lowering the upload value to 1, but still get high bufferbloat.


FWIW, I also get high bufferbloat with QOS disabled. If I change to traditional QOS (with all preset values) I get a bufferbloat score of A or A+.

 

[FreshJR]

Yes...I think I did all correct...
Here's a quote from my earlier post in this thread.

I've since changed my wan overhead to 18, since I have WOW cable internet.
Thank you for the help.

Let me get the output of

/jffs/scripts/FreshJR_QOS -debug

Does traffic show up correctly in QOS Statistics?
What router do you have so we can narrow it down into a router specific issue if this poort performance occurs with other people with the same setup.

 

[skeal]

Yes...I think I did all correct...
Here's a quote from my earlier post in this thread.

I've since changed my wan overhead to 18, since I have WOW cable internet.
Thank you for the help.

The part where you say this:

My internet is capped @ 60 down and 5 up. I consistently get speeds of 10% over those limits, so I left alone.
I've even tried lowering the upload value to 1, but still get high bufferbloat.

This concerns me as I had a similar situation and did a speed test with ookla and used those values as 100% Then I calculated my bandwidth settings as 95% down and 90% up and I have great results. Don't trust your isp to give you real time speed capabilities. You need to get a sense of what your network connection performs like in real life situations.

 

[Andy1932]

Let me get the output of

/jffs/scripts/FreshJR_QOS -debug

Does traffic show up correctly in QOS Statistics?
What router do you have so we can narrow it down into a router specific issue if this poort performance occurs with other people with the same setup.

RT-AC86u with the latest beta2
Traffic seems to be routing correctly.

admin@RT-AC86U:/tmp/home/root# /jffs/scripts/FreshJR_QOS -debug
adaptive QOS: Classes Present: 8
adaptive QOS: Undf Prio: 2
adaptive QOS: Undf FlowID: 1:13
adaptive QOS: ***********
adaptive QOS: Down Band -- 61440
adaptive QOS: Downrates -- 3072,12288,9216,6144,6144,18432,3072,3072
adaptive QOS: ***********
adaptive QOS: Up Band -- 5120
adaptive QOS: Uprates -- 256,1024,768,1536,512,512,256,256
adaptive QOS: ***********
adaptive QOS: Net = 1:10
adaptive QOS: VOIP = 1:11
adaptive QOS: Gaming = 1:12
adaptive QOS: Others = 1:13
adaptive QOS: Web = 1:14
adaptive QOS: Streaming = 1:15
adaptive QOS: Downloads = 1:16
adaptive QOS: Defaults = 1:17
adaptive QOS: ***********
adaptive QOS: FreshJR fakeTC is enabled
adaptive QOS: (Adaptive QOS is turned ON in router UI)

 

[Andy1932]

The part where you say this:

My internet is capped @ 60 down and 5 up. I consistently get speeds of 10% over those limits, so I left alone.
I've even tried lowering the upload value to 1, but still get high bufferbloat.

This concerns me as I had a similar situation and did a speed test with ookla and used those values as 100% Then I calculated my bandwidth settings as 95% down and 90% up and I have great results. Don't trust your isp to give you real time speed capabilities. You need to get a sense of what your network connection performs like in real life situations.

I've run speedtest many times the last few weeks and at different times of the day. My posted speeds are consistently conservative. I have lowered them in my testing, but it hasn't seemed to make a difference.

 

[FreshJR]

RT-AC86u with the latest beta2
Traffic seems to be routing correctly.

The debug log looks to be in working order.

What happens when you lower the rates by 10-15%?
Either traffic is NOT routing into QOS categories, or your limits are too high.

Also check system log to see if QOS is not stuck in continually occurring restart loop.

Nothing else besides these three possibilities explains your results.

 

[Manuela]

I wanted to test if compatible version worked in Stock ASUS Firmware without problem.

These steps followed:

1. Turn ON QoS

2. Create Directory by PuTTY

mkdir /jffs/scripts/

3. Send Script by CMD

cd C:\putty\
pscp -pw mypasswork -scp C:\Putty\FreshJR_QOS_compatible_v1.txt admin@192.168.2.1:/jffs/scripts/FreshJR_QOS

4. Install by PuTTY

sh /jffs/scripts/FreshJR_QOS -stock_install

5. Testing and working.

6. After Turn OFF QoS, uninstall by PuTTY and Reboot

/jffs/scripts/FreshJR_QOS -uninstall

7. Turn ON QoS, Re-install to see if it works again.

8. When I tried to create again the director tells me that it is already exists.

mkdir: can't create directory '/jffs/scripts/': File exists

9. Send Script By CMD

cd C:\putty\
pscp -pw mypasswork -scp C:\Putty\FreshJR_QOS_compatible_v1.txt admin@192.168.2.1:/jffs/scripts/FreshJR_QOS

10. Install by PuTTY and now show me this

admin@RT-ACXXX:/tmp/home/root# sh /jffs/scripts/FreshJR_QOS -stock_install
 Creating environemnt to trigger scripts post USB Mount
grep: invalid option -- x
BusyBox v1.17.4 (2018-03-26 10:27:41 CST) multi-call binary.

Usage: grep [-HhnlLoqvsriwFE] [-m N] [-A/B/C N] PATTERN/-e PATTERN.../-f FILE [FILE]...

Search for PATTERN in FILEs (or stdin)

Options:
        -H      Add 'filename:' prefix
        -h      Do not add 'filename:' prefix
        -n      Add 'line_no:' prefix
        -l      Show only names of files that match
        -L      Show only names of files that don't match
        -c      Show only count of matching lines
        -o      Show only the matching part of line
        -q      Quiet. Return 0 if PATTERN is found, 1 otherwise
        -v      Select non-matching lines
        -s      Suppress open and read errors
        -r      Recurse
        -i      Ignore case
        -w      Match whole words only
        -F      PATTERN is a literal (not regexp)
        -E      PATTERN is an extended regexp
        -m N    Match up to N times per file
        -A N    Print N lines of trailing context
        -B N    Print N lines of leading context
        -C N    Same as '-A N -B N'
        -e PTRN Pattern to match
        -f FILE Read pattern from file

Placing FreshJR_QOS entry into script_usbmount
 FreshJR QOS has been enabled
adaptive QOS: Scheduled Persistence Check -> No modifications necessary

11. I wait 10 minutes and QoS does not work.

12. I Reboot the router and wait 10 minutes and now QoS works.

I already have one day use it and it works without problems, just that error in step 10 and 11, but it works.

Thanks FreshJR.

 

[Andy1932]

The debug log looks to be in working order.

What happens when you lower the rates by 10-15%?
Either traffic is NOT routing into QOS categories, or your limits are too high.

Also check system log to see if QOS is not stuck in continually occurring restart loop.

Nothing else besides these three possibilities explains your results.

Gotcha. Thanks for looking into it. I'll keep messing with the values and see if it changes anything. FWIW, I'm not really complaining about anything. I'm just getting bad results with this test:
http://www.dslreports.com/speedtest

Any other tests to check bufferbloat?