[Release] FreshJR Adaptive QOS (Improvements / Custom Rules / and Inner workings)

[FreshJR]

@FreshJR Can I suggest an uninstall/install command, I can see users not fully reading the OP and wiping the firewall-start file, here is the basic standard that was suggested.

That way you can handle things like missing shebangs automatically;

        if [ ! -f "/jffs/scripts/firewall-start" ]; then
            echo "#!/bin/sh" > /jffs/scripts/firewall-start
        elif [ -f "/jffs/scripts/firewall-start" ] && ! head -1 /jffs/scripts/firewall-start | grep -qE "^#!/bin/sh"; then
            sed -i '1s~^~#!/bin/sh\n~' /jffs/scripts/firewall-start
        fi

Along with a standardised entry like;

/jffs/scripts/FreshJR_QOS # Added By FreshQOS

The goal of the install instructions is to wipe firewall-start if coming from v1.4.

This is because that's where the old script was installed too/as.

But yes, I will automate it so install / uninstall can be one command instead of 3.

It already follows a standard entry. Is the problem a missing comment?

Giving it a try. Let's see if it resolves the intermittent WiFi calling issues.

That's my only reason for pursing QOS modifications further. I had the same issue as you describe.

I really hope it was caused by adaptive QOS categorization and not router wireless instability/compatability.

Also note, the included ports are for T-Mobile (USA) and Facetime. Check if your cell phone carrier uses different ports!

If it was not caused by QOS and you find a solution, PM me.

 

[bbunge]

So far so good on my AC66U B1 with 380.67. As I had reset the router after the firmware upgrade I transfered the script file to the router with WSFTP then ran the other steps with Putty. Ran the firewall-start script from Putty without restarting the router.

Sent from my P01M using Tapatalk

 

[Vexira]

No it seems like that change was done by RMerlin currently present in his new beta.

I thought the WAN packet overhead was always included, even in the old versions. In this beta I see there is a WAN overhead drop down toggle, so give credit to him where its due, and those down the line aswell who came up with the codel /fq_codel algorithms, iptables, and traffic control engines. Amazing work all around!

lol what i was trying to ask was does the new script factor in the the packet overhead, or is that handled by qos, cause from my understading, pacet over head is deducted form total avalable bandwidth, keep in mind i could be wrong, i was just wondering if the script handels it when it does bandiwdth assignment, as in deducts the amout form available bandwidth or does qos handle that already.
Sorry for the confusing question.

 

[FreshJR]

lol what i was trying to ask was does the new script factor in the the packet overhead, or is that handled by qos, cause from my understading, pacet over head is deducted form total avalable bandwidth, keep in mind i could be wrong, i was just wondering if the script handels it when it does bandiwdth assignment, as in deducts the amout form available bandwidth or does qos handle that already.
Sorry for the confusing question.

The script keeps all original QOS behavior.

This means the, newly implemented, WAN overhead toggles function the same with or without the script.

I did not read up on the new algorithms yet as before we only had one option.

So YES with fq_codel, no with sfq, but I did not create that change so I don't have more information.

 

[bbunge]

There are a lot of questions over the wan packet overhead and there is no simple answer! FreshJR script works with sfq, codel and fq_codel but doesn't impact wan packet overhead. On my rig i use the preset for Ethernet VLAN as my ADSL2 modem seems to take care of everything else.

Sent from my P01M using Tapatalk

 

[FreshJR]

There are a lot of questions over the wan packet overhead and there is no simple answer! FreshJR script works with sfq, codel and fq_codel but doesn't impact wan packet overhead. On my rig i use the preset for Ethernet VLAN as my ADSL2 modem seems to take care of everything else.

Sent from my P01M using Tapatalk

I can see the packet overhead being accounted for with the new queing algorithms.

Before I used to hit my QOS limits on speedtest. This means that the overhead was not accounted for (I just thought that speedtest included the size of the packet headers as part of the speed reading).

Now I am getting slightly lower speeds than my QOS settings. This means that it has to be subtracting the overhead from defined bandwidth.

If this is working correctly, this can mean that upload/download can be set closer to ISP speeds. I still have more testing to do, but ad always I will post my findings.

 

[jpclarke]

Brilliant script, I find that amazon prime video and YouTube are not tagged correctly so will try and add this to the script somehow when I set this up.

I set the qos bandwidth to auto, if the script can use the figures from this it would be really powerful especially when scheduled to run regularly and bandwidth regularly fluctuating.

 

[MarioCaires]

Brilliant script, I find that amazon prime video and YouTube are not tagged correctly so will try and add this to the script somehow when I set this up.

I set the qos bandwidth to auto, if the script can use the figures from this it would be really powerful especially when scheduled to run regularly and bandwidth regularly fluctuating.

You are right about Amazon Video as is identified as "Web surfing", however Youtube is correctly identified to me as "Streaming".
Regarding the Bandwidth set to auto, it was mentioned several times that for this script to work properly it needs to be set to manual!

 

[jpclarke]

You are right about Amazon Video as is identified as "Web surfing", however Youtube is correctly identified to me as "Streaming".
Regarding the Bandwidth set to auto, it was mentioned several times that for this script to work properly it needs to be set to manual!

I know the current script uses the manual entries but would be interested to see if the bandwidth figures can be extracted from those obtained from the auto setting.

If the auto figures can be extracted it would allow auto refresh of the minimum bandwidth assigned to each queue on a schedule, this is however entirely based on how auto bandwidth gets its figures, how often and if these figures can be extracted.

Thanks

 

[Vexira]

band

Brilliant script, I find that amazon prime video and YouTube are not tagged correctly so will try and add this to the script somehow when I set this up.

I set the qos bandwidth to auto, if the script can use the figures from this it would be really powerful especially when scheduled to run regularly and bandwidth regularly fluctuating.

width set to auto is for a 10gb connection.

 

[icornish]

This looks really interesting... can it be used to implement different QOS for different devices - eg, my kids devices have lower QOS priority than the main TV, and my work PC has different settings all together (and is most important)?

 

[FreshJR]

This looks really interesting... can it be used to implement different QOS for different devices - eg, my kids devices have lower QOS priority than the main TV, and my work PC has different settings all together (and is most important)?

That was ASUS's implementation from the beginning, but they botched it, BAD!

See the attached image. On that page is where you setup device priority for adaptive QOS. The colored ranking determines device priority for traffic within EACH qos container.

The issue is that each device has a guaranteed bandwidth of 16kbps. Go to my original post and read the rate/ciel explanation to see why that is terrible. To make things short, ASUS's setup means that if a device with a higher priority can consume all the bandwidth in a QOS category, the devices below it will get nothing (actually 2 kB/s but that's so low its basically nothing).

For example, if a high priority and a lower priority device are watching video, the low priority device will get 2 kB/s at all times while the higher priority device is buffering. That is a terrible choice for distributing bandwidth between devices of different priorities.

Previously it wasn't possible to modify these guaranteed rate for the device subclasses, but now it seems to be a possibility with Rmerlins TC interposer! I would like to look this behavior next and fix it aswell, now that it is possible.

Currently I would recommend not setting device priority and leaving it on (Default). When all devices are in the same category they randomly get a share of the total bandwidth. This prevents scenario where lower priority devices get 2 kB/s max and instead has both devices getting adequate shares.

 

[gmangt4]

Sorry for what might seem like a silly question, I had issues with Adaptive QOS so turned it off but I seen your script and tried it and it fixed my issues, my question is I applied the original patch while running 380.66_6 and I have now flashed 380.67, do I need to run the un-install script or will the reflash have wiped it? Can I just run the new one without any other commands needed? Or what do I do? Thanks in advance.

 

[FreshJR]

No it should still work.

You can check for script output in system log. There should be output, but you may need to scroll up.

If you can't find any. Go to the qos page and click apply. It will rerun the script and you should have output within a minute.

 

[MarioCaires]

@FreshJR I have categorized my torrent port traffic to "Downloading" and it made a big difference, however it seems some traffic is still being categorized as "Others" and it is obviously much more noticeable when downloading... Watching at the QoS Statistics it can still identify 50% of the traffic as "Downloading" and the other half as "Others"!
Do you know of any reason for this to happen?

 

[FreshJR]

@FreshJR I have categorized my torrent port traffic to "Downloading" and it made a big difference, however it seems some traffic is still being categorized as "Others" and it is obviously much more noticeable when downloading... Watching at the QoS Statistics it can still identify 50% of the traffic as "Downloading" and the other half as "Others"!
Do you know of any reason for this to happen?

I just tested it and got 100% of traffic in my desired catagory.

So post your two TC commands (for upload/download) so I can check for errors.

Go to system log -> port forwarding, and check that your define ports are not being forwarded into something else by UPnP and NAT-PMP (both of which good torrent clients utilize)

Go to netowrk tools -> netstat -> netstat-nat -> by source IP -> select PC -> sort by dst-port. See what ports are being used by your PC.

^ repeat previous using windows task manger -> resource monitor -> network -> and check which ports your torrent client is actually using in windows.

TC is really low level stuff, it doesn't make mistakes. When I said I have 100% identified when filtering by port, I mean 100%. In my test I sent it to the last container just so it wouldn't blend with any other traffic.

 

[eddiez]

@FreshJR I have categorized my torrent port traffic to "Downloading" and it made a big difference, however it seems some traffic is still being categorized as "Others" and it is obviously much more noticeable when downloading... Watching at the QoS Statistics it can still identify 50% of the traffic as "Downloading" and the other half as "Others"!
Do you know of any reason for this to happen?

You might want to be more specific which traffic you are expecting to be recognized as 'download'. 'Some traffic' is not really clear .... You mean torrents?

 

[FreshJR]

You might want to be more specific which traffic you are expecting to be recognized as 'download'. 'Some traffic' is not really clear .... You mean torrents?

He made a custom rule to filter torrents by port. I understood.

He said the port identification is only catching some of the traffic on his port. That should not be possible unless the ip/tcp/udp header is non standard. I have not looked into if tc identifies offsets correctly for non standard headers.

Also guys I didn't write the custom rule write up yet but make sure to take it easy on the rules. Every packet will have to checked against your rules. U don't want too many. I'd recommend 5 at most. Port ranges are possible so I hope u didn't input many rules one by one without instructions for port ranges available.

 

[MarioCaires]

I have NAT-PMP and uPnP disabled in the client as I have created a port forwarding rule in the router, however in the router in the Port Forwarding tab I have another port opened for Teredo... maybe this is it...?
BTW which torrent client have you used for the test? Using uTorrent here and in the "Resource Monitor" under "TCP Connections" I am getting an awful lot of different local ports used by uTorrent...

Will create another rule for the Teredo port and report back.

Besides your 2 custom rules, I have created another 2, one for torrents and the other for my IPTV service so not abusing to much...

EDIT: I have added another rule for the Teredo port, however the issue is still there.

These are the 2 customs rules (incoming and outgoing traffic) I have created initially for the uTorrent port:

${tc} filter add dev br0 protocol all prio 1 u32 match ip dport 62500 0xffff flowid ${Downloads}

${tc} filter add dev eth0 protocol all prio 1 u32 match ip sport 62500 0xffff flowid ${Downloads}

 

[FreshJR]

I have NAT-PMP and uPnP disabled in the client as I have created a port forwarding rule in the router, however in the router in the Port Forwarding tab I have another port opened for Teredo... maybe this is it...?
BTW which torrent client have you used for the test? Using uTorrent here and in the "Resource Monitor" under "TCP Connections" I am getting an awful lot of different local ports used by uTorrent...

Will create another rule for the Teredo port and report back.

Besides your 2 custom rules, I have created another 2, one for torrents and the other for my IPTV service so not abusing to much...

EDIT: I have added another rule for the Teredo port, however the issue is still there.

These are the 2 customs rules (incoming and outgoing traffic) I have created initially for the uTorrent port:

${tc} filter add dev br0 protocol all prio 1 u32 match ip dport 62500 0xffff flowid ${Downloads}

${tc} filter add dev eth0 protocol all prio 1 u32 match ip sport 62500 0xffff flowid ${Downloads}

The commands look good. Don't know what to tell you.

Try port 62496 with 0xfff8 for the mask. It will open 62496-62503. Maybe utorrent has a glitch and is off by one. (That mask will not work with port 62500, will explain why when I update original post)

As for my client, I user deludge on windows and transmission on mac. I loved utorrent back in the day since it was 300kB and very functional but today it is bloated and has ads everywhere, no thank you.

Deludge UI doesn't blend into windows very well, but it's not adware. Seems to respect my port setting.

If windows says utorrent is using other ports, then utorrent is using other ports. Router can't do magic.