[Release] FreshJR Adaptive QOS (Improvements / Custom Rules / and Inner workings)

[FreshJR]

@randomName

You are over thinking it. Just enable it once and leave it enabled.

Everything else is just network variance.

There are different hosts, expierencing different loads, along with different levels of congestion through both peering networks and your local ISP backbone.

I can notice quicker webpage load times at 4am vs 4pm.

 

[JohnSmith]

I have explained it many times. LACK OF SANE BANDWIDTH ALLOTMENT.

To see for yourself, simply run two speed tests simultanously on devices of different priorities.

- One device will get 99.9% of the bandwidth
- The other device will get 00.1% of the bandwidth.

If you think that is acceptable behavior and a functioning QoS system, then more power to you.

Now for devices of the same priority, the bandwidth allocation is alloted fairly.

Thanks for the info FreshJR, I might change it back to all devices have Default Priority on my next firmware upgrade to test, but V384.10_2 has been quite stable so far, with my current settings (14+ days so far), and with all 40+ devices on the network, along with gaming, streaming, uploads, Netflix, etc. at the same time, nobody in the family has complained so far. Thanks for everything you have done, and its too bad the "Adaptive QOS | Bandwidth Monitor" tab that has the Priority tabs, could not just be color coordinated tabs (or separate from device priority color tabs), so that you could sort by color your devices so you can have your high value devices listed at the top to keep an eye on them.

 

[dave14305]

I can make modifications so that (53,123 and now 853) goes into NetControl instead of it being 0-rated but I didn't perform A/B testing to see which scenario would be more responsive. The current behavior gets me a 30-33ms response time per query. (every ms does counts for DNS but I didn't test the other configuration).

It also could be worth looking into not only what is happening with DNS requests, but also what is happening to DNS responses directed to the router and their interaction with QOS. (Not quite sure where those end up currently)

Now that DoT will potentially see broader adoption with Merlin 384.11, any further experiments with excluding port 853 from Downloads?

 

[FreshJR]

Now that DoT will potentially see broader adoption with Merlin 384.11, any further experiments with excluding port 853 from Downloads?

Yes, that change will be included in next release which I just started actively working on a few days ago.

I hope I can finish it this week.

I will have Router DNS outgoing traffic (eg. DNS request from router) bypass QOS and make them whitelisted/zero rated.

I have not done A/B testing but I think since dns requests are so small and insignificant size wise, not subjecting the traffic to queues(qos system),may be more responsive and the resulting QOS bandwidth inaccuracies insignificant.

(I may make a toggle for users who want have the router DNS traffic placed into net control but on the other hand keeping the UI simple may lead to less questions)

—

On another note:

Ideally I’d like to have INCOMMING traffic towards the router not whitelisted. This would make local processes on the router such as VPN server and Download Master compatible with QOS.

I am still in the same position as a few years ago have not figured out an implementation. I have tried reading more manuals to figure out how to do it but still not familiar enough with the topic at hand.

I do know to get that to working, I would have to create an IFQ to mirror traffic from the input and forward chains, but I am still a little shaky on how it would affect hardware acceleration and the commands required to properly implement it under the hood. I do also wonder if pointing TC to the newly IFB would work after I figure out how to set it up.

This is because eth0 in TC is not related to eth0 in iptables. I have no idea why the two references don’t line up and where in the routing structure this disconnect is present.

This is just something that’s been a wish on the back of my mind for a while. And I sure wouldn’t mind if the network engineers at ASUS, who are already familiar with how all this works, would fix this in a future firmware release. So QOS can work as expected instead of the broken state it has been abandoned it.

If any asus rep wants a bug report explaining how VPN server / client are incompatible with QOS, send me a message for discussion. Is the issue awareness??!?

I have implemented a dirty fix to the router acting as a VPN client but the VPN server is still left in a glitched state. 1/2 functionality is better then nothing, but full functionality would be great, and is not impossible.

 

[NeoEpsilon]

@FreshJR
Last time I tried this script + YazFi, the Guest subnet (192.168.2.0) traffic was detected but only the upload.
I added "192.168.2.96/28" into the custom rule as Others, guest IP range is 100-109.
Is this normal..?
Amazing scripts by the way...

 

[CriticJay]

I will have Router DNS outgoing traffic (eg. DNS request from router) bypass QOS and make them whitelisted/zero rated.

I have not done A/B testing but I think since dns requests are so small and insignificant size wise, not subjecting the traffic to queues(qos system),may be more responsive and the resulting QOS bandwidth inaccuracies insignificant.

Terrific idea, I like this!

 

[RMerlin]

I will have Router DNS outgoing traffic (eg. DNS request from router) bypass QOS and make them whitelisted/zero rated.

Shouldn't that go in the Net Control category instead? Ideally, DNS traffic should have a high priority, to ensure smoother operations under high traffic load.

Same would be good for port 853 traffic (DoT).

 

[Kingp1n]

Yes, that change will be included in next release which I just started actively working on a few days ago.

I hope I can finish it this week.

I will have Router DNS outgoing traffic (eg. DNS request from router) bypass QOS and make them whitelisted/zero rated.

I have not done A/B testing but I think since dns requests are so small and insignificant size wise, not subjecting the traffic to queues(qos system),may be more responsive and the resulting QOS bandwidth inaccuracies insignificant.

(I may make a toggle for users who want have the router DNS traffic placed into net control but on the other hand keeping the UI simple may lead to less questions)

—

On another note:

Ideally I’d like to have INCOMMING traffic towards the router not whitelisted. This would make local processes on the router such as VPN server and Download Master compatible with QOS.

I am still in the same position as a few years ago have not figured out an implementation. I have tried reading more manuals to figure out how to do it but still not familiar enough with the topic at hand.

I do know to get that to working, I would have to create an IFQ to mirror traffic from the input and forward chains, but I am still a little shaky on how it would affect hardware acceleration and the commands required to properly implement it under the hood. I do also wonder if pointing TC to the newly IFB would work after I figure out how to set it up.

This is because eth0 in TC is not related to eth0 in iptables. I have no idea why the two references don’t line up and where in the routing structure this disconnect is present.

This is just something that’s been a wish on the back of my mind for a while. And I sure wouldn’t mind if the network engineers at ASUS, who are already familiar with how all this works, would fix this in a future firmware release. So QOS can work as expected instead of the broken state it has been abandoned it.

If any asus rep wants a bug report explaining how VPN server / client are incompatible with QOS, send me a message for discussion. Is the issue awareness??!?

I have implemented a dirty fix to the router acting as a VPN client but the VPN server is still left in a glitched state. 1/2 functionality is better then nothing, but full functionality would be great, and is not impossible.

@FreshJR will the updated script have IPV6 working or this will take more time?

 

[StefanoN]

Hi
I'm trying to configure the application, in the image you can see my settings, the band values are 95% of the values that are given by speedtest (I still have an ADSL + 20MB download and 1 MB upload). In the log, however, I see these messages:
Apr 29 18:09:19 kernel: HTB: quantum of class 10010 is small. Consider r2q change.
Apr 29 18:09:19 kernel: HTB: quantum of class 10014 is small. Consider r2q change.
Apr 29 18:09:19 kernel: HTB: quantum of class 10017 is small. Consider r2q change.
Looking at older posts, I found some indications that I don't understand very well . Can you give me some additional info ?
I have an RT-86U with rmerlin firmware 384.10.2.

Thanks

 

[Kingp1n]

Hi
I'm trying to configure the application, in the image you can see my settings, the band values are 95% of the values that are given by speedtest (I still have an ADSL + 20MB download and 1 MB upload). In the log, however, I see these messages:
Apr 29 18:09:19 kernel: HTB: quantum of class 10010 is small. Consider r2q change.
Apr 29 18:09:19 kernel: HTB: quantum of class 10014 is small. Consider r2q change.
Apr 29 18:09:19 kernel: HTB: quantum of class 10017 is small. Consider r2q change.
Looking at older posts, I found some indications that I don't understand very well . Can you give me some additional info ?
I have an RT-86U with rmerlin firmware 384.10.2.

Thanks

These are normal, from 1st 3 post:
Is this an issue?

Code:
HTB: quantum of class 10016 is big. Consider r2q change.

No

 

[StefanoN]

These are normal, from 1st 3 post:
Is this an issue?

Code:
HTB: quantum of class 10016 is big. Consider r2q change.

No

Hi
Sorry I didn’t remember those post.
Thanks

 

[randomName]

So I'm trying the latest stock firmware and trying to install FreshJR per the instructions. I've installed the stock firmware, restored to factory and initialized all settings. Set up the router, then tried the install and this came up,
The system cannot find the file specified.
C:\FreshJR_QOS-master\scripts: No such file or directory
The folder has everything in it shown in the directions:

FreshJR_QOS.sh
FreshJR_QoS_Stats.asp
Manual Install - Windows - FreshJR QOS.bat
putty.exe
pscp.exe

Do I need to paste the curl code somewhere?

EDIT: The system log shows:
Apr 29 17:10:37 kernel: jffs2: warning: (780) jffs2_sum_write_data: Not enough space for summary, padsize = -1717

EDIT2: I didn't enabled AiPortection or QoS. I was going to wait till after the script was installed. Could that be it?

 

[FreshJR]

Shouldn't that go in the Net Control category instead? Ideally, DNS traffic should have a high priority, to ensure smoother operations under high traffic load.

Same would be good for port 853 traffic (DoT).

- Standard behavior, without the VPN fix, was that all router generated traffic was whitelisted.

- With the VPN client fix, all router generated traffic is assigned a downloads_category mark but an exception has been made for traffic on ports 53,123 (DNS)

- Traffic on those two ports remains whitelisted allowing DNS request to be zero-rated. (Original behavior for router generated traffic)

@dave14305 just brought up the point that the catch_all rule placing router traffic info downloads should be updated to include 853 as not to ruin responsiveness.

I get good DNS response times ~25ms for uncached requests, so I never experimented subjecting DNS traffic into the QOS system. Doesn’t hurt to perform A/B testing but it should perform identically as long as net control is not constrained.

@randomName do you have teamviewer.

I have no idea why the batch script isn’t running on your pc. Id have to see it in person.

I have tested it the batch script for paths with spaces and also fresh windows10 install. It worked with those.

 

[randomName]

@FreshJR Does Telnet also have to be enabled? I only enabled SSH

EDIT: the Manual Install.bat file does work and I do get to the cmd where I enter my credentials but that's when it says it can't find the script. Here is what it says in cmd:

Spoiler

[x] FreshJR_QOS.sh C:\FreshJR_QOS-master\FreshJR_QOS.sh
[x] FreshJR_QoS_Stats.asp C:\FreshJR_QOS-master\FreshJR_QoS_Stats.asp
[x] putty.exe C:\FreshJR_QOS-master\putty.exe
[x] pscp.exe C:\FreshJR_QOS-master\pscp.exe

Getting router login information

Router username:
Router password:
Router ipaddress:

Transferring files onto the router

The system cannot find the file specified.
C:\FreshJR_QOS-master\scripts: No such file or directory

The system cannot find the file specified.
The server's host key is not cached in the registry. You
have no guarantee that the server is the computer you
think it is.
The server's bla bla bla key fingerprint is:

If you trust this host, enter "y" to add the key to
PuTTY's cache and carry on connecting.
If you want to carry on connecting just once, without
adding the key to the cache, enter "n".
If you do not trust this host, press Return to abandon the
connection.
Store key in cache? (y/n)

Even had a USB drive installed...

EDIT2: Ok, so the folder created was in a restricted folder due to windows security settings changes. I did get the files to transfer over but when I opened up QoS it didn't give me a fq_codel option

 

[FreshJR]

Stock firmware doesn’t not support fq-codel or the WebUI.

Stock firmware has a limited, but functioning, version of the script.

I’ll look into recreating that error message on my end.

Edit: error message was a due to a windows permissions issue. The batch script couldn’t create a folder in the current directory and a subsequent command failed to launch. Try running it from the desktop??

I’ll see if I can make the batch installer request admin rights. Not familiar with windows’ terrible script environment, but hopefully it’s possible.

If you guys are running the installer sandboxed or something goofy, just open the batch script in notepad and see the lines of code for yourself. It doesn’t do anything malicious.

 

[randomName]

What worked for me was to run it from my Downloads folder.

 

[FreshJR]

After researching IPv6 standards, it turns out I have ran into quite a roadblock with IPv6 support.

The release is delayed until I figure out the most appropriate way of dealing with the current limitations at hand.

--

A popular rule with the script was to consolidate local devices into a continous iprange and then create QOS rules for that consolidated range.


Drawback 1 .... IPv6 doesn’t support manual IP assignments. This means it is not possible to get a continuous ranges of IP addresses representing clusters of local devices .

(This is not something I would recommended due with IPv6 due to tracking concerns. Keep reading for details.)

Originally, stateless IPv6 had EIU64 assignment as a standard. This meant that the last 64 bits of IPv6 addresses were were assigned the MAC address of the device accessing the internet, with an extra FFFE appended as filler.

This approach allowed terrible cross network tracking/analytics. Luckily, this behavior has been long abandoned.

The current approach for stateless IPv6 is known as privacy extension. Privacy extension means that devices will continually change the last 16 hexadecimal characters of its IP address on a rolling basis, and at any given time the last 64 bits of their IPv6 address will be random assignments.

Drawback 2 .... If intending to use state-less IPv6, is NOT possible to create local device rules since device IP addresses do not remain constant.

There is an alternate mode of IPv6 which can support local device IP rules. It is called statefull IPv6.

In statefull IPv6, the router runs a DHCPv6 server and assigns each device a static/constant IPv6 address from the dhcp pool.
(It works the same as ipv4 dhcp, except that these DHCP assigned addresses are now publicly routable rather than behind nat).

Drawback 3 .... If intending to use state-full IPv6, it IS possible to create local device rules. Having QoS limited to the state-full mode of IPv6 limits functionality and once again introduces tracking compromises (to a lesser extent that EUI64).

Drawback 4 ....State-full IPv6 support would be simple if iptables supported filtering by destination MAC address. Unfortunately, it does not. That would be a super simple workaround for not having a static localIP available as a filterable element.

—It is my opinion that Stateless IPv6 (with privacy extensions) is the only proper mode of IPv6 operation for consumer devices—


**


Currently some alternatives exist.

1) Since destination-mac is not available in iptables, a hacky workaround would be to get the source MAC for outgoing traffic and assign a connmark corresponding to it.

With this, the connmark for incomming traffic can be checked as a hacky way to filter based destination MAC. (Needs research and stability testing)

2) Use ebtables to filter on destination mac. (I need to do research on Linux packet transversal in order to see if the ebtables POSTROUTING is even evaluated for incomming WAN packet transversal)

3) Use TC to filter with manually set offsets, to filter on destination MAC since it has access to the full link layer packet. (Not something I want to do, since I want to squash the “traffic existing prior / during 5minute wait” with this release.)

Different Asus models would have different packet offsets under different network conditions, leading to support headaches.

--

Anyone have an answer to the ebtables proposed solution? I will be doing my own research and starting a separate thread on that topic.

 

[CriticJay]

Thanks for all your hard work! I do agree that Stateless IPv6 w/Privacy Ext is the way to go for us consumers/prosumers - that matches up with research I did on the subject some months ago.

For those of us who live in jurisdictions where IPv6 isn't really required yet (i.e. North America), perhaps it's preferable to simply disable it on the router so we have access to the full range of QoS tweaks?

 

[randomName]

So with stock firmware and using the functional FreshJR Adaptive QoS mod, it's required to have an USB drive 24/7 installed on the router. How can I make the USB drive accessible ONLY through one wired device, my PC, and not via my smart TV or any other WiFi device? My wifi is password protected but I just noticed my TV can see my router and the USB drive contents.

Thanks

 

[Vexira]

After researching IPv6 standards, it turns out I have ran into quite a roadblock with IPv6 support.

The release is delayed until I figure out the most appropriate way of dealing with the current limitations at hand.

--

A popular rule with the script was to consolidate local devices into a continous iprange and then create QOS rules for that consolidated range.


Drawback 1 .... IPv6 doesn’t support manual IP assignments. This means it is not possible to get a continuous ranges of IP addresses representing clusters of local devices .

(Nor would something like this be recommended due to tracking concerns. Keep reading for details.

Originally stateless IPv6 had EIU64 assignment as a standard. This meant that the last 64 bits of an IPv6 address the MAC address of the device accessing the internet with an extra FFFE appended as filler.

This approach would lead to the terrible cross network tracking/analytics. Luckily, this behavior had been long abandoned.

The new approach for stateless IPv6 is know as privacy extension. Privacy extension means that a device will continually change the last 16 hexadecimal characters of its IP address on a rolling basis, to random assignments.

Drawback 2 .... It is NOT possible to create local device rules with state-less IPv6 since device IP addresses will continually change to random assignments.

There does exist a stateful operation of IPv6 where the router runs a DHCPv6 server and assigns each device a permanent IPv6 suffix portion within the user-defined dhcp pool.
(It works the same as ipv4 dhcp, except that now DHCP assigned addresses are publicly routable).

Drawback 3 .... local devices rules are supported if wanting to use state-full IPv6, but only supporting state-full IPv6 limits functionality and introduces tracking compromises.

Drawback 4 .... iptables doesn’t support filtering by destination MAC address, which would be a super simple workaround for not being able to having localIP as a filtering field in state-full operation.

—It is my opinion that Stateless IPv6 (with privacy extensions) is the only proper mode of IPv6 operation for consumer devices—


**


Currently some alternatives exist.

1) Since destination-mac is not available in iptables, a hacky workaround would be to get the source MAC for outgoing traffic and assign a connmark corresponding to it.

With this, the connmark for incomming traffic can be checked as a hacky way to filter based destination MAC. (Needs research and stability testing)

2) Use ebtables to filter on destination mac. (I need to do research on Linux packet transversal in order to see if the ebtables POSTROUTING is even evaluated for incomming WAN packet transversal)

3) Use TC to filter with manually set offsets, to filter on destination MAC since it has access to the full link layer packet. (Not something I want to do, since I want to squash the “traffic existing prior / during 5minute wait” with this release.)

Different Asus models would have different packet offsets under different network conditions, leading to support headaches.

--

Anyone have an answer to the ebtables proposed solution? I will be doing my own research and starting a separate thread on that topic.

I'm actually quite saddened to hear the relase is delayed I was quite looking forward to it, would it be possible to perhaps relase the script without IPv6 as a beta relase then the full version once the IPv6 conundrum is resolved?