[Release] FreshJR Adaptive QOS (Improvements / Custom Rules / and Inner workings)

[Yo mans 2]

Putty to display the "command prompt" running on the router.

You open putty and connect to the router. Then you run those commands inside putty, NOT command prompt.

Yes I did that it could not find the folder FreshJR_QOS. Anyway Im holding off for now.
Cheers. if it works would be killer.

 

[FreshJR]

Yes I did that it could not find the folder FreshJR_QOS. Anyway Im holding off for now.
Cheers. if it works would be killer.

FreshJR_QOS is a file (with no extension), not a folder.

That file was supposed to be uploaded to the router using pscp.

I have no idea how it isn't working, since if you literally paste the commands one by one, everything should be ok.

Are you sure your router was 192.168.1.1, with user name admin, with all the files located under C:\putty, when uploading? pscp should of also prompted you for router password if everything went alright

 

[Yo mans 2]

Yes I logged in no idea why it could not find it. I uploaded it twice still could not find it. I am on an older firmware. 380.69 or so. 382 will not work on this network the xboxes become useless no connection on any of them and wifi is flaky.

 

[FreshJR]

you use script v1.92 for 380.69.
The gaming rules will work. Just paste them into their locations.

Send me the output of

ls /jffs/scripts/

in putty

 

[Yo mans 2]

you use script v1.92 for 380.69.
The gaming rules will work. Just paste them into their locations.

Send me the output of

ls /jffs/scripts/

in putty

It says the scrpit is there. I will try again.

 

[FreshJR]

It says the scrpit is there. I will try again.

Well no, don't try again. Beta3 will NOT work with your firmware. You have to install v1.92 from the first post.

Note: Filenames ARE case sensitive if you are retyping the commands instead of copy/paste

 

[FreshJR]

Feb 9 22:58:43 kernel: ERR[parse_qos_conf:932] Can't set new QoS conf while QoS is started!
Feb 9 22:58:43 kernel: ERR[ioctl_iqos_op_config:3592] parse qos_conf error!!

What is that?

No idea, but I stepped through a verbose QOS debug log on both my machine and another users.

Non of the TC commands, which is what this script modifies, were malformed.

This might be a message the occurs randomly using normal/stock ASUS qos. This message is not repeatable every time QOS starts, but even when it appears, QOS still works as you would expect it too and I haven't noticed any downsides.

You should be okay to ignore those messages.

---

There are 2 portions to the script.

1) Parse (nvram get bwdpi_app_rulelist ) and write user variables to FreshJR_QOS.vars
2) read FreshJR_QOS.vars and use to values to intercept & overwrite original TC commands

I stress tested it by spamming portion (1) and receiving no errors.
I checked verbose debug logs verify functionality of portion (2)

Both worked and did not produce error messages during testing.

**shrugs sholder & lifts hand in the air **

Stress test below:

Feb  9 11:46:23 admin: Adaptive QOS: Started (Reading User Variables)
Feb  9 11:46:28 admin: Adaptive QOS: Started (Reading User Variables)
Feb  9 11:46:28 admin: Adaptive QOS: Started (Reading User Variables)
Feb  9 11:46:28 admin: Adaptive QOS: Started (Reading User Variables)
Feb  9 11:46:29 admin: Adaptive QOS: Started (Reading User Variables)
Feb  9 11:46:29 admin: Adaptive QOS: Started (Reading User Variables)
Feb  9 11:46:30 admin: Adaptive QOS: Started (Reading User Variables)
Feb  9 11:46:30 admin: Adaptive QOS: Started (Reading User Variables)
Feb  9 11:46:32 admin: Adaptive QOS: Started (Reading User Variables)
Feb  9 11:46:32 admin: Adaptive QOS: Started (Reading User Variables)
Feb  9 11:46:32 admin: Adaptive QOS: Started (Reading User Variables)
Feb  9 11:46:33 admin: Adaptive QOS: Started (Reading User Variables)
Feb  9 11:46:33 admin: Adaptive QOS: Started (Reading User Variables)
Feb  9 11:46:34 admin: Adaptive QOS: Started (Reading User Variables)
Feb  9 11:46:34 admin: Adaptive QOS: Started (Reading User Variables)
Feb  9 11:46:34 admin: Adaptive QOS: Started (Reading User Variables)
Feb  9 11:46:35 admin: Adaptive QOS: Started (Reading User Variables)
Feb  9 11:46:35 admin: Adaptive QOS: Started (Reading User Variables)
Feb  9 11:46:36 admin: Adaptive QOS: Started (Reading User Variables)
Feb  9 11:46:36 admin: Adaptive QOS: Started (Reading User Variables)
Feb  9 11:46:36 admin: Adaptive QOS: Started (Reading User Variables)
Feb  9 11:46:37 admin: Adaptive QOS: Started (Reading User Variables)
Feb  9 11:46:37 admin: Adaptive QOS: Started (Reading User Variables)
Feb  9 11:46:38 admin: Adaptive QOS: Started (Reading User Variables)
Feb  9 11:46:38 admin: Adaptive QOS: Started (Reading User Variables)
Feb  9 11:46:38 admin: Adaptive QOS: Started (Reading User Variables)
Feb  9 11:46:39 admin: Adaptive QOS: Started (Reading User Variables)
Feb  9 11:46:39 admin: Adaptive QOS: Started (Reading User Variables)
Feb  9 11:46:40 admin: Adaptive QOS: Started (Reading User Variables)
Feb  9 11:46:40 admin: Adaptive QOS: Started (Reading User Variables)
Feb  9 11:46:41 admin: Adaptive QOS: Started (Reading User Variables)
Feb  9 11:46:41 admin: Adaptive QOS: Started (Reading User Variables)
Feb  9 11:46:41 admin: Adaptive QOS: Started (Reading User Variables)
Feb  9 11:46:42 admin: Adaptive QOS: Started (Reading User Variables)
Feb  9 11:46:42 admin: Adaptive QOS: Started (Reading User Variables)
Feb  9 11:46:43 admin: Adaptive QOS: Started (Reading User Variables)
Feb  9 11:46:43 admin: Adaptive QOS: Started (Reading User Variables)
Feb  9 11:46:43 admin: Adaptive QOS: Started (Reading User Variables)
Feb  9 11:46:44 admin: Adaptive QOS: Started (Reading User Variables)
Feb  9 11:46:44 admin: Adaptive QOS: Started (Reading User Variables)
Feb  9 11:46:45 admin: Adaptive QOS: Started (Reading User Variables)
Feb  9 11:46:45 admin: Adaptive QOS: Started (Reading User Variables)
Feb  9 11:46:45 admin: Adaptive QOS: Started (Reading User Variables)
Feb  9 11:46:46 admin: Adaptive QOS: Started (Reading User Variables)
Feb  9 11:46:46 admin: Adaptive QOS: Started (Reading User Variables)
Feb  9 11:46:47 admin: Adaptive QOS: Started (Reading User Variables)
Feb  9 11:46:47 admin: Adaptive QOS: Started (Reading User Variables)

I still have 1 more debugging idea, to see if I am causing it, but I do not have the time this weekend to execute it./

 

[FreshJR]

Just for everyone's information and revisiting a prior issue.

Https traffic is tricky since it's next to impossible to identify it.

With ASUS default settings, traffic matching these filters was directed into net control

Prio12 (0x8009) --> Net Control
Prio21 (0x8012) --> Net Control
Prio22 (0x8013) --> Net Control
Prio23 (0x8014) --> Net Control

Upon use, I saw a lot of HTTPS downloads being classified as 0x8013. So in the script I made this modification shown below

Prio22 (0x8013) --> Others

Going forward, I am seeing that HTTPS downloads are both hitting 0x8013 & 0x8014. The default setup going forward (beta4 and up) will over-riding both rule22 & 23

Prio22 (0x8013) --> Others
Prio23 (0x8014) --> Others

Feel free to change the tag back into ${Net} if you disagree with these change.

These newest changed will match what user MedWatt was reporting.

I may fireup wireshark and see how these https packets actually look. I feel like netcontrol should only be " ACK, SYN, FIN, RST " packets. Hopefully rule21 catches these packets. More investigation in this area is desired, since I noticed NetControl disproportionally swelling.

 

[MysticGold04]

Paste your output of

 cat /jffs/scripts/firewall-start

Sounds like the script isn't launch launching due to an improper entry.

There should be two messages after turning on QOS related to the script.

---

There was a typo in beta2 for the additional error message. An update is going to be released today.

I get the message that it's running the script, but no additional output after that.

I have Merlin 380.69_2 loaded, so I have v1.92 of your script loaded. I think my problem is two-fold. I did not make it executable, and I did not edit it for my networks' proper IP addresses. I will try editing and installing again. Thank you for your work on this.

 

[FreshJR]

The chmod in the original instructions makes it executable.

Slow down and re-read the first post.
If you follow those instructions, everything should be fine.

We can work out any issues after you try installing it again.

I am willing to bet the EOL syntax on your script is not linux compatible. This will not make the script run.

This section explicitly warned about that

Warning if you are opening or editing the script and saving with a texteditor within windows, know that Windows and Linux new lines (ENTER) are different and incompatible, use notepad++. In notepad++ EOL conversion has to be set to Unix/OSX. This is done after opening the script by Edit -> EOL Conversion -> Unix/OSX

A member recently pointed out a built in dos2unix terminal command that will achieve this step, in a direct maner. I will update the original post with this simpler version.

 

[MysticGold04]

... most likely, although I did not edit the script at all. I will slow down a little and read

 

[FreshJR]

Those previous error messages were bugging me. If you get can get those error message, then try testing this script version out. This time around, FakeTC has to be updated, whereas before it was optional. Just follow instructions below.

I cannot reproduce the error messages on my end. I restarted QOS too many times already trying to find them.

If those messages have been squashed then hopefully this can be the last version. It may have been a timing issue, but too many BETA's been released today.


Prerequisites:

You will need putty and pscp, as well as the script. Also you need to be on fw 382+

Spoiler

http://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html

You do not have to install these executables, they are portable. Move them, and the script, into a new folder called Putty on your C:\ drive.

It should look like this:

• C:\Putty\putty.exe
• C:\Putty\pscp.exe
• C:\Putty\FreshJR_QOS_beta4.txt
• C:\Putty\FreshJR_QOS_fakeTC_beta4.txt

Install/Update Guide:
1) turn OFF qos
2) In command prompt

cd C:\putty\
pscp -scp C:\Putty\FreshJR_QOS_beta4.txt admin@192.168.1.1:/jffs/scripts/FreshJR_QOS
pscp -scp C:\Putty\FreshJR_QOS_fakeTC_beta4.txt admin@192.168.1.1:/jffs/scripts/FreshJR_QOS_fakeTC

3) In putty

dos2unix /jffs/scripts/FreshJR_QOS
dos2unix /jffs/scripts/FreshJR_QOS_fakeTC
chmod 0755 /jffs/scripts/FreshJR_QOS
chmod 0755 /jffs/scripts/FreshJR_QOS_fakeTC
sed -i '/FreshJR_QOS/d' /jffs/scripts/firewall-start
/jffs/scripts/FreshJR_QOS install

4) turn ON qos


<< OUTDATED VERSION, NEWER RELEASE AVAILABLE >>

 

[Martineau]

+ downrates=972,1945,5836,3891,2918,972,1945,972
+ uprates=46,92,92,184,138,46,276,46
+ nvram get qos_downrates
+ [ 972,1945,5836,3891,2918,972,1945,972 != ]
[: !=: argument expected
+ nvram get qos_uprates
+ [ 46,92,92,184,138,46,276,46 != ]
[: !=: argument expected
+ logger Adaptive QOS: Parsing QOS config -> No Change
+ echo
+ echo -e \033[1;32m FreshJR QOS was successfully installed \033[0m
 FreshJR QOS was successfully installed
+ echo -e \033[1;32m (Adaptive QOS is turned OFF in router UI) \033[0m
 (Adaptive QOS is turned OFF in router UI)
+ echo

Changed LINE 217 to

if [ ${downrates} != "$(nvram get qos_downrates)" ] || [ ${uprates} != "$(nvram get qos_uprates)" ] ; then

 

[FreshJR]

Changed LINE 217 to

if [ ${downrates} != "$(nvram get qos_downrates)" ] || [ ${uprates} != "$(nvram get qos_uprates)" ] ; then

going to ninja that in there as you are the only download so far.

Thanks!!

I didn't catch that error since my nvram variable already set & turned into a string :/

 

[Martineau]

Thanks!!

I didn't catch that error since my nvram variable already set & turned into a string :/

The joy of programming! - that's why we have Betas

 

[Vexira]

If you dodon mind me saying but with the script i noiced better ping with a download going, and thanks

 

[FreshJR]

If you dodon mind me saying but with the script i noiced better ping with a download going, and thanks

Most likely due to this change. https://www.snbforums.com/threads/r...-and-inner-workings.36836/page-40#post-380272

 

[Yo mans 2]

Those previous error messages were bugging me. If you get can get those error message, then try testing this script version out. This time around, FakeTC has to be updated, whereas before it was optional. Just follow instructions below.

I cannot reproduce the error messages on my end. I restarted QOS too many times already trying to find them.

If those messages have been squashed then hopefully this can be the last version. It may have been a timing issue, but too many BETA's been released today.


Prerequisites:

You will need putty and pscp, as well as the script. Also you need to be on fw 382+

Spoiler

http://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html

You do not have to install these executables, they are portable. Move them, and the script, into a new folder called Putty on your C:\ drive.

It should look like this:

• C:\Putty\putty.exe
• C:\Putty\pscp.exe
• C:\Putty\FreshJR_QOS_beta4.txt
• C:\Putty\FreshJR_QOS_fakeTC_beta4.txt

Install/Update Guide:
1) turn OFF qos
2) In command prompt

cd C:\putty\
pscp -scp C:\Putty\FreshJR_QOS_beta4.txt admin@192.168.1.1:/jffs/scripts/FreshJR_QOS
pscp -scp C:\Putty\FreshJR_QOS_fakeTC_beta4.txt admin@192.168.1.1:/jffs/scripts/FreshJR_QOS_fakeTC

3) In putty

dos2unix /jffs/scripts/FreshJR_QOS
dos2unix /jffs/scripts/FreshJR_QOS_fakeTC
chmod 0755 /jffs/scripts/FreshJR_QOS
chmod 0755 /jffs/scripts/FreshJR_QOS_fakeTC
sed -i '/FreshJR_QOS/d' /jffs/scripts/firewall-start
/jffs/scripts/FreshJR_QOS install

4) turn ON qos


<< CURRENT VERSION FOR FW v382+ >>

Would this work in the stock ASUS firmware 384?

 

[FreshJR]

Would this work in the stock ASUS firmware 384?

no

 

[FreshJR]

This post has been edited a trillion times. It is worthwhile to reread it for latest discoveries!

Here's a quick investigation into Gaming download traffic, specifically Steam content.

steam download
   QOS container: "Others - 0x80130000" - (Would actually be Net Control without script over-riding rule 22)
   dscp: cs1 (0x20 = 0010 00)
   size: 1500
   src ip: varies, depends on the CDN
   IP options header: does not exist
   type: tcp
      sport: 443
      dport: dynamic ( ephemeral ports 49152-65535)
      data: encrytped, no discernible pattern

steam download
   QOS container: "Gaming"
   dscp: cs1 (0x20 = 0010 00)
   size: 1500
   src ip: varies, depends on the CDN
   IP options header: does not exist
   type: tcp
      sport: 80
      dport: dynamic ( ephemeral ports 49152-65535)
      data: no discernible pattern

CS:S
  QOS container: "Gaming"
  dscp: cs1 (0x20 = 0010 00)
  size: variable, but typically 100-600
  src ip: varies on the hoster
  IP options header: does not exist
  type: udp
     sport: dynamic ( typically 27015 27016, but depends on individual hoster configuration )
     dport: 27005
     data: patterns found

Data was parsed using wireshark.

As you can see, game downloads will go into either "Others" or "Gaming".
The results are the same for console game downloads.


As a work around, use these two rules to supplement the existing gaming rule. (Original gaming rule is included in the code snippet below for reference)

realtc filter add dev br0 protocol all prio 2 u32 match ip dst 192.168.1.100/30 match mark 0x80000000 0x8000ffff flowid ${Gaming}        #Gaming - Routes Unidentified Traffic into "Gaming", instead of "Others", for the LAN clients specified
realtc filter add dev br0 protocol all prio 2 u32 match ip dst 192.168.1.100/30 match ip sport 443 0xffff flowid ${Downloads}            #Gaming - Routes port 443 traffic (https), for specified LAN clients, into "Downloads"
realtc filter add dev br0 protocol all prio 2 u32 match mark 0x80080000 0xc03f0000 match ip sport 80 0xffff flowid ${Downloads}        #Gaming - Routes "Gaming" traffic coming from port 80 into "Downloads"

DO NOT FORGET to configure the IP range corresponding to your gaming devices on the gaming rule. 192.168.1.100/30 will have to changed a total of 4 times. Your clients will need manual dhcp assignment.

This should work well for gaming devices since I do not think game are ever hosted on ports 80/443 themselves. Matchmaking is a different story, since that might be done on port 80/443, but we are more interested in prioritizing actual ingame traffic and NOT matchmaking, so it works out perfectly!

For those of you who also want an upload equivalent, try this.

iptables -D POSTROUTING -t mangle -o eth0 -s 192.168.1.100/30 -m mark --mark 0x40000000/0x4000ffff -j MARK --set-mark ${Gaming_mark}         #Gaming - Routes Unidentified Traffic into "Gaming", instead of "Others", from specified LAN devices in rule (line 1/2)
iptables -A POSTROUTING -t mangle -o eth0 -s 192.168.1.100/30 -m mark --mark 0x40000000/0x4000ffff -j MARK --set-mark ${Gaming_mark}        #Gaming - Routes Unidentified Traffic into "Gaming", instead of "Others", from specified LAN devices in rule (line 2/2)
iptables -D POSTROUTING -t mangle -o eth0 -s 192.168.1.100/30 -p tcp --dport 443 -j MARK --set-mark ${Downloads_mark}                         #Gaming - Routes port 443 traffic (https), from specified LAN clients, into "Downloads" (line 1/2)
iptables -A POSTROUTING -t mangle -o eth0 -s 192.168.1.100/30 -p tcp --dport 443 -j MARK --set-mark ${Downloads_mark}                        #Gaming - Routes port 443 traffic (https), from specified LAN clients, into "Downloads" (line 2/2)
realtc filter add dev br0 protocol all prio 2 u32 match mark 0x40080000 0xc03f0000 match ip dport 80 0xffff flowid ${Downloads}                  #Gaming - Routes "Gaming" traffic going to port 80 into "Downloads"

---

More adavnced QOS rules are also possible. It should be possible capture the results of the DNS look ups, insert those results into an ipset, and then mark those IP's in iptables with a desired qos packet mark.

While these advanced rules sounds simple, I am not familiar in how to set this up.

I see that the skynet script and selective routing for netflix threads use this method. This means there are significant resources available on this forum to learn how this method works and is implemented.

I would like to look into this when I have some time, but it won't be done anytime soon as QOS does everything I would like it to do.

Enjoy!!> The gaming rule struggle should be complete.