[Release] FreshJR Adaptive QOS (Improvements / Custom Rules / and Inner workings)

[MarCoMLXXV]

Did I hear someone say...

Are you on v1.91 by chance?

Thanks for the update @FreshJR! Uploading to my RT-AC68U now

 

[FreshJR]

Nothing major has changed

Just that missing https rule from ASUS, so partial https traffic does not bypass QOS anymore + some minor comment reorganization.

Many users were noticing https traffic bypass. I decided to stick unidentified https traffic into web surfing. You can change it to your needs. Still not sure whats the best location

 

[MarCoMLXXV]

'Minor reorganization' ?!? I think you did a great job on clarifying what part of the script does what and how to edit things.

I did some minor tweaking in the values per container to suit my needs (I don't need VOIP, as the landline is connected to the bridge router/modem provided by my ISP, so it doesn't influence my available bandwidth, so I can free distribute everything above 5% over the other buckets). Set max bandwidth to 14/146 (instead of 15/150 according to my plan), found that that's giving me the best results. Final testdrive for tonight was highly satisfying again:

Bufferbloat down is at 22, bufferbloat up is at 5 which are among the lowest values I've ever had, so happy user here Did the ping test from the router to the dutch Google server while performing speed test, only had 3 or 4 spikes above 15ms (30-34,ms which I consider neglectible), so I have nothing to complain about. Keep up the good work!

 

[Jack Yaz]

'Minor reorganization' ?!? I think you did a great job on clarifying what part of the script does what and how to edit things.

I did some minor tweaking in the values per container to suit my needs (I don't need VOIP, as the landline is connected to the bridge router/modem provided by my ISP, so it doesn't influence my available bandwidth, so I can free distribute everything above 5% over the other buckets). Set max bandwidth to 14/146 (instead of 15/150 according to my plan), found that that's giving me the best results. Final testdrive for tonight was highly satisfying again:

Bufferbloat down is at 22, bufferbloat up is at 5 which are among the lowest values I've ever had, so happy user here Did the ping test from the router to the dutch Google server while performing speed test, only had 3 or 4 spikes above 15ms (30-34,ms which I consider neglectible), so I have nothing to complain about. Keep up the good work!

I'd love those bufferbloat, but thanks to Virgin, I get latency on the modem end

http://community.virginmedia.com/t5...TG2492LG-and-CGNV4-Latency-Cause/td-p/3271492

 

[brummygit]

Hi @FreshJR I just spotted that your installation instructions need a quick update for v1.91 script. I had both v1.9 and v1.91 in my directory on my PC and followed the instructions to copy and paste commands blindly - you still copy v1.9 in that case. My bad, but may catch others out.

Also I'm assuming that all you need to do is the
pscp -scp C:\Putty\FreshJR_QOS_v1_91.txt admin@192.168.1.1:/jffs/scripts/FreshJR_QOS
in order to update from v1.9 to v1.91?

 

[FreshJR]

@brummygit

caught that too, sometimes when I edit posts too much they get tagged for moderator approval.

When I clicked save on the changelog and noticed to update the install instructions, the post was already invisible and locked. It is fixed now

 

[brummygit]

No problem for me, and thanks for your hard work. It's made a real difference here [emoji106]


Sent from my iPhone using Tapatalk

 

[Lacrocious]

@Lacrocious / Ron
Almost but not quite.....

Thanks FreshJR,
I feel silly leaving the # in the code I posted. Yes - what I was loading to my router was without the comment char. I am familiar with networking, but haven't gone anywhere near the level you have and am still learning about the filter formats you are using.

I guess I didn't understand the ${tc} vs. iptables. I did switch the iptables to a "-d" destination entry so it picks up traffic destined for the BackBlaze IP ranges. It does seem to pick up some of the backup traffic as File Transfer. I think it is still missing some traffic - maybe due to me missing the first of the pairs of iptables rules. I went with the iptables because of your comment under the Upload/outgoing traffic section for template rule 2 & 3 "NOT WORKING/USE IPTABLES..." I guess I generalized your Not Wworking comment for the whole thing, not just your example like that has a LAN IP vs. a WAN IP.

I am on 1.91 and yes, the areas are marked better than the previous versions I reviewed. I will give it a go over the next day or so and post some results.

One minor suggestion - I added the following line between the last to fi's to which log when this script is complete after restarting QOS. It is nice to know when it is done. You could add that to your template script if you think it would help others.

logger "Adaptive QOS: Modification Script Complete"

Thank you for your help and for the script in general.
- Ron

 

[ColinTaylor]

@FreshJR This may be a stupid question but in post#1, Example Setup 1, under "What would happen...", why is Downloads = 60 mbps and not 75 mbps?

 

[FreshJR]

@FreshJR This may be a stupid question but in post#1, Example Setup 1, under "What would happen...", why is Downloads = 60 mbps and not 75 mbps?

You are correct! I made a mistake.

So here is the setup

Downloads container 1 40 mbps rate / 100 mbps ciel
Web Surfing container 2 10 mbps rate / 100 mbps ciel
Streaming container 3 20 mbps rate / 100 mbps ciel

here is what happens step by step.

Downloads is offered 40mbps, uses 40mbps
Web Surfing is offered 10mbps, uses 5mbps
Streaming is offered 20 mbps, 20 mbps
**at this point 35mbps is free**
Downloads is offered remaining 35

Downloads has a total of 75mbps now

 

[FreshJR]

I guess I didn't understand the ${tc} vs. iptables. I did switch the iptables to a "-d" destination
- Ron

Quick correction.

In iptables the -d is to delete an entry, and -a is to append a new entry.
-d does NOT mean destination

If you, or anyone, needs to use one of the two types iptables examples, you need to copy both -d and -a entries per rule created.

Everytime the script runs

-d deletes the existing rule (if it exists)
-a recreates the rule

the reason we cannot use -a ONLY, is because iptables would create a new rule everytime it ran and you would have many additional duplicate rules.

As to the difference between iptables & tc (traffic control).

iptables marks the packets by reading header contents
traffic control reads marked packets, sorts those packets by mark, and bandwidth limits and polices those packets.

We are also further flexing traffic control by giving it the same task as iptables when dealing with our custom rules, that is we are instructing traffic control to read packet header contents, and then sort those packets by what it finds.

The reason we need to use iptables for two specific rules is because iptables reads packets at a different stage in the pipeline than traffic control. For the two rules marked as not working, that information is not in the packet when traffic control unwraps it, so we perform it at a different time with iptables.

 

[ColinTaylor]

In iptables the -d is to delete an entry, and -a is to append a new entry.
-d does NOT mean destination

Are you talking about plain iptables commands? Because -d is destination, -D is delete and -A is append.

 

[FreshJR]

Are you talking about plain iptables commands? Because -d is destination, -D is delete and -A is append.

ahh correct, correct.

The initial -A, -D is append and delete.
The other -s, -d is source and destination

Yes there should be no reason why your iptables equivalent of the tc template rule shouldn't work!

I skipped over reading the rule in depth and thought you simply took the LAN source ip template as in shown in the example instead of switching source/destination flags.

If you apply the defaults mark with iptables, then I think tc would then redirect it into other category due to our other changes. If you want in the defaults catagory just use the tc equivalent.

I also perfer the tc equivalent to avoid the whole "hacky" delete (-D) before append (-A) workaround I have going on in the script.

Feel free to keep using the tc equivalent, but copy paste it with the -D flag before your -A application to prevent duplicates. Sometimes we got lost with walls of text instead of face to face communication.

 

[Lacrocious]

@ColinTayor - Thank you, you beat me to pointing out I was using -s/-d not -A/-D.

@FreshJR - I am glad I was on the closer to the right track with the iptables than originally thought. I was winging it a bit, but thought I had it mostly working. I used the Append command w/o Delete, many times to try to get it working properly. Are my iptables messed up? Do I need to do a bunch of deletes? I did "iptables -L" to list them, and I don't see my entries. Thoughts?

I am working on transitioning to the tc approach.

 

[john9527]

I used the Append command w/o Delete, many times to try to get it working properly. Are my iptables messed up? Do I need to do a bunch of deletes? I did "iptables -L" to list them, and I don't see my entries. Thoughts?

iptables actually has 4 different tables.....raw, mangle, nat and filter (the default if not specified). You were making entries in the mangle table, so to see them use
iptables -t mangle -L

There is detail missing with just the -L flag....I'd recommend using -nvL

I'd also recommend getting an iptables flow chart....there are many on the net. This just happened to be the first one when I just did a search...
http://stuffphilwrites.com/2014/09/iptables-processing-flowchart/

 

[Lacrocious]

iptables actually has 4 different tables.....raw, mangle, nat and filter (the default if not specified). You were making entries in the mangle table, so to see them use
iptables -t mangle -L

There is detail missing with just the -L flag....I'd recommend using -nvL

I'd also recommend getting an iptables flow chart....there are many on the net. This just happened to be the first one when I just did a search...
http://stuffphilwrites.com/2014/09/iptables-processing-flowchart/

Thanks for the pointer to the flowchart. I tried your suggested addition to the -L list command and I can see the mangle table as well as the POSTROUTING chain and there are no entries listed. I will dig deeper tomorrow into iptables and the flowcharts to see what I can see.
- Ron

 

[FreshJR]

When adaptive qos is toggled off and on it wipes and recreates both iptables and traffic control environments.

It probably cleared itself out during your testing and toggling .

But in an scripted environment I decided to execute deletes before appends to prevent issues.

This is a thing to watch out for if you have your own iptables or tc changes applied outside a firewall start trigger

 

[FreshJR]

EDIT: URGENT, v1.91 had a typo mistake preventing it from working

rate was mistakenly passed to traffic control as XXX Kbit Kbit in this version

The version I uploaded accidentally had Kbit was appended twice when moving some stuff around.

It is fixed now in v1.92

Sorry

 

[Jack Yaz]

EDIT: URGENT, v1.91 had a typo mistake preventing it from working

rate was mistakenly passed to traffic control as XXX Kbit Kbit in this version

The version I uploaded accidentally had Kbit was appended twice when moving some stuff around.

It is fixed now in v1.92

Sorry

Which line(s)?

 

[MarCoMLXXV]

EDIT: URGENT, v1.91 had a typo mistake preventing it from working

rate was mistakenly passed to traffic control as XXX Kbit Kbit in this version

The version I uploaded accidentally had Kbit was appended twice when moving some stuff around.

It is fixed now in v1.92

Sorry

@FreshJR Thanks for the quick fix (even though I had no apparent issues with it?)

Where can v1.92 be downloaded? And may I suggest you remove the download link to v1.91 asap from the first posts?