[Release] FreshJR Adaptive QOS (Improvements / Custom Rules / and Inner workings)

[FreshJR]

It is frustrating seeing my Xbox downloads being treated as game traffic

Any reason this out of the box script rule didn't take care of it?!? (doubt the download was UDP)

            iptables -A POSTROUTING -t mangle -o br0 -m mark --mark 0x80080000/0xc03f0000 -p tcp --sport 80 -j MARK --set-mark ${Default_mark_down}
            iptables -A POSTROUTING -t mangle -o br0 -m mark --mark 0x80080000/0xc03f0000 -p tcp --sport 443 -j MARK --set-mark ${Default_mark_down}

Eg. If Gaming traffic && coming from ports 80/443 -->redirect into "Defaults" container which I repurposed for "Game Downloads"

Very curious as to why it that rule didn't hit.

I had a look at implementing per-device fairness and its very difficult to do alongside Adaptive QoS as requires a complete restructure of how the classes are implemented and becomes very fragile. It could be done with a completely custom script avoiding adaptive QoS, but would be hard to make generic and reusable (where FreshJR does a good job of hooking into Adaptive QoS).

We can implement a per-device equal bandwidth QOS system (with no traffic prioritization) easily if not wanting to use AdaptiveQOS at all.

You said combining both per-device and traffic prioritization made the system fragile.

Out of curiosity, did you try implementing fair-bandwidth allocation and then within each fair-bandwidth queue did you create an HTB to prioritize traffic by type?

(This is a pic of what I am explaining to make it easier to visualize).

EDIT:

I just decided not to even try due to the sheer number of HTB's created. I was worried about performance. But that setup would allocate bandwidth evenly per connected device, and then each devices queue would then be filtered/prioritized based on traffic classification. I just felt like that structure is WAYY to big. ((Is that additional layer of traffic prioritization really necessary with equal per device bandwidth??))

---

The current structure is one where each device is shares a part of the traffic allotted within the 8 traffic categories. It's not bad except asus has glitched the device prioritization feature due to poorly assigned rates. These become a problem when one device has a higher priority than another. This was fixable with the fakeTC approach but I decided not to fight the AdaptiveQOS system any furthur. Fixing the first set of rates was sufficient for a giant improvement.

 

[Gitsum]

Fresh, any plans on adding your script to the amtm project that "thelonelycoder" has?

 

[FreshJR]

Fresh, any plans on adding your script to the amtm project that "thelonelycoder" has?

I haven’t looked into it. I’ll see what I would have to change.

 

[QuikSilver]

Hopefully now it's correct

INBOUND:
iptables -D POSTROUTING -t mangle -o br0 -d 192.168.1.XXX/32 -j MARK --set-mark ${Streaming_mark_down} &> /dev/null
iptables -A POSTROUTING -t mangle -o br0 -d 192.168.1.XXX/32 -j MARK --set-mark ${Streaming_mark_down}

OUTBOUND:
iptables -D POSTROUTING -t mangle -o $wan -s 192.168.1.XXX/32 -j MARK --set-mark ${Streaming_mark_up} &> /dev/null
iptables -A POSTROUTING -t mangle -o $wan -s 192.168.1.XXX/32 -j MARK --set-mark ${Streaming_mark_up}

@FreshJR
Whats the point of the second lines for inbound and outbound? When I try to put those commands in the SSH terminal the second lines give an error.

Edited: OK, so I googled and see the -D command deletes from the table whereas -A appends....but command doesn't work unless the /dev/null portion is added.

 

[FreshJR]

@QuikSilver those lines are for pasting into the script and not directly into ssh. You can read the first post in detail, everything was explained.

The -D line is only there to suppress rules from being created and applied twice (duplicate).

/dev/null suppressed any error messages which is meant to suppress the “cannot delete non existing rule” error message

 

[NGRhodes]

Any reason this out of the box script rule didn't take care of it?!? (doubt the download was UDP)

            iptables -A POSTROUTING -t mangle -o br0 -m mark --mark 0x80080000/0xc03f0000 -p tcp --sport 80 -j MARK --set-mark ${Default_mark_down}
            iptables -A POSTROUTING -t mangle -o br0 -m mark --mark 0x80080000/0xc03f0000 -p tcp --sport 443 -j MARK --set-mark ${Default_mark_down}

Eg. If Gaming traffic && coming from ports 80/443 -->redirect into "Defaults" container which I repurposed for "Game Downloads"

Very curious as to why it that rule didn't hit.



We can implement a per-device equal bandwidth QOS system (with no traffic prioritization) easily if not wanting to use AdaptiveQOS at all.

You said combining both per-device and traffic prioritization made the system fragile.

Out of curiosity, did you try implementing fair-bandwidth allocation and then within each fair-bandwidth queue did you create an HTB to prioritize traffic by type?

(This is a pic of what I am explaining to make it easier to visualize).

EDIT:

I just decided not to even try due to the sheer number of HTB's created. I was worried about performance. But that setup would allocate bandwidth evenly per connected device, and then each devices queue would then be filtered/prioritized based on traffic classification. I just felt like that structure is WAYY to big. ((Is that additional layer of traffic prioritization really necessary with equal per device bandwidth??))

---

The current structure is one where each device is shares a part of the traffic allotted within the 8 traffic categories. It's not bad except asus has glitched the device prioritization feature due to poorly assigned rates. These become a problem when one device has a higher priority than another. This was fixable with the fakeTC approach but I decided not to fight the AdaptiveQOS system any furthur. Fixing the first set of rates was sufficient for a giant improvement.

I was making a comparison to the stock Adaptive QoS when referring to, your script works great.

Yes, that was the approach I was trying to utilise, as device fairness more important to me. One challenge is how to fairly allocate the hashed buckets so maintain the best guarantee bandwidth. Could make an assumption on the max devices on the network and make a fixed value (loosing fairness advantage when only a low number of devices. Managing dynamically would be the fix, but we would need to be able to find all devices active on the network and repeatedly poll and a whole level of complexity not worth the effort IMHO.

I didn't even try setting up all the HTB buckets as I had no ideas how to worth with the categorisation of Adaptive QoS.

 

[QuikSilver]

I actually just tested it and it works! I'm soo thrilled. Thank you FreshJR!

Just one last question: how can I modify the rule to include multiple devices? Right now I have my 3 streaming devices on 192.168.1.102 103 and 104.

Were you able to get this working completely? I also have multiple devices for PSvue. I've made changes to the script. Saved it, uploaded it to router, stopped/started QOS, and still see traffic from said devices NOT going into streaming as expected.

 

[QuikSilver]

@QuikSilver those lines are for pasting into the script and not directly into ssh. You can read the first post in detail, everything was explained.

The -D line is only there to suppress rules from being created and applied twice (duplicate).

/dev/null suppressed any error messages which is meant to suppress the “cannot delete non existing rule” error message

Thanks @FreshJR, I reread it and changed the script similar to what @tmcb82 was shooting for but I still see traffic showing as file downloads coming from the firetv devices. I clicked the QOS radio button off and on, but see no change after the custom script was replaced/uploaded. Does the router need to be rebooted?

 

[TicoMan]

Hi all... I need some help!

I am running Fresh's script - I only changed the priorities (made Video streaming higher), but I am having bandwidth allocations problems.

Thing is, my PS4 is consuming all the bandwidth when it is downloading something, and my IPTV box is not getting enough to work properly.

I think the data is being classified correctly and their priorities are correct:

admin@MyAsusRouter:/jffs/scripts# /jffs/scripts/FreshJR_QOS.sh -appdb "Web Streaming"
Web Streaming
Cat:  Streaming
Mark: 0x80040050 0xc03fffff
Prio: 6

admin@MyAsusRouter:/jffs/scripts# /jffs/scripts/FreshJR_QOS.sh -appdb "Sony Entertainment Network"
Sony Entertainment Network
Cat:  Gaming
Mark: 0x800800e0 0xc03fffff
Prio: 10

I am new to shell scripting and apart from the "Bandwidth Setup" section, everything else looks quite confusing to me.

Any ideas?

 

[FreshJR]

I still see traffic showing as file downloads coming from the firetv devices ... Does the router need to be rebooted?

No reboot required. Double check the IPs. You misconfigured something. The issue is that your traffic rule is misconfigured and not filtering as you expect it too.

Thing is, my PS4 is consuming all the bandwidth when it is downloading something, and my IPTV box is not getting enough to work properly.

Go to the QOS statistics page and check the following:

1) Is your PS4 game download traffic showing up in the grey default category?

2) where is your IPTV box traffic showing up in

(Note: it takes 5 minutes before script modifications are applied. After the 5 minute mark your traffic SHOUDLNT choke. Before the 5 minute mark you will have the crappy stock config)

 

[TicoMan]

By visual observation in the QOS statistics page.

1) Is your PS4 game download traffic showing up in the grey default category?

2) where is your IPTV box traffic showing up in

They were both in the grey default category. I then gave highest to the IPTV box, but it made no difference

 

[FreshJR]

They were both in the grey default category. I then gave highest to the IPTV box, but it made no difference

As a test. Can you

1) turn OFF both the IPTV box && the PS4.

2) Restart QOS (wait 5 min)

3) Turn on the IPTV box && PS4

The issue is that the IPTV box should NOT have been in the grey defaults category with the script modification.

Any connections established before/during the initial 5 minute modification period will remain in original catogores instead of the modified categories. (If that makes sense)

Or was the issue that you were looking at the resulting performance before the 5 minute application period was over since the main script parameters have NOT changed in within the last two years!!
—

If that doesn’t work. I will help you debug further since your results are abnormal.

 

[TicoMan]

As a test. Can you

1) turn OFF both the IPTV box && the PS4.

2) Restart QOS (wait 5 min)

3) Turn on the IPTV box && PS4

The issue is that the IPTV box should NOT have been in the grey defaults category with the script modification.

Any connections established before/during the initial 5 minute modification period will remain in original catogores instead of the modified categories. (If that makes sense)

Or was the issue that you were looking at the resulting performance before the 5 minute application period was over since the main script parameters have NOT changed in within the last two years!!
—

If that doesn’t work. I will help you debug further since your results are abnormal.

Thanks man, still nothing. It is as if the PS4 is his favorite device

 

[QuikSilver]

No reboot required. Double check the IPs. You misconfigured something. The issue is that your traffic rule is misconfigured and not filtering as you expect it too.

I probably did misconfigure. I'm trying to add the devices in a range .114-.120. I've added these lines in their respective parts. Once I get these working as expected, I plan to do something similar to the VOIP device. Then I'm done and bob's my uncle.

Down Rules:
            iptables -D POSTROUTING -t mangle -o br0 -d 192.168.1.114/31 -j MARK --set-mark ${Streaming_mark_down} &> /dev/null 
            iptables -A POSTROUTING -t mangle -o br0 -d 192.168.1.114/31 -j MARK --set-mark ${Streaming_mark_down} &>
           
            iptables -D POSTROUTING -t mangle -o br0 -d 192.168.1.116/30 -j MARK --set-mark ${Streaming_mark_down} &> /dev/null 
            iptables -A POSTROUTING -t mangle -o br0 -d 192.168.1.116/30 -j MARK --set-mark ${Streaming_mark_down} &>
           
            iptables -D POSTROUTING -t mangle -o br0 -d 192.168.1.120/32 -j MARK --set-mark ${Streaming_mark_down} &> /dev/null 
            iptables -A POSTROUTING -t mangle -o br0 -d 192.168.1.120/32 -j MARK --set-mark ${Streaming_mark_down} &>

Up Rules:
            iptables -D POSTROUTING -t mangle -o $wan -s 192.168.1.114/31 -j MARK --set-mark ${Streaming_mark_up} &> /dev/null
            iptables -A POSTROUTING -t mangle -o $wan -s 192.168.1.114/31-j MARK --set-mark ${Streaming_mark_up} &>
           
            iptables -D POSTROUTING -t mangle -o $wan -s 192.168.1.116/30 -j MARK --set-mark ${Streaming_mark_up} &> /dev/null
            iptables -A POSTROUTING -t mangle -o $wan -s 192.168.1.116/30-j MARK --set-mark ${Streaming_mark_up} &>
           
            iptables -D POSTROUTING -t mangle -o $wan -s 192.168.1.120/32 -j MARK --set-mark ${Streaming_mark_up} &> /dev/null
            iptables -A POSTROUTING -t mangle -o $wan -s 192.168.1.120/32-j MARK --set-mark ${Streaming_mark_up} &>

 

[dave14305]

I probably did misconfigure. I'm trying to add the devices in a range .114-.120. I've added these lines in their respective parts. Once I get these working as expected, I plan to do something similar to the VOIP device. Then I'm done and bob's my uncle.

Down Rules:
            iptables -D POSTROUTING -t mangle -o br0 -d 192.168.1.114/31 -j MARK --set-mark ${Streaming_mark_down} &> /dev/null
            iptables -A POSTROUTING -t mangle -o br0 -d 192.168.1.114/31 -j MARK --set-mark ${Streaming_mark_down} &>
          
            iptables -D POSTROUTING -t mangle -o br0 -d 192.168.1.116/30 -j MARK --set-mark ${Streaming_mark_down} &> /dev/null
            iptables -A POSTROUTING -t mangle -o br0 -d 192.168.1.116/30 -j MARK --set-mark ${Streaming_mark_down} &>
          
            iptables -D POSTROUTING -t mangle -o br0 -d 192.168.1.120/32 -j MARK --set-mark ${Streaming_mark_down} &> /dev/null
            iptables -A POSTROUTING -t mangle -o br0 -d 192.168.1.120/32 -j MARK --set-mark ${Streaming_mark_down} &>

Up Rules:
            iptables -D POSTROUTING -t mangle -o $wan -s 192.168.1.114/31 -j MARK --set-mark ${Streaming_mark_up} &> /dev/null
            iptables -A POSTROUTING -t mangle -o $wan -s 192.168.1.114/31-j MARK --set-mark ${Streaming_mark_up} &>
          
            iptables -D POSTROUTING -t mangle -o $wan -s 192.168.1.116/30 -j MARK --set-mark ${Streaming_mark_up} &> /dev/null
            iptables -A POSTROUTING -t mangle -o $wan -s 192.168.1.116/30-j MARK --set-mark ${Streaming_mark_up} &>
          
            iptables -D POSTROUTING -t mangle -o $wan -s 192.168.1.120/32 -j MARK --set-mark ${Streaming_mark_up} &> /dev/null
            iptables -A POSTROUTING -t mangle -o $wan -s 192.168.1.120/32-j MARK --set-mark ${Streaming_mark_up} &>

You’ve got some spacing issues before the -j in your last rules. Check those.

 

[FreshJR]

@QuikSilver also the -A rules shouldn't really have the "&>" at the end of them, but having that extra bit shouldn't have them fail.

Can you post the output of:

 iptables -vL -t mangle

 

[QuikSilver]

@QuikSilver also the -A rules shouldn't really have the "&>" at the end of them, but having that extra bit shouldn't have them fail.

Can you post the output of:

 iptables -vL -t mangle

Chain PREROUTING (policy ACCEPT 1707 packets, 226K bytes)
 pkts bytes target     prot opt in     out     source               destination                                                                                         

Chain INPUT (policy ACCEPT 1560 packets, 185K bytes)
 pkts bytes target     prot opt in     out     source               destination                                                                                         

Chain FORWARD (policy ACCEPT 147 packets, 40902 bytes)
 pkts bytes target     prot opt in     out     source               destination                                                                                         

Chain OUTPUT (policy ACCEPT 1504 packets, 1139K bytes)
 pkts bytes target     prot opt in     out     source               destination                                                                                         

Chain POSTROUTING (policy ACCEPT 1650 packets, 1179K bytes)
 pkts bytes target     prot opt in     out     source               destination                                                                                         

Chain BWDPI_FILTER (0 references)
 pkts bytes target     prot opt in     out     source               destination

 

[FreshJR]

@QuikSilver

None of the scripts iptables rules were applied (POSTROUTING is completely empty)

Are you sure you followed "Manual Install (for users wanting to make modifications):" instructions exactly as written>?

Anything abnormal in system log?

 

[QuikSilver]

@QuikSilver

None of the scripts iptables rules were applied (POSTROUTING is completely empty)

Are you sure you followed "Manual Install (for users wanting to make modifications):" instructions exactly as written>?

Anything abnormal in system log?

I read that and only difference was I was using WinSCP. I went back and did exactly as stated and now get....

Chain PREROUTING (policy ACCEPT 3156 packets, 460K bytes)
 pkts bytes target     prot opt in     out     source               destination
  394 69297 BWDPI_FILTER  udp  --  eth0   any     anywhere             anywhere

Chain INPUT (policy ACCEPT 2366 packets, 239K bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain FORWARD (policy ACCEPT 790 packets, 221K bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 2320 packets, 1144K bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain POSTROUTING (policy ACCEPT 3074 packets, 1363K bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 MARK       udp  --  any    br0     anywhere             anywhere             udp spt:500 MARK set 0x80060001
    0     0 MARK       udp  --  any    br0     anywhere             anywhere             udp spt:4500 MARK set 0x80060001
    0     0 MARK       udp  --  any    br0     anywhere             anywhere             udp dpts:16384:16415 MARK set 0x80060001
    0     0 MARK       tcp  --  any    br0     anywhere             anywhere             mark match 0x80080000/0xc03f0000 tcp spt:www MARK set 0x803f0001
    0     0 MARK       tcp  --  any    br0     anywhere             anywhere             mark match 0x80080000/0xc03f0000 tcp spt:https MARK set 0x803f0001
    3   228 MARK       all  --  any    br0     anywhere             Ooma_Telo            MARK set 0x80060001
  631  230K MARK       all  --  any    br0     anywhere             FireTV-LivingRm/31   MARK set 0x80040001
   10  1424 MARK       all  --  any    br0     anywhere             FireTV-L/30        MARK set 0x80040001
    0     0 MARK       all  --  any    br0     anywhere             192.168.1.120        MARK set 0x80040001
    0     0 MARK       udp  --  any    eth0    anywhere             anywhere             udp dpt:500 MARK set 0x40060001
    0     0 MARK       udp  --  any    eth0    anywhere             anywhere             udp dpt:4500 MARK set 0x40060001
    0     0 MARK       udp  --  any    eth0    anywhere             anywhere             udp spts:16384:16415 MARK set 0x40060001
    0     0 MARK       tcp  --  any    eth0    anywhere             anywhere             mark match 0x40080000/0xc03f0000 tcp spt:www MARK set 0x403f0001
    0     0 MARK       tcp  --  any    eth0    anywhere             anywhere             mark match 0x40080000/0xc03f0000 tcp spt:https MARK set 0x403f0001
    3   228 MARK       all  --  any    eth0    Ooma_Telo            anywhere             MARK set 0x40060001
   34  3818 MARK       all  --  any    eth0    FireTV-LivingRm/31   anywhere             MARK set 0x40040001
    8   650 MARK       all  --  any    eth0    FireTV-L/30            anywhere             MARK set 0x40040001
    0     0 MARK       all  --  any    eth0    192.168.1.120        anywhere             MARK set 0x40040001

Chain BWDPI_FILTER (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DROP       udp  --  eth0   any     anywhere             anywhere             udp spt:bootpc dpt:bootps
    4  1488 DROP       udp  --  eth0   any     anywhere             anywhere             udp spt:bootps dpt:bootpc

 

[FreshJR]

I read that and only difference was I was using WinSCP. I went back and did exactly as stated and now get....

look goods. should be working as expected