FlexQoS FlexQoS 1.0 - Flexible QoS Enhancement Script for Adaptive QoS

[JohnD5000]

So, I had FlexQoS loaded, a speedtest run on the modem (spdMerlin) results in around 235-240, but speedtest run on my laptop = 5 -6. Set bandwidth setting from manual (220) to automatic and speedtest returns to more normal number Set QoS off, again, more normal. Uninstall FlexQoS and use standard Merlin QoS and set manual (220) low speeds (around 4). Bandwidth setting to auto back to 235? What am I doing wrong? Speedtest is Mbps and Merlin setting is Mb/s. Do I need to adjust for that?

OK, I'm an idiot, I was putting UL number in DL, and DL number in UL

 

[andresmorago]

No, but re-enter them in all CAPS and it should show the real name.

after adding the new speedtest marks on caps, they still show under websurfing. both in chart and table



my debug just in case

Spoiler

FlexQoS v0.8.4 released 06/30/2020

Debug:

Undf Prio: 2
Undf FlowID: 1:15
Classes Present: 8
Down Band: 150528
Up Band  : 9728
***********
Net Control = 1:10
Work-From-Home = 1:11
Gaming = 1:17
Others = 1:15
Web Surfing = 1:12
Streaming = 1:13
Downloads = 1:14
Defaults = 1:16
***********
Downrates -- 22579, 30105, 30105, 15052, 15052, 22579, 7526, 7526
Downceils -- 150528, 150528, 150528, 150528, 150528, 150528, 150528, 150528
Downbursts -- 7992b, 36788b, 17588b, 7994b, 6397b, 4795b, 3198b, 3198b
DownCbursts -- 187125b, 187125b, 187125b, 187125b, 187125b, 187125b, 187125b, 187125b
***********
Uprates -- 1459, 1945, 1945, 972, 486, 1945, 486, 486
Upceils -- 9728, 9728, 9728, 9728, 9728, 9728, 9728, 9728
Upbursts -- 3198b, 3198b, 3198b, 3198b, 3198b, 3198b, 3198b, 3198b
UpCbursts -- 11196b, 11196b, 11196b, 11196b, 11196b, 11196b, 11196b, 11196b
iptables settings: <>>udp>>500,4500>>3<>>udp>16384:16415>>>3<>>tcp>>119,563>>5<>>tcp>>80,443>08****>7<>>udp>>3478:3481>>3<10.0.0.2>>tcp>51416>>>2<10.0.0.2>>both>51415>>>5
iptables -D POSTROUTING -t mangle -o br0   -p udp  -m multiport  --sports 500,4500  -j MARK --set-mark 0x80060001 >/dev/null 2>&1
iptables -A POSTROUTING -t mangle -o br0   -p udp  -m multiport  --sports 500,4500  -j MARK --set-mark 0x80060001
iptables -D POSTROUTING -t mangle -o eth0   -p udp  -m multiport  --dports 500,4500  -j MARK --set-mark 0x40060001 >/dev/null 2>&1
iptables -A POSTROUTING -t mangle -o eth0   -p udp  -m multiport  --dports 500,4500  -j MARK --set-mark 0x40060001
iptables -D POSTROUTING -t mangle -o br0   -p udp  --dport 16384:16415   -j MARK --set-mark 0x80060001 >/dev/null 2>&1
iptables -A POSTROUTING -t mangle -o br0   -p udp  --dport 16384:16415   -j MARK --set-mark 0x80060001
iptables -D POSTROUTING -t mangle -o eth0   -p udp  --sport 16384:16415   -j MARK --set-mark 0x40060001 >/dev/null 2>&1
iptables -A POSTROUTING -t mangle -o eth0   -p udp  --sport 16384:16415   -j MARK --set-mark 0x40060001
iptables -D POSTROUTING -t mangle -o br0   -p tcp  -m multiport  --sports 119,563  -j MARK --set-mark 0x80030001 >/dev/null 2>&1
iptables -A POSTROUTING -t mangle -o br0   -p tcp  -m multiport  --sports 119,563  -j MARK --set-mark 0x80030001
iptables -D POSTROUTING -t mangle -o eth0   -p tcp  -m multiport  --dports 119,563  -j MARK --set-mark 0x40030001 >/dev/null 2>&1
iptables -A POSTROUTING -t mangle -o eth0   -p tcp  -m multiport  --dports 119,563  -j MARK --set-mark 0x40030001
iptables -D POSTROUTING -t mangle -o br0   -p tcp  -m multiport  --sports 80,443 -m mark --mark 0x80080000/0xc03f0000 -j MARK --set-mark 0x803f0001 >/dev/null 2>&1
iptables -A POSTROUTING -t mangle -o br0   -p tcp  -m multiport  --sports 80,443 -m mark --mark 0x80080000/0xc03f0000 -j MARK --set-mark 0x803f0001
iptables -D POSTROUTING -t mangle -o eth0   -p tcp  -m multiport  --dports 80,443 -m mark --mark 0x40080000/0xc03f0000 -j MARK --set-mark 0x403f0001 >/dev/null 2>&1
iptables -A POSTROUTING -t mangle -o eth0   -p tcp  -m multiport  --dports 80,443 -m mark --mark 0x40080000/0xc03f0000 -j MARK --set-mark 0x403f0001
iptables -D POSTROUTING -t mangle -o br0   -p udp   --sport 3478:3481  -j MARK --set-mark 0x80060001 >/dev/null 2>&1
iptables -A POSTROUTING -t mangle -o br0   -p udp   --sport 3478:3481  -j MARK --set-mark 0x80060001
iptables -D POSTROUTING -t mangle -o eth0   -p udp   --dport 3478:3481  -j MARK --set-mark 0x40060001 >/dev/null 2>&1
iptables -A POSTROUTING -t mangle -o eth0   -p udp   --dport 3478:3481  -j MARK --set-mark 0x40060001
iptables -D POSTROUTING -t mangle -o br0  -d 10.0.0.2  -p tcp  --dport 51416   -j MARK --set-mark 0x80040001 >/dev/null 2>&1
iptables -A POSTROUTING -t mangle -o br0  -d 10.0.0.2  -p tcp  --dport 51416   -j MARK --set-mark 0x80040001
iptables -D POSTROUTING -t mangle -o eth0  -s 10.0.0.2  -p tcp  --sport 51416   -j MARK --set-mark 0x40040001 >/dev/null 2>&1
iptables -A POSTROUTING -t mangle -o eth0  -s 10.0.0.2  -p tcp  --sport 51416   -j MARK --set-mark 0x40040001
iptables -D POSTROUTING -t mangle -o br0  -d 10.0.0.2  -p tcp  --dport 51415   -j MARK --set-mark 0x80030001 >/dev/null 2>&1
iptables -A POSTROUTING -t mangle -o br0  -d 10.0.0.2  -p tcp  --dport 51415   -j MARK --set-mark 0x80030001
iptables -D POSTROUTING -t mangle -o br0  -d 10.0.0.2  -p udp  --dport 51415   -j MARK --set-mark 0x80030001 >/dev/null 2>&1
iptables -A POSTROUTING -t mangle -o br0  -d 10.0.0.2  -p udp  --dport 51415   -j MARK --set-mark 0x80030001
iptables -D POSTROUTING -t mangle -o eth0  -s 10.0.0.2  -p tcp  --sport 51415   -j MARK --set-mark 0x40030001 >/dev/null 2>&1
iptables -A POSTROUTING -t mangle -o eth0  -s 10.0.0.2  -p tcp  --sport 51415   -j MARK --set-mark 0x40030001
iptables -D POSTROUTING -t mangle -o eth0  -s 10.0.0.2  -p udp  --sport 51415   -j MARK --set-mark 0x40030001 >/dev/null 2>&1
iptables -A POSTROUTING -t mangle -o eth0  -s 10.0.0.2  -p udp  --sport 51415   -j MARK --set-mark 0x40030001
appdb rules: <000000>6<00006B>6<0D0007>5<0D0086>5<0D00A0>5<12003F>4<1400C2>5<0D00DE>5<1400B9>5<0D0023>2<0D007E>2<0D00D1>2<0D0054>2<13****>4<14****>4<1A****>5
realtc filter del dev br0 parent 1: prio 2 > /dev/null 2>&1
realtc filter del dev eth0 parent 1: prio 2 > /dev/null 2>&1
realtc filter add dev br0 protocol all prio 2 u32 match mark 0x80000000 0xc000ffff flowid 1:15
realtc filter add dev eth0 protocol all prio 2 u32 match mark 0x40000000 0xc000ffff flowid 1:15
realtc filter add dev br0 protocol all prio 2 u32 match mark 0x8000006B 0xc03fffff flowid 1:15
realtc filter add dev eth0 protocol all prio 2 u32 match mark 0x4000006B 0xc03fffff flowid 1:15
realtc filter add dev br0 protocol all prio 15 u32 match mark 0x800D0007 0xc03fffff flowid 1:14
realtc filter add dev eth0 protocol all prio 15 u32 match mark 0x400D0007 0xc03fffff flowid 1:14
realtc filter add dev br0 protocol all prio 15 u32 match mark 0x800D0086 0xc03fffff flowid 1:14
realtc filter add dev eth0 protocol all prio 15 u32 match mark 0x400D0086 0xc03fffff flowid 1:14
realtc filter add dev br0 protocol all prio 15 u32 match mark 0x800D00A0 0xc03fffff flowid 1:14
realtc filter add dev eth0 protocol all prio 15 u32 match mark 0x400D00A0 0xc03fffff flowid 1:14
realtc filter add dev br0 protocol all prio 20 u32 match mark 0x8012003F 0xc03fffff flowid 1:12
realtc filter add dev eth0 protocol all prio 20 u32 match mark 0x4012003F 0xc03fffff flowid 1:12
realtc filter add dev br0 protocol all prio 22 u32 match mark 0x801400C2 0xc03fffff flowid 1:14
realtc filter add dev eth0 protocol all prio 22 u32 match mark 0x401400C2 0xc03fffff flowid 1:14
realtc filter add dev br0 protocol all prio 15 u32 match mark 0x800D00DE 0xc03fffff flowid 1:14
realtc filter add dev eth0 protocol all prio 15 u32 match mark 0x400D00DE 0xc03fffff flowid 1:14
realtc filter add dev br0 protocol all prio 22 u32 match mark 0x801400B9 0xc03fffff flowid 1:14
realtc filter add dev eth0 protocol all prio 22 u32 match mark 0x401400B9 0xc03fffff flowid 1:14
realtc filter add dev br0 protocol all prio 15 u32 match mark 0x800D0023 0xc03fffff flowid 1:13
realtc filter add dev eth0 protocol all prio 15 u32 match mark 0x400D0023 0xc03fffff flowid 1:13
realtc filter add dev br0 protocol all prio 15 u32 match mark 0x800D007E 0xc03fffff flowid 1:13
realtc filter add dev eth0 protocol all prio 15 u32 match mark 0x400D007E 0xc03fffff flowid 1:13
realtc filter add dev br0 protocol all prio 15 u32 match mark 0x800D00D1 0xc03fffff flowid 1:13
realtc filter add dev eth0 protocol all prio 15 u32 match mark 0x400D00D1 0xc03fffff flowid 1:13
realtc filter add dev br0 protocol all prio 15 u32 match mark 0x800D0054 0xc03fffff flowid 1:13
realtc filter add dev eth0 protocol all prio 15 u32 match mark 0x400D0054 0xc03fffff flowid 1:13
realtc filter del dev br0 parent 1: prio 22 > /dev/null 2>&1
realtc filter del dev eth0 parent 1: prio 22 > /dev/null 2>&1
realtc filter add dev br0 protocol all prio 22 u32 match mark 0x80130000 0xc03f0000 flowid 1:12
realtc filter add dev eth0 protocol all prio 22 u32 match mark 0x40130000 0xc03f0000 flowid 1:12
realtc filter del dev br0 parent 1: prio 23 > /dev/null 2>&1
realtc filter del dev eth0 parent 1: prio 23 > /dev/null 2>&1
realtc filter add dev br0 protocol all prio 23 u32 match mark 0x80140000 0xc03f0000 flowid 1:12
realtc filter add dev eth0 protocol all prio 23 u32 match mark 0x40140000 0xc03f0000 flowid 1:12
realtc filter del dev br0 parent 1: prio 2 > /dev/null 2>&1
realtc filter del dev eth0 parent 1: prio 2 > /dev/null 2>&1
realtc filter add dev br0 protocol all prio 2 u32 match mark 0x801A0000 0xc03f0000 flowid 1:14
realtc filter add dev eth0 protocol all prio 2 u32 match mark 0x401A0000 0xc03f0000 flowid 1:14
andresmorago@RT-AC3100-0548:/tmp/home/root#

 

[dave14305]

after adding the new speedtest marks on caps, they still show under websurfing. both in chart and table

I think the problem is when I “fixed” the iptables rules behavior, I broke the AppDB rules behavior, AND I made the wildcard rules appear last in the list (based on the 0.8.3 behavior), but that means now that the LAST appdb rule will apply if both a specific Mark AND a wildcard Mark exist for the same category (e.g. 14).

I’ll need to think about how to fix that.

But if you say it appears in the chart incorrectly as well, then something else must be broken. Run:

tc -s filter show dev eth0 | grep 0x4014 -B1

 

[Sinner]

everytime i try n post a msg its telling me ive been blocked lol... maybe bug.. maybe intentional.. who knows on this forum.. im waiting 2yrs for the nimrod that runs it to explain to me why i cant post a photo as well... anyway @dave14305 think you could take a peek at my stats asp file quick sometime?

oh wow that one worked..

 

[Vexira]

everytime i try n post a msg its telling me ive been blocked lol... maybe bug.. maybe intentional.. who knows on this forum.. im waiting 2yrs for the nimrod that runs it to explain to me why i cant post a photo as well... anyway @dave14305 think you could take a peek at my stats asp file quick sometime?

oh wow that one worked..

I had to post images to a file hosting site then post the sharing url.

 

[Sinner]

I had to post images to a file hosting site then post the sharing url.

I was just trying to post my stats asp file in a code box.. maybe that was it. Got a HUGE popup msg saying "SORRY YOUVE BEEN BLOCKED" with big X's and sht lol

 

[andresmorago]

I think the problem is when I “fixed” the iptables rules behavior, I broke the AppDB rules behavior, AND I made the wildcard rules appear last in the list (based on the 0.8.3 behavior), but that means now that the LAST appdb rule will apply if both a specific Mark AND a wildcard Mark exist for the same category (e.g. 14).

I’ll need to think about how to fix that.

But if you say it appears in the chart incorrectly as well, then something else must be broken. Run:

tc -s filter show dev eth0 | grep 0x4014 -B1

hi dave
this is the output

ASUSWRT-Merlin RT-AC3100 384.18_0 Sun Jun 28 17:58:08 UTC 2020

andresmorago@RT-AC3100-0548:/tmp/home/root# tc -s filter show dev eth0 | grep 0x4014 -B1
filter parent 1: protocol all pref 23 u32 fh 829::800 order 2048 key ht 829 bkt 0 flowid 1:12  (rule hit 891924 success 54256)
  mark 0x40140000 0xc03f0000 (success 54256)

 

[chris.at]

After upgrading and automatic resorting the appdb rules, my DoT appdb rule shifting 1400C5 to NetControl seems to get overwritten and listed as WebSurfing (14**** behind it overrules 1400C5 at the moment) in the chart. Before when my rule was the last it was listed as NetControl. As far as I understand from your previous post there still is a discrepancy between display and real classification so no idea if it's NetControl or WebSurfing in reality. I already read your answer to the similar speedtest problem some posts above but wanted to confirm that it's not a single user or class effect. Thank you for keeping it updated btw, great job so far

Edit: same output as andresmorago:

admin@router:/tmp/home/root# tc -s filter show dev eth0 | grep 0x4014 -B1
filter parent 1: protocol all pref 23 u32 fh 829::800 order 2048 key ht 829 bkt 0 flowid 1:13  (rule hit 1746 success 545)
  mark 0x40140000 0xc03f0000 (success 545)

But on the integrated classification page DoT is shown as NetControl. So, who's right?

 

[dave14305]

hi dave
this is the output

ASUSWRT-Merlin RT-AC3100 384.18_0 Sun Jun 28 17:58:08 UTC 2020

andresmorago@RT-AC3100-0548:/tmp/home/root# tc -s filter show dev eth0 | grep 0x4014 -B1
filter parent 1: protocol all pref 23 u32 fh 829::800 order 2048 key ht 829 bkt 0 flowid 1:12  (rule hit 891924 success 54256)
  mark 0x40140000 0xc03f0000 (success 54256)

It looks like your tc rules for specific 14 marks aren’t there. So maybe we’re back to needing a longer delay in the startup. Run flexqos check and then run the tc command again to confirm they load correctly.

 

[dave14305]

anyway @dave14305 think you could take a peek at my stats asp file quick sometime?

No promises, but if you post it somewhere (pastebin, github, etc.) and explain the problem someone else might also be able to help. I’ve got a few of my own problems at the moment.

 

[dave14305]

After upgrading and automatic resorting the appdb rules, my DoT appdb rule shifting 1400C5 to NetControl seems to get overwritten and listed as WebSurfing (14**** behind it overrules 1400C5 at the moment) in the chart. Before when my rule was the last it was listed as NetControl. As far as I understand from your previous post there still is a discrepancy between display and real classification so no idea if it's NetControl or WebSurfing in reality. I already read your answer to the similar speedtest problem some posts above but wanted to confirm that it's not a single user or class effect. Thank you for keeping it updated btw, great job so far

Edit: same output as andresmorago:

admin@router:/tmp/home/root# tc -s filter show dev eth0 | grep 0x4014 -B1
filter parent 1: protocol all pref 23 u32 fh 829::800 order 2048 key ht 829 bkt 0 flowid 1:13  (rule hit 1746 success 545)
  mark 0x40140000 0xc03f0000 (success 545)

But on the integrated classification page DoT is shown as NetControl. So, who's right?

The Merlin page will show the color based on the original Mark, so it wouldn’t be aware that we changed it after-the-fact. But the fact that you also show just one rule with the tc grep makes me wonder what’s going on. If you run flexqos check does it help?

 

[dave14305]

hi dave
this is the output

ASUSWRT-Merlin RT-AC3100 384.18_0 Sun Jun 28 17:58:08 UTC 2020

andresmorago@RT-AC3100-0548:/tmp/home/root# tc -s filter show dev eth0 | grep 0x4014 -B1
filter parent 1: protocol all pref 23 u32 fh 829::800 order 2048 key ht 829 bkt 0 flowid 1:12  (rule hit 891924 success 54256)
  mark 0x40140000 0xc03f0000 (success 54256)

After upgrading and automatic resorting the appdb rules, my DoT appdb rule shifting 1400C5 to NetControl seems to get overwritten and listed as WebSurfing (14**** behind it overrules 1400C5 at the moment) in the chart. Before when my rule was the last it was listed as NetControl. As far as I understand from your previous post there still is a discrepancy between display and real classification so no idea if it's NetControl or WebSurfing in reality. I already read your answer to the similar speedtest problem some posts above but wanted to confirm that it's not a single user or class effect. Thank you for keeping it updated btw, great job so far

Edit: same output as andresmorago:

admin@router:/tmp/home/root# tc -s filter show dev eth0 | grep 0x4014 -B1
filter parent 1: protocol all pref 23 u32 fh 829::800 order 2048 key ht 829 bkt 0 flowid 1:13  (rule hit 1746 success 545)
  mark 0x40140000 0xc03f0000 (success 545)

But on the integrated classification page DoT is shown as NetControl. So, who's right?

LOL, I see my problem. I add the specific 14 rules at prio 22 (one priority up from the default 14**** prio 23), but then delete prio 22 when I replace the rule for 13****, so they get lost. I can fix that today.

 

[chris.at]

The Merlin page will show the color based on the original Mark, so it wouldn’t be aware that we changed it after-the-fact. But the fact that you also show just one rule with the tc grep makes me wonder what’s going on. If you run flexqos check does it help?

So DoT should be NetControl without the need to install a special rule? And you still need the flexqos check or do I interpret your follow up correctly?

 

[dave14305]

So DoT should be NetControl without the need to install a special rule? And you still need the flexqos check or do I interpret your follow up correctly?

You still need a special DoT rule if you also still have the default 14**** rule, since it will send all "14" marks to Web Surfing. Once I fix my bug with applying wildcard appdb rules, it should work with both the 1400C5 rule and 14**** rule in place. But not until 0.8.5 is released.

 

[dave14305]

Version 0.8.5

• Fix webui display of appdb rules overriding iptables rules
• Fix the appdb wildcard rules so we no longer delete and recreate the existing rule, which could remove other rules at the name prio level. Instead we now tc change the existing rule if it already exists, or else add a new rule if one didn't already exist.
• Fix other tc calls to use the tc variable that should normally point to realtc.
• Fix detection of the default undefined rule for 000000 traffic so that it appears above the existing Work-From-Home rule (also 000000 mark).

Long story short, this should fix the bug with appdb rules, but I changed the method of doing it, so it might need more exhaustive testing to be sure it didn't break anything else.

sh /jffs/addons/flexqos/flexqos.sh update

EDIT: if you updated within 36 minutes of this post, please update again. A hotfix was pushed.

 

[chris.at]

Looks good at first glance, thank you for your efforts!

 

[mbze430]

I dont know if this is something with FlexQoS or Merlin or ASUS. When transferring through VPNC. While incoming traffic via the VPNC the "Bandwidth Monitor" will show it is upload.

 

[andresmorago]

Version 0.8.5

• Fix webui display of appdb rules overriding iptables rules
• Fix the appdb wildcard rules so we no longer delete and recreate the existing rule, which could remove other rules at the name prio level. Instead we now tc change the existing rule if it already exists, or else add a new rule if one didn't already exist.
• Fix other tc calls to use the tc variable that should normally point to realtc.
• Fix detection of the default undefined rule for 000000 traffic so that it appears above the existing Work-From-Home rule (also 000000 mark).

Long story short, this should fix the bug with appdb rules, but I changed the method of doing it, so it might need more exhaustive testing to be sure it didn't break anything else.

sh /jffs/addons/flexqos/flexqos.sh update

EDIT: if you updated within 36 minutes of this post, please update again. A hotfix was pushed.

Thanks Dave. I now see the custom Speedtest Marks under file transferring.
Will test more on the way

 

[dave14305]

I dont know if this is something with FlexQoS or Merlin or ASUS. When transferring through VPNC. While incoming traffic via the VPNC the "Bandwidth Monitor" will show it is upload.

I don't know much about running a VPN client on the router, but there are still a couple hard-coded rules from FreshJR that are intended to solve the VPN upload/download problem. I don't know if they work. Run this and see if the OUTPUTs rule and the first POSTROUTING rule have any hits.

iptables -t mangle -nvL

 

[solstyce]

I was just trying to post my stats asp file in a code box.. maybe that was it. Got a HUGE popup msg saying "SORRY YOUVE BEEN BLOCKED" with big X's and sht lol

@Sinner @Vexira --- When I was troubleshooting my DHCP on FreshJRQoS with dave14305 I was having the same trouble posting my logs. Someone (forget who, sorry) pointed out that Cloudflare was blocking me due to my log file mentioning the hosts file. Inserting spaces, e.g. "/etc / hosts" got it past the filter.

I'd check what you're posting for mentions of that file, or other potentially sensitive files and experiment with removing / modifying them to get past the block.