[Release] FreshJR Adaptive QOS (Improvements / Custom Rules / and Inner workings)

[FreshJR]

Do you know the "mark" value for Spotify? Nothing I have tried moves the Spotify traffic away from Web Surfing and into Downloads. What am I doing wrong? I know the remote ports the IP address range I cannot get it to work. Please help me my friend or anyone that has had this issue.

Spotifys traffic is probably not accruing under the stopify mark or it’s ip range.

You can see what traffic is actively using bandwidth via app analysis. Spotify’s traffic is probably mirrored on your isps CDN to reduce bottlenecks and that would make it hard to identify only Spotify traffic (unless you dump the entire CDN into downloads, which isn’t a terrible idea)

 

[skeal]

Spotifys traffic is probably not accruing under the stopify mark or it’s ip range.

You can see what traffic is actively using bandwidth via app analysis. Spotify’s traffic is probably mirrored on your isps CDN to reduce bottlenecks and that would make it hard to identify only Spotify traffic (unless you dump the entire CDN into downloads, which isn’t a terrible idea)

Got it fixed!! Thanks bro!

 

[medwatt]

As for your original question.

Iptables is able to track bytes transferred per individual connections & you can re-assign a traffic mark based on this information into whatever you desire.

The iptable rule would be

#HTTPS
#This rule moves a connection marked as 0x8010 - 8017 into ${Downloads} when it transfers over 1mb.
-m mark --mark 0x80100000/0xc0380000 -m connbytes --connbytes 1024000 --connbytes-dir both --connbytes-mode bytes -j MARK --set-mark ${Downloads_mark_down}

Additionally you may be interested in this rule to move HTTP traffic away from WebSurfing. (some http CDN transfers are counted as Web Surfing)

#HTTP
#This rule moves a connection marked as 0x800d into ${Downloads} when it transfers over 1mb.
-m mark --mark 0x800d0000/0xc03f0000 -m connbytes --connbytes 1024000 --connbytes-dir both --connbytes-mode bytes -j MARK --set-mark ${Downloads_mark_down}

NOTE:

With HW acceleration on, only the first packet of every connection is parsed.

This means the connection would never be re-evaluated at a later point in time when it transfers more than 1MB so the iptable rule (sending it to downloads) would NEVER hit
​

With HW acceleration off, every individual packet passing through is parsed

Since the connection is continually evaluated, the connection will change categories after it transfers more than 1MB.​

@FreshJR, I've tried this and works exactly as you say. However, it only seems to work when downloading via a web browser and doesn't work if I'm downloading via a download manager.

For instance, when downloading via a web browser (Chrome), the traffic gets identified as "HTTP Protocol over TLS SSL", mark 8013, and initially goes under "web surfing". After 1mb has been downloaded, it gets transferred to "downloads".

When downloading via a download manager (IDM), the traffic gets identified as "SSL/TLS", mark 8014, and goes under "web surfing" and remains there.

Isn't this line supposed to take care of that entire range ?

#HTTPS
#This rule moves a connection marked as 0x8010 - 8017 into ${Downloads} when it transfers over 1mb.
-m mark --mark 0x80100000/0xc0380000 -m connbytes --connbytes 1024000 --connbytes-dir both --connbytes-mode bytes -j MARK --set-mark ${Downloads_mark_down}

 

[FreshJR]

@medwatt

Yes it should of. Can you PM a link to an 8014 download. Will have to look into it.

Better yet, copy that rule and change it to

0x8014000/c03f0000

and see if it starts working. But the range shouldn’t have been the issue unless I did my math wrong.

Edit: On another note, keep in mind bandwidth is tracked per INDIVIDUAL connection.

If your download manager opens 16connections for a single download, then each of those connections will be redirected into {downloads} after individually transfering more than 1MB. Aka it would take 16mb for a “WebSurfing” download to get redirected.

Why are you using multi connection to speed up downloads?? Servers don’t typically limit individual connections to below your current ISP package anymore.

 

[medwatt]

@FreshJR,

If your download manager opens 16connections for a single download, then each of those connections will be redirected into {downloads} after individually transfering more than 1MB. Aka it would take 16mb for a “WebSurfing” download to get redirected.

That is exactly what is happening. I had actually changed the minimum size to 5mb (to allow enough bandwidth for some image heavy websites) and so I was required to wait until all connections had reached 5mb for me to notice that the traffic has been redirected to the downloads container.

Why are you using multi connection to speed up downloads?? Servers don’t typically limit individual connections to below your current ISP package anymore.

Mainly for pause and resume features. When you're on a 5mbits connection, having the option to pause and resume downloads the next day is very handy. Besides, downloading via chrome doesn't hit the top speed for me compared to the download manager.


I tried seeing if the download manager uses specific ports so that I can redirect all traffic coming from it to the downloads container. It uses random port numbers. What do you think we can do to force traffic from the download manager into the downloads container ?

 

[FreshJR]

1) Easiest, limit your download manager to 1 connection per download and have it switch over after 5MB. You don’t need multiple connections for resume functionality and multiple connections isn’t boosting your speeds at your ISP speed

2). Create a virtual network adapter, bridge it with the real one. (Virtual adapter should get assigned a new IP address, if not try static assignment from the properties menu). Use another program to bind the download manager to the 2nd IP/network adapter. QOS the second IP.

3) Run a proxy on the router. QOS the proxy port.

4) If you can’t figure out how to host your own proxy. Pay for a VPN or proxy external to your network. QOS the vpn or proxy port.

5) Find a download manager that can use fixed ports instead of the ephemeral range.

Probably more solutions if you get creative. Where there’s a will there’s a way.

Number1 is the easiest.

The proxy solution may come in handy for other users if their Dropbox/GoogleDrive/Backup software cannot be reliably have rules created to move it away from WebSurfing (If present).

I don’t use that software so I am not aware to how it is currently you identified and the degree of difficulty to move it away.

 

[Vexira]

@FreshJR I'm curious would in theory it be better if the QoS scanned every package rather then the bypass done by hardware acceleration?
I'm just wondering

 

[FreshJR]

@FreshJR I'm curious would in theory it be better if the QoS scanned every package rather then the bypass done by hardware acceleration?
I'm just wondering

No that would be slower.

With HW acceleration it scans the FIRST packet. Determines what it is. Evaluates that packet through all iptable entries. Evaluates that packet through the tc filters chain until it finds a match match. And NOW finally marks anything else comming from that unique connection into a category.
(This entire process takes time)

All subsequent packets pass right through ( but without the long evaluation chain / scanning procedure) and STILL end up in the correct category.

You don't really want to be doing that per EVERY packet unless you have too. That is why HW acceleration yields faster maximum speeds. (No it wouldn't boost DPI accuracy evaluating every packet since DPI was designed for the first packet).

You only need want to scan every packet if you want to implement rules based on bandwidth transferred or some other advanced feature. I don't recommended turning off HW acceleration unless you need the bandwidth transferred rule or your CPU is so fast that the evaluation process per packet is insignificant.

--

As always feel free to experiment. What I say isn't absolute. I'd be glad if someone finds another parameter for that can be tweaked for better results.

 

[Vexira]

No that would be slower.

With HW acceleration it scans the FIRST packet. Determines what it is. Evaluates that packet through all iptable entries. Evaluates that packet through the tc filters chain until it finds a match match. And NOW finally marks anything else comming from that unique connection into a category.
(This entire process takes time)

All subsequent packets pass right through ( but without the long evaluation chain / scanning procedure) and STILL end up in the correct category.

You don't really want to be doing that per EVERY packet unless you have too. That is why HW acceleration yields faster maximum speeds. (No it wouldn't boost DPI accuracy evaluating every packet since DPI was designed for the first packet).

You only need want to scan every packet if you want to implement rules based on bandwidth transferred or some other advanced feature. I don't recommended turning off HW acceleration unless you need the bandwidth transferred rule or your CPU is so fast that the evaluation process per packet is insignificant.

--

As always feel free to experiment. What I say isn't absolute. I'd be glad if someone finds another parameter for that can be tweaked for better results.

Thanks that's exactly what I was trying to understand well I hope with a faster CPU it will be a lot faster in general, I'm just waiting till I have enough money and I might get that ax88u if the shop I normally by it form has finally gotten it back in stock.

I noticed that during the 5 minutes wait im getting odd CPU spikes, putting it down to the router getting old.

 

[FreshJR]

No cpu usuage is normal. The QOS structure is getting setup behind the scenes which takes work.

(The structure actually gets set up, deleted, then reset up a few times for some reason)

 

[Vexira]

No cpu usuage is normal. The QOS structure is getting setup behind the scenes which takes work.

(The structure actually gets set up, deleted, then reset up a few times for some reason)

Ahh so that's why it's close to % in spikes I see.

 

[L&LD]

Thanks that's exactly what I was trying to understand well I hope with a faster CPU it will be a lot faster in general, I'm just waiting till I have enough money and I might get that ax88u if the shop I normally by it form has finally gotten it back in stock.

I noticed that during the 5 minutes wait im getting odd CPU spikes, putting it down to the router getting old.

No, it's not the router.

I've noticed it too on every router I've tried it on. Which is why I recommend waiting at least 5 minutes after any QOS changes to test if the new rules are working.

 

[Vexira]

No, it's not the router.

I've noticed it too on every router I've tried it on. Which is why I recommend waiting at least 5 minutes after any QOS changes to test if the new rules are working.

I've noticed major CPU spikes I'm starting to really think about replacing it due to the spikes I get even with big downloads, I'm just going to give it to family cause they need it to replace Thier ISP one.

But I'm wondering if the modded bursts will make a difference is preformace I tend to wait at least 10 minutes die to spanning tree protocol being on.

 

[L&LD]

I've noticed major CPU spikes I'm starting to really think about replacing it due to the spikes I get even with big downloads, I'm just going to give it to family cause they need it to replace Thier ISP one.

I just finished delivering and installing an RT-AC86U with almost all features enabled and that too spikes when hitting the downloads hard. Almost as much as my RT-AC3100 did with no features enabled at all though.

That is comparing a dual-core 32bit 1.4GHz processor ('AC3100) to the dual-core 64bit 1.8GHz processor ('AC86U). Although the 'AC86U could breathe a little easier (with my Gbps ISP), the most impressive demonstration was the wireless improvement. At about 40' and two walls, the 'AC86U could give about 150Mbps more upload and almost 120Mbps more download throughput to my AC8260 laptop in high-performance battery mode.

I should be saving for the next big changes in routers (AX final) with their 2.2/2.4? GHz processors, but this little 'AC86U really impressed me with a full M&M Config based on RMerlin firmware, amtm, Diversion, Skynet, pixelserv-tls, Stubby, FreshJR QoS, ntpMerlin and YazFi. Including AiProtection enabled (all).

 

[Vexira]

I just finished delivering and installing an RT-AC86U with almost all features enabled and that too spikes when hitting the downloads hard. Almost as much as my RT-AC3100 did with no features enabled at all though.

That is comparing a dual-core 32bit 1.4GHz processor ('AC3100) to the dual-core 64bit 1.8GHz processor ('AC86U). Although the 'AC86U could breathe a little easier (with my Gbps ISP), the most impressive demonstration was the wireless improvement. At about 40' and two walls, the 'AC86U could give about 150Mbps more upload and almost 120Mbps more download throughput to my AC8260 laptop in high-performance battery mode.

I should be saving for the next big changes in routers (AX final) with their 2.2/2.4? GHz processors, but this little 'AC86U really impressed me with a full M&M Config based on RMerlin firmware, amtm, Diversion, Skynet, pixelserv-tls, Stubby, FreshJR QoS, ntpMerlin and YazFi. Including AiProtection enabled (all).

My only fear is that because Netgear is releasing a 2.2ghz quad core that Asus will release something soon and I will be missing out on a better CPU that would smoke the Netgear in QoS preformace.

 

[L&LD]

My only fear is that because Netgear is releasing a 2.2ghz quad core that Asus will release something soon and I will be missing out on a better CPU that would smoke the Netgear in QoS preformace.

Yes, more is always better, of course, but I find that unless you're transferring very large files across the city/country(s), 100Mbps is past the 'good enough' mark for VPN performance.

The RT-AC86U should do that easily.

But just like, no, actually worse, than you, I'm still waiting for the 8 antennae, 8 streams, i3+ powered quad-core Intel 'Core' CPU with 8GB ram or more with an Asus logo and powered by RMerlin, of course.

 

[RMerlin]

My only fear is that because Netgear is releasing a 2.2ghz quad core that Asus will release something soon and I will be missing out on a better CPU that would smoke the Netgear in QoS preformace.

Can't compare them by clock rate. Broadcom vs Qualcomm, vastly different architectures.

 

[Vexira]

Can't compare them by clock rate. Broadcom vs Qualcomm, vastly different architectures.

Understandable, I'm just anticipating that a new unit will super seed the current one once ax is out for draft or to compete with the other unit form Netgear.

 

[RMerlin]

Understandable, I'm just anticipating that a new unit will super seed the current one once ax is out for draft or to compete with the other unit form Netgear.

Broadcom doesn't have anything faster than the BCM4908 at the moment, so on Broadcom's end of things, don't expect anything faster for a few years.

 

[Vexira]

Broadcom doesn't have anything faster than the BCM4908 at the moment, so on Broadcom's end of things, don't expect anything faster for a few years.

That's promising news now all I have to do is wait for the stock to come back at the PC shop I use since it's the most reasonable in price and I'll get one next paycheck.