Need help with mangle rules on hEX

[DanH]

OK I have a couple of mangle rules set up on a hEX. This has solved my bufferbloat issue. The issue is:

When I use certain speedtests like dslreports or Net Analyzer I get full speed (350Mbps).
When I use other speed tests or downloading games off of steam: They both cap out at 200Mbps.

When I turn off the mangle rules,:I get full speed on the other speed tests and steam. And the DSl reports speeds and quality tank.

I only have two mangle rules, they just prioritize traffic to my PC over everything else. I could understand my issue if it was just maxing out the router CPU, but it is not. I get 45% usage when it maxes out on DSL reports speedtest and get 25-30% cpu usage when it pegs at 200 Mbps. Also it doesnt make sense that it can max out one and not the other, but turning off mangles essentially reverses the issue.


Any idea what is going on?

 

[kvic]

I would check how frequent the mangle rules are visited in each of the two major test cases...

 

[DanH]

I gave up on it, and decided if I have a large known download to do like a game or something I will turn ON Fasttrack (when I do that, my bufferbloat goes to a C, but my speed on all tests goes to 350Mbps). When I don't have a large download, I will turn OFF fasttrack, my bufferbloat goes to an A+ and my speeds for certain things (which is weird) will cap at that 200Mbs mark. I can live with that. Also I am going to order a ER-X and see how fq-codel does. My wife is getting a tad irritated I keep buying router, but I am enjoying it.


Anyway here is what it looks like:
With fastrack OFF (mangle rules ON)

With fastrack ON (mangle rules OFF)

 

[kvic]

If you observation was correct, you seemed to pin point the cause to the mangle rules for QoS. I was thinking your rules might be poorly written in a way that they get hit on every single packet meeting the criteria. There is a feature in iptables that only first packet in a stream gets marked and subsequent packets will be automatically marked. Much more efficient. Do you think you're using that?

Anyway, getting a ER-X is also fun. You can re-purpose it as a smart switch later (can do that to your RB750r3 with a new firmware too)

 

[DanH]

I basically just mark a connection as PC_Connection with passthrough. Then I mark the packets for uploads to that connection, and downloads to that connection. Then I create 3 mangle rules: 1 each for up and down to the pc, and one for everything else. The queue type I am using is PFIFO, which I think is packet first in first out.

I will be honest, if you do some googling and looking through forums, and presentations everything is very vague. It seems like 99.9% of any info related on the subject is coming from people who don't really know all that much about it. You can usually tell when someone understands something by how they break it down, and you can tell when you watch a presentation when someone is hurriedly glossing over something because their knowledge of it is paper thin.

 

[kvic]

Check if Winbox allow you to specify 'MARK' or 'CONNMARK'. The latter shall be more efficient. Hmm..I probably spent more time reading ubnt forum than mikrotik's. People on ubnt's tend to speak longer... I believe that's because there are a larger proportion of ppl from the state. Maybe a language thing. I went through a few MUM video's. Those from the state are more presentable. Those from EU contain a bit more technical details..

 

[DanH]

Yeah I think it is a mixture of language barrier, and the non professsional people don't know a whole lot. Whereas the professional people are either not necessarily the type of people who help for free or so beyond speaking at a level that a novice can understand, that there help isn't really helpful. UBNT forums are much more novice friendly. A typical Mikrotik forum response is just some vague response with a link to the wiki.

 

[DanH]

My understanding is queues only work on packet marks not connection marks. But connection marks are used for basically all the other stuff. That is why I mark the connection for pass through then mark the upload and download packets for the PC and assign them to that connection mark for pass through. That way I can queue the PC stuff and gain the efficiency from connection marking. Hope that makes sense.

 

[kvic]

Your description makes sense after I went back to the wiki page: http://forum.mikrotik.com/viewtopic.php?t=73214#p371300

Looking closer into the "ros code" in Option #1, actually a few things I don't quite get:

1) once you mark a new packet "VOIP" with "mark-connection", why still need a new china to mark every packet where "connection-mark" = "VOIP"? I'll its necessity. Shall there be an automatic way or tc filter (underlying linux command for queues) simply makes use of "connection-mark"?

2) "Mark everything else" seems unnecessary too. I'll challenge if there is way to specify default behavior for all unmarked packets.

Assume "VOIP" is any packet to/from your PC..

 

[DanH]

Basically fast path doesn't work with queues at all period end of story (despite what some say). If you have queues they dont work if fastpath is on. Its either/or not both.

So I set up 3 simple queues, no mangle rules necessary at all:

• One is the routers address (192.186.88.1) at unlimited bandwidth priority 1, PFIFO,
• Two is the PCs address (192.186.88.214) at 250 Mbps Down/15 Mbps Up priority 2 PFIFO,
• Three is everything else on the LAN (192.186.88.0/24) at 175 Mbps Down/15 Up priority 8 PFIFO. That seems to work really well.

I learned you can do simple queues without mangle rules, and they apply in sequence hence the three rules in the order they are in.

Still have issues with certain things not running like they should, for some reason Steam Downloads and OOKLA speedtest do something weird where they will top out at 180 Mbps with fastpath off (whereas its 350 Mbps with it on), but other speedtests dont (DSL Reports is a perfect 350Mbps with fast path on or off). Not sure why that is, but I think it must have to do with the packets themselves, and that ~180ish Mbps is the max the router can do routing them without fastpath.

Still plan on getting a ER-X SFP but don't expect it to do any better, but who knows. I also think that the very most this router will do is roughly ~350 Mbps with queues on. I also learned that PFIFO works best for me over pcq or any other type.

Also fastpath is terrible for bufferbloat. Not sure what goes on, but the mere act of turning on queues even if I don't limit bandwidth at all will turn a C or D bufferbloat (like ~400-500 ms) to an A or A+ (20-50ms). Theoretically they aren't even doing anything, but yet they are...

All in all it doesn't work in ways you would expect that is for sure. But like I said it works well right now for decent speeds all around and if I know I have a huge steam download I can just turn on fastpath.

 

[kvic]

Nice progress. MT7621A CPU alone (i.e no HW offload) can do around 500Mbps routing. With QoS on, I found ER-X can max out around ~500Mbps too.

Current firmware in ER-X and SFP max out 500/500 full duplex. A few of us come to believe the firmware can be changed to support 1000/1000 full duplex. For all users who care shall show UBNT encouragement to fix the issue on this thread.

 

[DanH]

Nice progress. MT7621A CPU alone (i.e no HW offload) can do around 500Mbps routing. With QoS on, I found ER-X can max out around ~500Mbps too.

Current firmware in ER-X and SFP max out 500/500 full duplex. A few of us come to believe the firmware can be changed to support 1000/1000 full duplex. For all users who care shall show UBNT encouragement to fix the issue on this thread.

Yeah I am going to order one and give it a try, be interesting to see what the real world speeds are like.

 

[DanH]

Well, I didn't get the ER-X SFP, since I found someone selling their practically new USG-PRO-4 for $200. I know it doesn't run EdgeOS, and it is currently lacking features, but I think it will end up being a good buy. I also think it will end up being a little more future proof then the ER-X. Beside I still have the hEX and a ERL.

 

[kvic]

I'm pretty sure all USG's run EdgeOS under the hood. The GUI is replaced to have Unifi look. There may be additional software to support Unifi-only features. They look sexy...Personally I had little confidence in the whole USG series due to one product's poor performance in Ars Technica's test.

 

[DanH]

Yeah the usg didn't do well, and the reviews have been underwhelming for the pro as well. All that considered I still think it will perform better than a ER-X and like I said I already have the hEX which has the same hardware. So the ER-X would have been a side grade. I also think that over the year, the USG line is gonna get better as they evolve it. Plus when we move into a house this year I plan on replacing the Orbi and Netgear switch with Unifi APs and a Unifi switch. I think the USG-Pro would be a better fit for that, and I think I will dig the analytics. Plus it was $200!

Back on topic though, it looks like (at least for me) with just basic firewall rules and a 3 simple queues, the hEX tops out somewhere between 180-350 Mbps. It seems like the types of traffic definitely effects whether that speed is going to be at the top or lower end of that band. I am hoping the USG-Pro can get a solid 350mbps.