New hEX

[kvic]

The new hEX (RB750r3) runs RouterOS 6.x which is the latest major RouterOS release. Some known limitations..the new hEX/RouterOS v6.x doesn't support:

• automatic QoS such as fq_codel

• IKEv2 IPsec VPN; IKEv2 is more efficient than L2TP

• per-port VLAN configuration; HW is capable but currently not implemented in FW

• VTI for IPsec tunnels

Some of the features are promised in RouterOS 7. It is the next major version and has been in the making for many years. However, the new hEX is crippled to include only 16MB flash. At the moment no words from Mikrotik if v7 will fit in 16MB or less.

Btw, EdgeRouter X supports all above features (but as I mentioned before and elsewhere, UBNT cripples ER-X in a different way).

 

[paraplu]

The performance is actually very different, When you start doing heavy QoS for instance or VPNs. However for most the hEX gives you the performance of the ERL for less as far as hardware acceleration is concerned and perhaps even QoS.

The hEX also has a switch chip so throughput is limited. The CCR has CPU connected ports and the CCR is actually slightly (probably by 1ms) faster. When it comes to multi gigabit the hEX wont cut it. Still it would be nice to have multi gigabit internet.

To get mikrotik routerOS at a low price and a capable platform is the advantage of the hEX

Running this Hex at home for some time and I have not seen any differences compared to the CCR1009. Only using 2 ports: WAN and LAN which goes to a solid switch. I do not use Qos as this is not needed for 500/500 fibre, and VPN seems to be running fine as well with >100mbps (IPsec). Thinking of selling my CCR1009, anyone?

Addition: I noticed that this device handles multi-core very well, compared to the CCR. The CCR would occupy a single CPU 100% with speedtest.net tests with pppoe, while this device seems to be able to split the load evenly between all 4 threads/cores.

 

[kvic]

Addition: I noticed that this device handles multi-core very well, compared to the CCR. The CCR would occupy a single CPU 100% with speedtest.net tests with pppoe, while this device seems to be able to split the load evenly between all 4 threads/cores.

Interesting note. Parallel programming has been a challenge for a long time. People as good as Mikrotik seem can't put the many cores in CCR's to efficient use. On the hEX, how evenly among the four 'cores' do you see under different usage scenario like speedtest and ipsec vpn? In ER-X as I posted in another forum, UBNT can't make use of all four 'cores' for in IPsec for one direction (..the other direction is okay).

Another minus for CCR is that its FW re-orders IPsec packets if HW accelerator is enabled. The packet re-ordering upsets majority of PC users.

 

[System Error Message]

Running this Hex at home for some time and I have not seen any differences compared to the CCR1009. Only using 2 ports: WAN and LAN which goes to a solid switch. I do not use Qos as this is not needed for 500/500 fibre, and VPN seems to be running fine as well with >100mbps (IPsec). Thinking of selling my CCR1009, anyone?

Addition: I noticed that this device handles multi-core very well, compared to the CCR. The CCR would occupy a single CPU 100% with speedtest.net tests with pppoe, while this device seems to be able to split the load evenly between all 4 threads/cores.

RouterOS has a lot of old code that isnt parallelised and their dev team isnt that big. I do hope they will do better in ROS 7 as the CCR has a lot more potential. However routerOS is still better than edgeOS. The other part to the CCR are the libraries and compilers from Tilera as that does play a significant role but it also depends on how much mikrotik makes use of it for the CCR.

However when you need processing capability you usually would use a CCR for small ISPs and businesses as they will have a lot of filters, rules and QoS. One of the main uses for a CCR is to have many PPP clients while performing some QoS on them like what you will find for an ISP as that was one of mikrotik's focus for it when they were developing the CCR as thats one of the uses that actually uses all the cores. Many elements of routerOS still arent threaded such as their proxy (which affects hotspot performance).

 

[kvic]

However when you need processing capability you usually would use a CCR for small ISPs and businesses as they will have a lot of filters, rules and QoS.

I have a question dangling in my mind for a long while. What is "filter rule" in Mikrotik speak, especially in the context for their official benchmark. For example, they quote "25 filter rules" with a specific throughput. What is a "filter rule"? What do you they mean by "25 filter rules"?

I would naively think one rule is one check in iptables. "25 filter rules" means a packet processed after 25 check's/comparison's. But if that's the case, the performance hit with "25 filter rules" in their numbers is more than I would expect from iptables.

Have you looked into this before?

 

[System Error Message]

I have a question dangling in my mind for a long while. What is "filter rule" in Mikrotik speak, especially in the context for their official benchmark. For example, they quote "25 filter rules" with a specific throughput. What is a "filter rule"? What do you they mean by "25 filter rules"?

I would naively think one rule is one check in iptables. "25 filter rules" means a packet processed after 25 check's/comparison's. But if that's the case, the performance hit with "25 filter rules" in their numbers is more than I would expect from iptables.

Have you looked into this before?

under firewall, the filter page. For QoS its the simple queues in queues. Basically they have traffic pass through 25 rules (25 rules that are related to each other). its much much more than you would normally expect at home or in small businesses to have 25 rules that co relate to one another as traffic would have to pass through each rule. Normally when you configure the routers you would avoid that and try to make the rules a little parallel (like using packet marks or type of traffic or even binding it to an interface or direction).

 

[kvic]

Basically they have traffic pass through 25 rules (25 rules that are related to each other). its much much more than you would normally expect at home or in small businesses to have 25 rules that co relate to one another as traffic would have to pass through each rule.

For SOHO users, I would assert packets no need to pass more than two "filter rules". On average, all packets pass through 1.x filter rule where x is approaching 0. I won't expect such quality from consumer routers.

So for Mikrotik benchmark, SOHO users shall be looking at "None (fast path)" numbers?

 

[System Error Message]

For SOHO users, I would assert packets no need to pass more than two "filter rules". On average, all packets pass through 1.x filter rule where x is approaching 0. I won't expect such quality from consumer routers.

So for Mikrotik benchmark, SOHO users shall be looking at "None (fast path)" numbers?

Correct, fast path is used for bridging/routing whereas NAT can use hardware NAT.

the 25 filter rules (no fast path) for routing is usually what i would use if you dont plan on using QoS. Mikrotik configuration can get complex easily.

 

[kvic]

...fast path is used for bridging/routing whereas NAT can use hardware NAT.

Where did you get this?

One of the heavy customisation that MikroTik did to Linux kernel is re-write almost the whole flow of packet processing. The "fast path" allows packets to skip the whole network stack very early on. I got it from one of their presentations.

 

[System Error Message]

Where did you get this?

One of the heavy customisation that MikroTik did to Linux kernel is re-write almost the whole flow of packet processing. The "fast path" allows packets to skip the whole network stack very early on. I got it from one of their presentations.

because in order to use NAT you must enable connection tracking. When you enable it you lose fast path. the hardware acceleration for NAT uses the chip and can have connection tracking. Fast path not only bypasses the whole flow but it also uses hardware as well. Its why loads of low end routers will do wirespeed layer 3 routing in the past.

In a SOHO environment NAT is a must but in the case your own internal network does layer 3 segmentation or even bridging thats where you can use fastpath.

If you read mikrotik's documentation/wiki the technical bits are explained there and the conditions required for fastpath or mikrotik's version of hardware NAT.

 

[kvic]

Okay, I think I misread your response #28. It makes sense now. FastPath or NAT..somewhat mutually exclusive!

People might ask why not a mixture of both? Seems there is: *drum* FastTrack. Seems major part of FastPath + NAT (i.e. connection tracking) = FastTrack. It's available in some models including RB750r3.

Now the question is how does FastTrack performance compared to FastPath? I bet it's closer to traditional NAT than FastPath..

 

[kvic]

In other words, "None (fastpath)" vs "25 filter rules" are comparing FastPath (no NAT'ed traffic) and NAT'ed traffic.

It's less a comparison of NAT'ed traffic with 1 filter rule vs 25 filter rules.

That's closer to my original thought and understanding about iptables. Traversing 25 filter rules shall not cost a significant impact when compared to, say, only 2 filter rules.

 

[System Error Message]

In other words, "None (fastpath)" vs "25 filter rules" are comparing FastPath (no NAT'ed traffic) and NAT'ed traffic.

It's less a comparison of NAT'ed traffic with 1 filter rule vs 25 filter rules.

That's closer to my original thought and understanding about iptables. Traversing 25 filter rules shall not cost a significant impact when compared to, say, only 2 filter rules.

Its because not only will you have NAT but you will also have a few filters running as well for security. Fasttrack is hardware NAT for mikrotik but you can configure it which is something you cant do on edgeOS.

 

[kvic]

Fasttrack is hardware NAT for mikrotik but you can configure it which is something you cant do on edgeOS.

I'm sure FastTrack is not simply HWNAT. Almost certainly incorrect. My understanding is that FastTrack is FastPath+NAT. It's a heavy customization in Linux kernel that alters packet processing flow for speed. Of cos with such a framework in place, it doesn't prevent MikroTik from using NAT hardware available on certain platform. In other words, it's a much more sophisticated implementation that Broadcom's CTF wanted to achieve. But CTF fails in both functionality and stability.

Not sure what did you mean EdgeOS can't do in this context..

 

[System Error Message]

I'm sure FastTrack is not simply HWNAT. Almost certainly incorrect. My understanding is that FastTrack is FastPath+NAT. It's a heavy customization in Linux kernel that alters packet processing flow for speed. Of cos with such a framework in place, it doesn't prevent MikroTik from using NAT hardware available on certain platform. In other words, it's a much more sophisticated implementation that Broadcom's CTF wanted to achieve. But CTF fails in both functionality and stability.

Not sure what did you mean EdgeOS can't do in this context..

You can use fastTrack as a way of prioritising certain packets so you can still use QoS on other packets. on edgeOS if you enable QoS you cant use hardware acceleration.

fastTrack dose require the hardware for it as not all routerboards actually support it. For example on an old routerboard like the RB450G you can put the rules in, it will say fasttrack enabled but in stats you will not see any packets going through. On my CCR which is a much newer routerboard the packets do actually go through fasttrack.

 

[kvic]

You can use fastTrack as a way of prioritising certain packets so you can still use QoS on other packets.

I think people don't have to "use" fastTrack. it's automatically made use of on supported platforms. Based on RB750r3's benchmark numbers, I concur FastTrack and QoS can live nicely together. Also QoS HW on MT7621A is utilised. Otherwise, "25 simple queues" with 1500-byte packet can't achieve near 2Gbps max throughput.

on edgeOS if you enable QoS you cant use hardware acceleration.

Seems true. For example, QoS HW is not even implemented in ER-X's firmware.

fastTrack doe require the hardware for it as not all routerboards actually support it.

What is the hardware?

 

[System Error Message]

I think people don't have to "use" fastTrack. it's automatically made use of on supported platforms. Based on RB750r3's benchmark numbers, I concur FastTrack and QoS can live nicely together. Also QoS HW on MT7621A is utilised. Otherwise, "25 simple queues" with 1500-byte packet can't achieve near 2Gbps max throughput.



Seems true. For example, QoS HW is not even implemented in ER-X's firmware.



What is the hardware?

Its automatically enabled but the EdgeOS doesnt allow you to selectively use it. For example you may want to use fasttrack for important traffic like gaming, web, VOIP, streaming and use QoS on all other traffic. This means that even on a gigabit connection when someone starts downloading a torrent file with many seeds, it will not cause bufferbloat or take up all the bandwidth.

As i edited in my previous post some older routerboards if you try to use fasttrack on them will say enabled but you will not see any packets going through. If you actually read mikrotik's documentation you will see that it does use the hardware.

 

[kvic]

For example on an old routerboard like the RB450G you can put the rules in, it will say fasttrack enabled but in stats you will not see any packets going through. On my CCR which is a much newer routerboard the packets do actually go through fasttrack.

In ROS, how do people check and tell if a packet is going through FastTrack, QoS or/and FastPath?

 

[System Error Message]

In ROS, how do people check and tell if a packet is going through FastTrack, QoS or/and FastPath?

the default rules already include it but if you go to IP settings it tells you how many packets, or you can add a passthrough or see the rule stats.

 

[DanH]

I just ordered one of these yesterday, arrives tomorrow. Pretty excited to mess around with it and learn routerOS. I am really interested to see how it compares to the ERL I have now. We are moving into a home next year and plan on moving up to wired APs (from the Orbi) and either a ERPro or CCR9, so this time with these less powerful routers should hopefully make that decision easier. Is the routerOS wiki the best place to start learning?