Question about dedicated routers/pfsense boxes and their uses cases

[theramenman]

Due to work, I have to move once again and I cant really take my old setup with me for various reasons. Since I'm starting anew and Im still kind of a noob/pretty rusty on what little I already knew, I had a few questions.
Given:
At my new place Ill finally be living in an area with fibre so I can get 600 down and 350 up for a reasonable price. I have a rather tech savvy family so pretty much everyone is constantly on the internet and I have no doubt I'll be maxing out the connection a decent amount of time every day. I will be connecting my NAS, my desktop and 2 AP's to the router so I need at least 4 gigabit LAN ports.
Questions:
1. I've seen a lot of hubub on reddit and other forums I frequent about pfsense boxes and dedicated routers from Mikrotik, Ubiquiti, etc. What use cases do they have?
2. Are they significantly faster in heavy wired use cases than your average high end consumer router?
3. Would something like a Mikrotik Hex be good enough for my use case or will it get maxed out?
4. Would something like a Mikrotik RB2011iL-IN be more suitable, especially since it'd allow me forgo using a switch should I add more devices (most likely relatively low priority stuff like a Roku or a console) later on?
5. Would a Pfsense box make more sense? Mini computers like QOTOM-Q190G4 running Pfsense seem to be popular in the DIY segment, would something like that work better?

If I said anything incorrect or have any blatant misconceptions, please correct me and let me know. I've tried to read up on the topic, especially other posts on SNB forums, but most of the other posters already seemed to know what they were doing. Thanks in advance everyone.

 

[L&LD]

You should wait at least 24Hrs before bumping your own thread (common 'net etiquette).

Anyone who has seen it by now would have answered. Anyone who can answer hasn't checked the site yet.

Patience.

 

[theramenman]

You should wait at least 24Hrs before bumping your own thread (common 'net etiquette).

Anyone who has seen it by now would have answered. Anyone who can answer hasn't checked the site yet.

Patience.

Welp, sorry about that. I did post it late last night so thought no one saw it. Forgot that this forum had a number of views feature. Sorry everyone.

 

[System Error Message]

a lot of people arent from the US you know, the US isnt the center of internet.

A dedicated router is a device dedicated to routing, what you're asking is about non consumer routers.

These routers differ in the they are very flexible with the exclusion of ubiquiti edgerouters. They can be used to create all sorts of networks. Speed isnt what the determining factor rather it is the featureset that makes it the determining factor. For example both with mikrotik and any linux/unix box you can create all sorts of networks, you can assign multiple networks to the same interface and you can create routes and route in various ways, things you will not find on a consumer router nor ubiquiti edgerouters.

What determines if you need such a router is more of what you'll be doing (not what you'll be connecting behind it). If you need a complicated QoS, some good firewalls and so on you will need it. If you have a complicated network you want to put together you will need it otherwise they dont really offer benefit over consumer routers.

Speed wise they tend to do better than consumer routers because of their better firmware but this isnt really the deciding factor. For example if you need QoS and firewall ubiquit edgerouter (which so many swear by all around) is a big no for your WAN speeds. You need 950Mb/s of NAT forwarding and the ubiquiti edgerouter will only do at max 200Mb/s without hardware acceleration. dual core ARM based broadcom A9 which many consumer routers use will do up to 500Mb/s of NAT forwarding without hardware acceleration. The speeds you see for recent mikrotik MIPS boxes include hardware acceleration so i doubt even they can handle your speeds. I suggest you go with either a PPC based mikrotik or pfsense or some linux/unix box.

If you dont need the features that these routers offer than choose away based on hardware NAT. Ubiquiti edgerouters are considered a consumer router because their actual aim and design is for consumers, but they are sold with information not meant for consumers i.e. if you were planning on using their devices for core routing. The speeds listed by ubiquiti is if you use their routers as a layer 3 switch. The ERL is at best only capable of NAT at 1.3Gb/s with hardware acceleration. So if hardware acceleration can be used any router will do as long as the firmware is good. I even argued on ubiquiti forums about their restrictions for configuration and they said its so "you dont shoot yourself in the foot", non consumer routers do not design with this in mind. When looking at the speeds listed by manufacturer, bridging is essentially layer 2 switching and routing is layer 3 switching. NAT involves processing and in many cases is equivalent of 5 firewall rules so with mikrotik i take the lowest speed listed as the speed i will be seeing for 1500 byte packets. If you use PPPOE and/or VLANs with your ISP you will need a router with even better CPU but mikrotik's RB1100AHx2 is good for 1Gb/s of NAT with PPPOE and vlan without using hardware acceleration.

So unless you can use hardware acceleration you should strike ubiquiti off your list. Ubiquiti edgerouters are essentially a consumer router, its their marketing and information that makes people think they arent. Their marketing is pretty interesting and they essentially are selling to consumers with information meant for non consumer use, places like buildings where you would be having different subnets for various places and to filter internal traffic that doesnt involve using NAT, only routing and this is where ubiquiti's routing performance comes into play. For internet use, the performance for NAT is far below routing performance especially if you have processing overheads to consider.

Perhaps reddit users need to learn a bit more english and more of what the routers actually are. If you want firmware quality even ASUS is doing a good job at that especially with RMerlin's firmware so ubiquiti edgerouters have no more advantage. RMerlin's firmware actually has more features than ubiquiti edgerouter such as the ability to choose your QoS algorithm just like what mikrotik has too.

With an x86 linux/unix box like pfsense or even a linux server theres no doubt about the throughput but setting one up takes more effort and research. It does however offer the most flexibility such as in setting up a UTM and doing much much more.

Armada-380 platform has better throughput without acceleration than the broadcom ARM A9 so you may want to take a look at that.

 

[theramenman]

Very interesting, thank you for all the info. Just one quick question: why do you disregard hardware acceleration? Is it only useful in very specific/limited use cases? And as far as I'm aware, my ISP doesn't use PPPOE or vlan. As for ASUS, yeah, I previously had an AC68U and it was a solid router, probably the most stable consumer router I had ever used, especially with merlin installed.

 

[sfx2000]

Very interesting, thank you for all the info. Just one quick question: why do you disregard hardware acceleration? Is it only useful in very specific/limited use cases?

Because it's not needed in a right/proper design... not everyone is in a Broadcom centric world - and there's a few platforms that can outperform Broadcom solutions with no special sauce... Broadcom has a lot of special sauce that works well in small memory footprints and low cost barriers - which is great for OEM's trying to maximize profits - and that's business enablement...

Once out of the consumer space - Marvell, and Intel come to mind, and then there are the exotics like Tilera and Cavium based devices, along with Freescale's PowerPC platforms. And you'll spend a bit more for some of them, as they might just be routing - not switching, no wifi... you'll have to add them.

Broadcom is good with "Router in a Box" that many all-in-one Consumer AP/Routers tend to use - and it's cheap, and it works well enough until it doesn't - which in many cases these days is Gigabit WAN connections....

 

[theramenman]

Wow, that clarifies a lot. Thanks sfx. Getting back to the throughput problem, is there a good way in general of finding out if a router is good enough for my needs? For example I was looking at the RB2011iL-IN since it seems to offer a decent number of ports, and RouterOS seems to have some really good features and the PoE out is helpful. If I look at the bottom of the product page for it, it has a bunch of different speeds depending on routing or bridge mode, configuration and what seems to be file or packet sizes?

 

[System Error Message]

Depends if you need QoS. If you do you wont be able to reach your needed throughput. Mikrotik NAT acceleration works but it does so by accelerating the connection so all the filtering has to be done before the connection is accelerated. They call it fasttrack so i suggest you read about it on their wiki if you need hardware acceleration and check if that routerboard supports it.If you learn how to use mikrotik fasttrack it helps to understand hardware acceleration in consumer routers too, how they work. Essentially once a connection is accelerated it will bypass all filters and rules but i think you can use mangle to filter accelerated connections though i would expect some drop of throughput if you do as last time i tried using mangle on fasttrack it worked and fasttrack was still working. It was difficult to discern if it really worked since the CCR1036 spends 0% CPU for maxing out my connection with or without fasttrack. Still at least im 10G internet ready.

If you need stuff that cant be accelerated than consider their PPC based routerboards as PPC, x86, TILE can handle the load. If you use PPPOE you must use a PPC at least. You may want to look at the RB3011 as that also has POE out, better switch arrangement (2Gb/s to CPU per switch chip) and 1 CPU connected port which is the SFP port. You may get better throughput with that. Without hardware acceleration it does around 500Mb/s last time it was in beta but it uses qualcomm's IPQ 8064 so the performance should be too difficult to figure out. Switch arrangement may be complicated but if you connect 2 ethernet cables from switch 1 to switch 2 (bonding) you could use switch 2 to connect to CPU, still have SFP and not have to worry about bottlenecks as you could use switch 1 for WAN if needed but also for some LAN devices so that the path to WAN goes through switch 2 rather than the same link back and forth.

Switch chips have always been the bottleneck if you connect both WAN and LAN to them but mikrotik gives you an alternative using 2 switch chips for your to connect them around so both WAN and LAN dont flow through the same link. The RB1100AHx2 has this arrangement too but with 1Gb/s links instead, 1 or 2 CPU connected port and the last port or 2 connected via PCI bus. Its important to look at the block diagram for choosing a routerboard and setting it up in order to get the most out of it.

Some boards like the RB450G and RB850gx2 lets you exclude eth1 from the switch so it connects direct to CPU. The RB450g does not support fasttrack and the RB850gx2 uses same CPU as RB1100Ahx2 but with half the clocks.

You could also consider pfsense if you dont want to go the embedded route. Essentially MIPS > ARM > TILE > PPC > x86 in order of how much throughput loss they take when you add rules, QoS and other things like VPN but it doesnt necessarily mean which CPU is faster as you have multicores for all of them at different clocks and Tile's manycore. The CCR1009 may be as fast or faster than your PC for NAT as it has 9 cores vs the 2 or 4 x86 cores in most desktops.

 

[Trip]

Wow SEM, smashing the keys on this thread. I'll try to be a bit more brief.

First off, UBNT. Questionable marketing and quirky limitations aside, they have their place. For some quick and dirty fq_codel on a link of <150Mb/s and a simple topology, the $50 ER-X is at least a decent choice -- not to mention other use-case and model combinations. But yes, their stuff can be limiting; just have to know the use-case.

Getting back to the task at hand: 950Mb/s+ routing. For benchmarks, I often go by medium packet size, factor in required packet processing (VPN, PPoE, etc.), then estimate down even a bit further just to play it safe. In MikroTik's case, I'd average 512-byte values for 25 simple rules and 25 ip rules, then round down a bit. So for the RB3011, that would be ~1Gb/s. Might be a bit harsh or a bit generous, project-depending. Honestly, for the price, if your going with MT hardware, I'd just plunk down for a CCR1009-PC and be done with it. 9Ghz of CPU for the price/form factor/power draw smokes a lot of other choices, assuming rOS is agreeable to you.

Otherwise, I'd do a linux-based distro on a power-efficient Celeron, i3 or i5 with Intel NIC(s). If you want fq_codel for QoS, there's IPFire. Otherwise pFsense is always a popular choice.

Hope some of that helps!

 

[System Error Message]

mikrotik is implementing fq_codel soon in their next major release (ROS 7) but i wouldnt hold my breath. They never listen to me regarding the flaws and stuff that mikrotik lacks despite the brand still being good. For example DNScrypt was suggested years ago and still mikrotik doesnt implement as it is useful if you have ISPs hijacking your DNS requests.

he needs 950Mb/s of NAT not routing and usually the 25 ip filter rules would be an equivalent. For MIPS its at most 1/4 of the routing speed if you dont add any configs.

 

[theramenman]

The RB850Gx2 for about $100 from an authorized reseller seems like an exceedingly good deal, especially as it seems capable of handling my connection if I understood correctly what you guys said. Since for now it seems to be all that I need and it falls within my meager budget of about $200 (moving has been rough on the wallet), it looks like a good choice. What do you guys think? I can also get a mini x86 PC with 4 ports (Intel NIC, although, one less port than Id have with the RB850Gx2), a quad core intel cpu, 2 gigs of DDR3L and a small SSD for around $200 if I wanted to go down the Pfsense route.

 

[System Error Message]

The intel CPU would be faster. you could still use mikrotik on intel as well or even pfsense, though routerOS costs money on x86 for a one time license fee while pfsense is free. As long as the NICs are intel and not realtek it is a good choice.

 

[garyd9]

Very interesting, thank you for all the info. Just one quick question: why do you disregard hardware acceleration? Is it only useful in very specific/limited use cases? And as far as I'm aware, my ISP doesn't use PPPOE or vlan. As for ASUS, yeah, I previously had an AC68U and it was a solid router, probably the most stable consumer router I had ever used, especially with merlin installed.

I dropped using an Asus router (w/ Merlin) due to limitations of routing speed. Those limitations were helped by "hardware acceleration," but there's always a tradeoff with those tricks. For example, the acceleration has no impact for IPv6 traffic. It's also of limited value with QOS enabled (and no value for some QOS options.) It also bypasses the normal network stack, so many "advanced" tricks (such as some ebtables filtering) are completely useless.

How much of an impact did losing acceleration have for me? I have a 180 megabit downlink on comcast. Using comcast's own speed test against a local server, I could easily fill that pipe with acceleration enabled with ipv4 and about 40% CPU utilization on the router. With ipv6 (which can't use broadcom's acceleration), my max downlink speeds dropped slightly, and the router's CPU maxed out. (This was a AC3200 overclocked to 1.2 GHz.) If you have a 600 megabit downlink, would you be happy only getting 170 megabit because your desktop/notebook machine prefers IPv6 over IPv4?

So, to summarize - when it works, it works great. When it doesn't, the limitations REALLY show.

Oh, and in contrast, my pfsense router barely gets to 20% utilization routing ipv4 or ipv6 (either one) downlink at my full speeds, and that's with QOS (shaping) and "snort" (kind of like the "AI Protection in asus routers) running.

I hate to say it, but today's consumer routers are falling behind the needs of consumer broadband. On the other hand, most consumers wouldn't even notice that their not getting the speeds they pay for.

EDIT: Sorry, I just realized that I'm about a week out of date on this thread. Well, better late than never.