ER-X: small is beautiful

[System Error Message]

Maybe true 10 or 15 years ago. Quite a different story on today's tier one companies. There was a wake up call for myself a few years ago..

On the SoC, believe it or not, I have the impression MT7621AT is more advanced than BCM4709. Not HW crypto which BCM is lacking. Not HW QoS which BCM also lacks. Not HW NAT which BCM again lacks (it has so called CTF which is a software patch). But the design of the "switch" block. It's actually better not called a switch. But I don't have a better name for it or the paradigm of such a "switch" design.



It's a little device. Power consumption and heat dissipation are both negligible. Borrowing WikiDev's photo (SoC under heatsink):

I think it would be very interesting once you have test bench ready and can benchmark this little guy in future. *grin*

My test bench is close to be ready to be able to have the minimum to test high end stuff. you may want to take a look at my test bench thread and see the progress and comment on the test bench itself and the various things listed down as well.

@sfx2000 perhaps we should make a stickied thread where we discuss all about hardware architecture for routers.

 

[sfx2000]

@sfx2000 perhaps we should make a stickied thread where we discuss all about hardware architecture for routers.

I thought about this, but decided against it - because there are a lot of great platforms, both HW and SW, and that thread could easily turn into a fanboi/flame thread - many paths exist between the present and the ideal, and the challenge faced here is that all paths are indeed good - whether it is QCA, Broadcom, Marvell, Intel, MediaTek and then the underlying OS's and architecture choices... I appreciate the diversity of the platforms offered, and share in the knowledge gained here by the hive mind - and contribute where I can..

And with many paths - folks will align towards certain ones - knowing that it is the "one true path"... this is not a mess I would like to be associated with...

There's a really interesting read by Scott Adams (Dilbert, but this is a book he wrote) called "God's Debris" - it's a philosophy piece... it's a worthwhile read if you can find it...

 

[kvic]

It's a nice little design - the MediaTek/Ralink chip isn't exceptional, but it is capable - not better or worse than the BCM equivalent - it's an older design, and the newer ARMv7 chips do have more horsepower and better internal bandwidth (then you have Marvell, with both the Armada XP and Armada 38* series, used in conjunction with the LinkStreet SOHO - those are beasts...)

I didn't look at companion wifi chips and how good/bad they're integrated with the SoC. On SoC alone plus its SDK features, MT7621AT beats BCM4708/9. Period

The real advantage of the ER-X, and I'm sure one would agreed is the software - EdgeOS is a fork of VyOS, and it is a real network based OS, and extremely scalable, flexible, and functionality that is hard to beat...

Both EdgeOS and VyOS forked from Vyatta which once was open source (?). First Edgerouter was released in 2012. EdgeOS development shall start more than a year ago I would think. First VyOS release was December 2013. So it can't be EdgeOS forked from VyOS.

Personally I credit Vyatta for birth of EdgeOS. Vyatta is nice. Network Admin shall like it. Vyatta though focuses on router's configuration management. It doesn't change core and fundamental functionalities of a router. That's still Linux albeit with Linux 3.10 which is eon's better than Linux 2.6.32/36..

My days are still short with EdgeOS but I grow to like Vyatta. Save lots of effort and hassle in re-configuring a router..

(in my designs back in the past life - we looked at several solutions, Ralink was one of them, we ended up going with Marvell as they were already inside our Vendor ecosystem, whereas Ralink/Realtek and the link were not - Broadcom, while in our ecosystem, did not have a part that met our needs at the time, whereas Marvell did, so we went that route)

Why not share some of the nice features of Armada? Would be interesting to hear it.

 

[sfx2000]

Why not share some of the nice features of Armada? Would be interesting to hear it.

Because it's just HW, and I've elaborated on that platform in other threads...

The nice things I like about Armada is that it has a rich set of interfaces, and configurable at that on boot - it uses uBoot which is fairly easy to build out, and the surrounding platform is fairly high performance across the board - FOSS issues being out of scope - the closed source environment, toolchains, etc - they're easier than most to work with... their switches are good, the ethernet PHY's are good, and they support more than just Linux through the bootloader...

Once one signs the NDA/MOU - it's a great platform to work with - Linksys has had more than a fair amount of pain after the promise of being "open"...

Marvell is a good base to build something on - it's just a costly platform compared to some other chipsets (however, cheaper that Intel in this sector/segment) - esp in the consumer market - the paradox there is reduce BOM and increase Margin - BCM is very well suited there...

Don't pass judgement on the BCM4708/9 - it's a solid chipset held back by an old SDK - and that is another problem...

 

[System Error Message]

bring on the fanboys .
Its more to do with what the hardware will be used for. different hardware platforms are good for different things so it depends on what features the router will have.

 

[kvic]

Don't pass judgement on the BCM4708/9 - it's a solid chipset held back by an old SDK - and that is another problem...

CTF in Broadcom's implementation is much to be desirable. IMO they failed.

Take the same idea and in pure software, the Mikortik guys have done a much better job. That's basically their "fastpath" implementation in newer RouterOS. Pure software level optimisation in the kernel stack, they figure schemes to bypass most of the stack (hence "cutting through") for various usage scenarios. Look at Mikrotik's numbers with fastpath...impressive.

I always found funny when SEM claims Mikrotik doesn't use HW acceleration and accuse another vendor does...hence conclude Mikrotik is superior...lol

Back to the MediaTek SoC, ASIC features such as HWNAT and crypto engine have been utilised in EdgeOS as of v1.9.0.

There is one missing piece. The MediaTek SoC also has HW QoS. It'll be very interesting to watch out and see the performance jump once EdgeOS enables it. But I heard it's quite a piece of challenging work.

 

[System Error Message]

I always found funny when SEM claims Mikrotik doesn't use HW acceleration and accuse another vendor does...hence conclude Mikrotik is superior...lol

In the past this is true but when they came out with fast track i claimed that mikrotik doesnt rely on hardware acceleration as only some of their hardware is benchmark with fastpath or fasttrack. With mikrotik hardware acceleration can be controlled and used in various ways or even as a QoS feature where all the stuff you want to have priority such as gaming can be given fasttrack for lower latency and faster speeds so you can accelerate some traffic only if you choose to. Mikrotik doesnt build their featureset by relying on fasttrack so the numbers they get for their higher end routers are software only as that is the likely scenario their customers are going to be using it with, software NAT with firewall and QoS. With ubiquiti however their entire marketing relies on their hardware acceleration which many of their customers will end up finding to be useless in their scenario. The marketing and comparisons done by ubiquiti is what i scorn at.

 

[kvic]

Re-visited forwarding speed in ER-X to satisfy my own curiosity! Did a bunch of switching and routing tests. I think it would be interesting to some of you too.

http://kazoo.ga/revisit-forwarding-speed-in-er-x/

In short, ER-X shall excel at 500/500 WAN or below. Pls refer to the write up for details.

cc @pege63

 

[kvic]

IPsec performance benchmarked. I use AES-128/SHA1 for ESP and AES-256/SHA256 for keying. By all means go with AES-256/SHA256 for ESP and give yourself extra comfort at zero extra penalty.

Note that one can only enjoy AES-256/SHA256 on ER-X platform with the Mediatek SoC. No where else in the edgerouter line has crypto acceleration for SHA256! Not even in ER-Pro.

Read details here: http://kazoo.ga/edgerouter-x-ipsec-benchmarked/

Impressive throughput given ER-X a little device:

 

[kvic]

My original blog post on the switch in ER-X receives many hits from around the world. While the information provided there is mostly true for MT7621A SoC, there is a catch in Ubiquiti's implementation in ER-X (and ER-X-SFP).

I realised the pitfall through tests of my own ER-X. Now the finding is summarised in this new post: http://kazoo.ga/re-visit-the-switch-in-edgerouter-x/

Believe it or not I was unable to find such info about ER-X mentioned anywhere on the net. If I did, perhaps I would have thought twice before my ER-X purchase...even thought I would still buy it at the end as practically little impact to my use case.

It's this reason that I won't recommend ER-X for 1000/1000 Internet until Ubiquiti 'fixes' its firmware. With competition from Mikrotik in its new hEX aka RB750r3, maybe everyone has a reason to re-consider...

 

[Terry Inge]

Thanks guys for this awesome thread.

I'm looking for a wired-only (don't need the wi-fi since I have APs for that) router that can handle PPPoE gigabit to deploy across multiple sites and be cost effective (residential customers).
Padavan and MediaTek SoC are front-runners and looking at the CPU in both ER-X and hEXr3 I'm baffled on their poor performance compared to Padavan on N56U (A1 or B1). Unless there's a hardware snafu, it shows the firmware for both UBNT and MKTK is seriously lacking - and a testament to Padavan's skills...

In my case, having Padavan on the two devices above would rock but it looks highly unlikely...

 

[System Error Message]

@kvic looking at your tests, why is the upload much slower than download? This is forwarding right, not NAT?

For business or even industrial routers upload and download speed shouldnt differ especially if you're testing each direction at a time.

 

[sfx2000]

CTF in Broadcom's implementation is much to be desirable. IMO they failed.

Broadcom's CTF isn't that bad - it's hamstrung a bit by the SDK that supports chips with and without, and part of the challenge there is documentation, or lack thereof, of how to leverage the CTF.KO that they include...

Part of the challenge with cut-through switching is that it's basically a short-cut or fast lane, and with predictable traffic, it can be very fast, as it looks at the header and makes decisions, unlike store and forward switching... so one can do more with less so to speak. The challenge is with integration with other applications, where there might be additional levels of treatment, like in a gateway router w/NAT and QOS vs. a simple layer 3 router that CTF can get into trouble - and even there, there are some simple rules that can work...

The challenges with CTF are not unique to Broadcom, FWIW...

 

[sfx2000]

It's this reason that I won't recommend ER-X for 1000/1000 Internet until Ubiquiti 'fixes' its firmware. With competition from Mikrotik in its new hEX aka RB750r3, maybe everyone has a reason to re-consider...

I don't think there is a fix needed to be honest - within the HW constraints, and the cost, it's an honest 500/500 Mbps device that can do a decent job, and that ain't bad...

That it has a strong feature set and an approachable WebGUI, that's a plus... again, considering the price...