A new round of router tests by Ars Technica

[kvic]

Article: http://arstechnica.com/gadgets/2016...build-faces-better-tests-tougher-competition/

MikroTik hEX beats Ubiquiti USG by a mile. USG shares the same hardware as ER-lite.

Personally I suspect some configuration/FW issue in the USG...in the benefit of doubt..

@System Error Message, a bit surprise you didn't break this news to us!

 

[L&LD]

Thanks for the link!

I am also surprised how much better the Kong firmware works for connections up to 200Mbps too.

The Ubiquiti EdgeRouter Pro, while technically the 'winner' in this write-up, is a long way from mainstream use in a home (much too loud). While the price is not unreasonable, unless someone has a 'network closet/room' and a proper rack to mount this on, this piece of equipment is relegated to the hardcore users (most are on this forum, and you know who you are ).

Looking forward to seeing the write-up with the wireless testing. Hope it isn't too far away.

 

[System Error Message]

I didnt share the news because this was using hardware acceleration. People dont buy mikrotik for their hardware acceleration, they buy it for use with the features the OS has. Once you start using QoS the speeds like with the ERL drop down to 100Mb/s. Hardware acceleration isnt magic.

The newer mikrotik MIPS is faster than the older gen. The older gen did up to 300Mb/s of NAT and the newer ones seem to manage 400Mb/s of NAT.

my modified mikrotik CCR1036 makes a lot less noise than the ERPRO.

Perhaps we should highlight this article if you think it very relevant to what the majority look for. @thiggins should put the article as a highlight on the main site

 

[YeOldeStonecat]

Thanks for the link!

I am also surprised how much better the Kong firmware works for connections up to 200Mbps too.

The Ubiquiti EdgeRouter Pro, while technically the 'winner' in this write-up, is a long way from mainstream use in a home (much too loud). While the price is not unreasonable, unless someone has a 'network closet/room' and a proper rack to mount this on, this piece of equipment is relegated to the hardcore users (most are on this forum, and you know who you are ).

Looking forward to seeing the write-up with the wireless testing. Hope it isn't too far away.

Love the Pro...run the back end of our server room on one, have quite a few at clients.
He missed a major firmware released by mere weeks...ver 1.9.0
Added a lot of new QoS features and performance improvements.
They even have a new "basic" section in the GUI now for QoS.

The USG just got another firmware upgrade this month, wonder if that smoothed out issues they had at 800 megs on USG.

 

[sfx2000]

Interesting article - OpenWRT and pfSense, out of the box, generally work, but respond well to tuning - so my guess is that to keep things "even" perhaps, was to not do the typical tweaks one does - for example with pfSense, ensuring that PowerD is enabled (for Intel), and the AMD box need the similar tweak with Intel and AMD with the on-die sensor vs. ACPI, which does make a difference..

I'm surprised that the APU1d4 was even considered honestly, as the SG-2220 would have perhaps been a better choice - while the APU1d4 decent device, it's not really going to do well with WAN speeds above 200Mbits - it runs out of breath due to slow clocks (AMD Bobcat dual core @ 1.0GHz) and small caches - it's a nice enough board, just not going to compare well to something that has twice as many cores and much higher clock speeds... (bobcat performs similar to the old-school Atom's (think original Atom, not Silvermont or later).

The rest of the mix makes sense... even though putting the Edge Router Pro into the mix perhaps is a mismatch compared to the other gear... the USG was interesting, but instead of doing the ER-P, they should have considered tossing in the ER-X or ER-L instead..

Also the Big Honking router representative - goes to show that there's some tuning potential for the factory firmware that perhaps Netgear hasn't considered yet when compared to DDWRT.

 

[System Error Message]

Love the Pro...run the back end of our server room on one, have quite a few at clients.
He missed a major firmware released by mere weeks...ver 1.9.0
Added a lot of new QoS features and performance improvements.
They even have a new "basic" section in the GUI now for QoS.

The USG just got another firmware upgrade this month, wonder if that smoothed out issues they had at 800 megs on USG.

I hate the ERPRO, My issue with it wasnt about using it as a router, but trying to use it as a router like pfsense. Trying to run other software on it was a disaster as the software did install and work but wouldnt integrate with the other software, features and the router itself.

My issue with ubiquiti edgerouters is that they rely on tricks and cant really do the job when you actually do something useful on them. Their marketing is horrible and they advertise their weakest link which is speed which relies on hardware acceleration. There are many frustrated customers to bought an edgerouter thinking it will be fast but the moment they use something that breaks hardware acceleration, things get real slow. They shame cisco and make unfair comparisons with mikrotik.

Although mikrotik never relied on tricks, they've been slow on software and it seems like they are going to fall far behind compared to everyone else. They have a huge customer base so they definitely have the money to develop the network features required for their routers and switches but they dont.

So it looks like a linux server is the way to go with making a decent router.

 

[sfx2000]

So it looks like a linux server is the way to go with making a decent router.

Yep, and I would include the various BSD's in that camp - the Linux side does have a larger body of knowledge, but any *NIX box will make a fine router/server in the right hands...

 

[kvic]

I am also surprised how much better the Kong firmware works for connections up to 200Mbps too.

Also the Big Honking router representative - goes to show that there's some tuning potential for the factory firmware that perhaps Netgear hasn't considered yet when compared to DDWRT.

Indeed, DD-WRT performance was another surprise IMO. Seems to me the guys behind DDwrt or Kong truly knows networking stuff and the system optimisation aspect of it!

Just want to emphasise Kong in that test performs _extremely_ well.

 

[kvic]

The USG just got another firmware upgrade this month, wonder if that smoothed out issues they had at 800 megs on USG.

I think he did with the older firmware, and then again after firmware update. I believe that's released this month (but I didn't read his text in detail..lol).

The older firmware actually performs much better. Able to scale across the board. Even though the result is still a disappointment but I would guess it's his mistake in config or a silly bug in the firmware.

Looking at the firmware update? Seems to me ubnt realised there is some design flaw in it and try to address and fails miserably.

I just hope it's not representative of ER-lite and ER-X. USG fw builds on top of EdgeOS. Let's pray it's only USG...

It's "dead" silent on the forum over there. that btw show forums like snb have huge value.

 

[System Error Message]

I think the speed with USG is because of the features it runs. For example try running squid3, squidguard and perhaps an anti virus all to protect web browsing. The ERPRO does this around 200Mb/s (90Mb/s per core).

 

[kvic]

I think the speed with USG is because of the features it runs. For example try running squid3, squidguard and perhaps an anti virus all to protect web browsing. The ERPRO does this around 200Mb/s (90Mb/s per core).

I'm not familiar with USG other than knowing it shares the same EdgeOS baseline. If it's running some kind of real-time anti virus, that could be the culprit. Personally I care less about USG but the little guys like ER-lite and ER-X which run a slightly newer incarnation of pure EdgeOS.

SEM, I have a sincere suggestion to you. Want to start benchmarking routers? You don't have to buy faraday's cage and lease expensive equipment. You can fairly quickly setup the tests like Ars. Try and benchmark various *WRT on consumer all-in-one. It'll be very interesting to audience here and elsewhere. Also provide positive motivation to 3rd party developers and drive the momentum up back to major consumer vendors.

Waiting for your tests *grin*

 

[kvic]

I think the speed with USG is because of the features it runs. For example try running squid3, squidguard and perhaps an anti virus all to protect web browsing. The ERPRO does this around 200Mb/s (90Mb/s per core).

The Ars tests workload can't really stress the potential of ER-Pro. At the extreme of the test, it's only driving around 200 Kpps of "small" packets. ER-Pro is capable of 2 million pps according to ubnt datasheet.

 

[System Error Message]

well i have the hardware to benchmark it but setting it up isnt exactly easy.

Right now i have the CCR1036 and ERPRO which i can both benchmark. To benchmark them i will need to use 2 servers that have quad port NICs, so its more of a question of bonding and how well the test will run with bonding or if there is some sort of complicated setup that i could do but havent figured out yet.

I have just the right number of servers to benchmark the CCR1036, 2 servers have SFP+, 2 have the quad port NICs. Its just a matter of wiring them up (hopefully the SFP+ card would stop falling off the riser) and setting up both the OS, software and the routers themselves. The powerful servers get SFP+ while the less powerful ones get the quad port NICs. It is fun overclocking the LGA 1366 to double their frequencies but im more worried about AMD platforms coping even with overclocking.

I would also need to know what to test, NAT? IPV6? QoS?

 

[YeOldeStonecat]

I think he did with the older firmware, and then again after firmware update. I believe that's released this month (but I didn't read his text in detail..lol).

The older firmware actually performs much better. Able to scale across the board. Even though the result is still a disappointment but I would guess it's his mistake in config or a silly bug in the firmware.

Yeah I read the article..the firmware "out of the box" ran at 200 megs....the updated firmware ran at 800 megs but flakey. But...note the date of the article, it's before the newer firmware I'm talking about was released. Add to that...the actual testing by the author was likely mid summer. So well before the updated firmware released in mid September.

With the different paths of the USG and the EdgeRouters....even though the USG shares the same platform of the ERL...I'd wager performance now differs. The ER series remains lean and mean. The USG has been integrating with broader functionality....converging...into the Unifi controller. She runs a bit more bloated than the ERs. I don't use the USGs at all at our clients because of that....I keep things separate.

 

[thiggins]

What I didn't see what a baseline measurement between the two test machines with no router.

 

[sfx2000]

What I didn't see what a baseline measurement between the two test machines with no router.

His first test run was through a switch, no router in place, to establish a baseline...

 

[System Error Message]

I could get my test bench up and running next week, If you guys could help me by telling me what sort of tests and config it would help.

 

[thiggins]

His first test run was through a switch, no router in place, to establish a baseline...

Sorry, I'm not seeing it. Where is it in the plots?

 

[thiggins]

Ok. I see the bandwidth graph. And I see that the line plots are % of router tested vs. straight connection.

Something looks wonky with the 10Kfile / 10 client test.

 

[kvic]

His first test run was through a switch, no router in place, to establish a baseline...

Ok. I see the bandwidth graph. And I see that the line plots are % of router tested vs. straight connection.

Something looks wonky with the 10Kfile / 10 client test.

Yeah those graphs are generated with NetData. Pretty and modern (vector based).

I find the NetData charts tell interesting stories. Visual comparison makes a huge impact. I posted in the other forum a mini-guide how to read those test charts. Perhaps smart kids and grandpa there all can decipher with a blink of their eyes but not myself. Let me find out and share. Hopefully will be helpful to some readers here.

EDIT:

Three performance metrics the tests try to measure

1. # of concurrent NAT'ed connections

The tests repeat with 10, 100, 1K and 10K concurrent connections. On the result chart, it's respectively represented by the 1st, 2nd, 3rd and 4th green bar (from left to right) of each set of four bars.

2. effect of packet size

"large" size packet, "medium" size packet and "small" size packet. It's represented respectively by the 1st set of four green bars, 2nd set, and 3rd set (from left to right).

3. packet per second

the tests also try to stress the limit of pps. For the "large" size packet, it's designed to test between 90 Kpps - 130 Kpps (from the 1st green bar to the 4th). For the "medium" packet, the tests target between 100 Kpps - 130 Kpps. For the "small" packets, the tests hammer between 160 Kpps - 200 Kpps.

These are rough numbers I got from looking at the reference chart (of the switch).