ER-X: small is beautiful

[kvic]

This little device is a piece of joy to hold in hand and simply look at it. That pretty much sums up my impression everything about ER-X, including its performance. Too much to praise both aesthetics and performance...let alone its price. Just some quick comments for now.

On and off I think I spent about 8 hours in total setting it up for my LAN at home. This also included converting my Asus all-in-one into a "super" access point, wasting a few hours I believe on following a IKEv2 IPsec server guide on UBNT forum (ended up scratching that and came up with my own better way imo), and adjusting configs in both ER-X and the Asus "super" AP for their split roles.

On ER-X, I have eth0 as WAN. eth2-4 as main LAN and also part of switch0. eth1 as LAN2 Also have a HE 6in4 tunnel. Asus "super" AP is attached from its WAN port to eth4 on ERX. Both devices together can provide 7 usable LAN ports. Too many for me so decided to have eth1 on ERX for a separate LAN. Not sure about its use but I know it will be fun.

IPv4 setup on ERX is straight forward as it can be done from its WebUI. Everything IPv6 has to go with its CLI. This includes IPv6 firewall. Don't forget to set it up if you have native IPv6 or a HE tunnel (I had IPv6 wide open for more than a day). I have switch0 on ERX and br0 on Asus "super" AP static IPv6 addresses. I can't recall why I need a static address on Asus but perfectly doable. Here is a screenshot of my ER-X Dashboard:


The Smart Queue under QoS is very easy to setup. Under the hood, it runs FQ-CoDel assisted by HTB. Went with the default mostly. Punch in 100/100 bandwidth. Click Apply. That's it.

This little device amazes me in many ways. If I have to mention one for now, it's the superb IPv6 stack. I don't have native IPv6 from my ISP yet but the HE tunnel seems to work much better in ERX than Asus (when it was a router). I used to only get occasionally max. ~60Mbps on Asus. Now consistently getting more than that on ERX. Surprise. Dslreports' test gives the tunel A+/A+/A rating. Regardless how much substance in the ratings, I'm happy to see many A's. *grin*

EDIT

List of write-up:

http://kazoo.ga/re-visit-the-switch-in-edgerouter-x/
http://kazoo.ga/edgerouter-x-ipsec-benchmarked/
http://kazoo.ga/revisit-forwarding-speed-in-er-x/
http://kazoo.ga/er-x-forwarding-speed/
http://kazoo.ga/erx-use-tls-certificate-for-gui/
http://kazoo.ga/edgerouter-x-smart-queue/
http://kazoo.ga/the-crypto-engine-in-edgerouter-x/
http://kazoo.ga/the-switch-in-edgerouter-x/
http://kazoo.ga/edgerouter-x-meeting-the-cli
http://kazoo.ga/edgerouter-x-ssh-auto-authentication/
http://kazoo.ga/edgerouter-x-small-is-beautiful/

 

[pete y testing]

whats the asus "super " access point lol , so you have just wan bypassed your dsl-ac68u right and using the ubiquiti as the router ? why it took you 8 hours to set it up is a bit strange

Dslreports' test gives the tunel A+/A+/A rating.

on 100/100M your always going to get that result no matter what you use

 

[kvic]

whats the asus "super " access point lol , so you have just wan bypassed your dsl-ac68u right and using the ubiquiti as the router ? why it took you 8 hours to set it up is a bit strange

Thanks for your comments There was a bit learning curve there for a EdgeOS/Vyatta newbie like myself. Those eight hours were well spent. But you're right. ER-X is super easy to set up. The first 15 minutes spent can cover 90% of all home use I believe.

I hope to elaborate on turning my RT-AC56U into a "super" access point on another day. lol

on 100/100M your always going to get that result no matter what you use

Hmm..I would disagree. Certainly not in my experience even with native IPv4. The A+/A+/A result was conducted on a Hurricane Electric 6in4 tunnel. So there was additional overhead. I've never seen such good test report on my Asus (when it was a router).

Even testing on native IPv4, I've never seen triple A's on Asus. With Adaptive QoS, the best buffer bloat score I got on Asus was an A . Without QoS, a B+ at best. And that's native IPv4. Forget about through a 6in4 tunnel.

 

[System Error Message]

Did you try dslreports using fiber? Every profile is different. Using DSL or cable doesnt use as many channels/tests as fiber does so it doesnt really stress the router.

 

[kvic]

Did you try dslreports using fiber? Every profile is different. Using DSL or cable doesnt use as many channels/tests as fiber does so it doesnt really stress the router.

I use dslreports "gigabit/fiber" profile for my tests.

You're damn right about different profiles having impact on final score, and how the actual tests conducted behind the scene.

edit:

dslreports' test was not on my checklist to begin with. After finished the setup of my LAN, I was about to enjoy some video streamed over IPv6. I could see the video is crisp! I thought what wonder ER-X did.

To figure out I checked the video bitrate. I found it was streaming at 3Mbps. That was a huge jump from the usual 1.8Mbps I was watching. The streaming service dynamically adjusts based on network performance. That's how I started looking at HE 6in4 tunnel performance on ER-X

 

[Trip]

on 100/100M your always going to get that result no matter what you use

Sorry pal, but in 99% of cases that statement is dead wrong. Saturate a pipe of *any* width without proper QoS and packet handling and you're headed for all kinds of degradation: bufferbloat, latency and jitter just to name a few.

As a side note, we've deployed several dozen ER-X's and they're often great for anything under 150Mb of total WAN -- ie. all DSL and most cable deployments in the US (hate to break it to folks, but the national average for throughput is still <15Mb/s download as of Q4 '15). For triple-digit Mb cable or fiber WAN, you have to start going to more powerful options to get full utilization with even a small amount of CPU-based services active.

 

[KenZ71]

Ashamed to admit I've tried and failed 3 or 4 times to setup my Edgerouter.

My own fault, I am rushing through the directions to restore service so the family can get back to Netflix. I'll have to send them to the grocery store next time I try.

 

[pete y testing]

Sorry pal, but in 99% of cases that statement is dead wrong. Saturate a pipe of *any* width without proper QoS and packet handling and you're headed for all kinds of degradation: bufferbloat, latency and jitter just to name a few.

i stand by the comment and as system error message has stated it depends oh how it was tested

As a side note, we've deployed several dozen ER-X's and they're often great for anything under 150Mb of total WAN -- ie. all DSL and most cable deployments in the US

hate to break it to you but this site is populated by more than just americans and we see many here with 500M plans etc so a total throughput of 150M is pretty low these days

yes its cheap and does a good job but lets not sugar coat a very small niche market device that many wouldnt want to or wish to spend the time to setup

I hope to elaborate on turning my RT-AC56U into a "super" access point on another day. lol

lets not make it more than it is , its a simple access point , nothing super about it

its quite simply this easy

https://www.asus.com/support/faq/1005259

and turns any asus router into a switch and access point only

 

[sfx2000]

Sorry pal, but in 99% of cases that statement is dead wrong. Saturate a pipe of *any* width without proper QoS and packet handling and you're headed for all kinds of degradation: bufferbloat, latency and jitter just to name a few.

Bufferbloat with WAN connections at or above 100Mbps is generally not a problem...

In any event, it's vastly overrated in a general context...

 

[sfx2000]

Ashamed to admit I've tried and failed 3 or 4 times to setup my Edgerouter.

My own fault, I am rushing through the directions to restore service so the family can get back to Netflix. I'll have to send them to the grocery store next time I try.

Middle of the night - have a plan written up, and execute... and a rollback if it doesn't work..

ER's aren't that much different than pfSense or RouterBoards - need to have a plan when deploying. Then it will work...

 

[kvic]

Bufferbloat with WAN connections at or above 100Mbps is generally not a problem...

In any event, it's vastly overrated in a general context...

When clients can saturate the WAN bandwidth, regardless its speed 100Mbps or above, they are strived of the scarce resource. Latency, jitters and dropped packets all increase. Whether it's called buffer bloat or not doesn't matter much. The problem is there.

The problem isn't there for home users 'cos rarely people could stress the limit of upload bandwidth when >= 100Mbps. Given that said by the nature of "cheap" fiber to the home deployed around the world, upload will be more likely to suffer from "buffer bloat" than download. In my ER-X setup, I actually only enable Smart Queue on upload.

 

[System Error Message]

Bufferbloat with WAN connections at or above 100Mbps is generally not a problem...

In any event, it's vastly overrated in a general context...

Not true, bufferbloat regardless of bandwidth is important, it really depends on how QoS is set up and games nowadays are starting to use more bandwidth. Even with a 1Gb/s connection the gamers on the same network as torrent users

will essentially run into issues. QoS setup also plays a big role as i've managed to get A+ bufferbloat rating but with a D in quality from packet loss dealing with the inherent nature of VDSL as speeds arent consistent. I also found my speeds dropping from interference as more and more iny my area are ditching virgin media after finally realising how unreliable cable (VM is stingy on upload as you get 50% claimed) is but with VDSL1 + vectoring which is what BT uses, with more subscribers you get interference and cross talk. Sadly they have no plans to upgrade to vdsl2 and fttp seems to in the winds for them.

 

[kvic]

As a side note, we've deployed several dozen ER-X's and they're often great for anything under 150Mb of total WAN -- ie. all DSL and most cable deployments in the US (hate to break it to folks, but the national average for throughput is still <15Mb/s download as of Q4 '15). For triple-digit Mb cable or fiber WAN, you have to start going to more powerful options to get full utilization with even a small amount of CPU-based services active.

This advice might be true when ER-X first came out . The firmware is more capable now with a few HW offload implemented as you know.

In my ER-X setup, I don't have HW offloads enabled except IPsec offload. Below is my Dashboard with a up/down combined bandwidth of ~150Mbps (tx+rx of eth0):



The CPU utilisation is about 60%. Around 10% is actually the overhead of the HE 6in4 tunnel. Note that most of download traffic is through the HE tunnel (i.e. tun0). So there are much room for higher speed Internet.

I don't have HW offload enabled (except for IPsec). I have: 1 NAT rule, 12 IPv4 firewall rules, 9 IPv6 rules, and QoS enabled.

 

[sfx2000]

Not true, bufferbloat regardless of bandwidth is important, it really depends on how QoS is set up and games nowadays are starting to use more bandwidth. Even with a 1Gb/s connection the gamers on the same network as torrent users

It's conditional - but again, I'll posit that it's generally not an issue once one hits 100Mbit/Sec on the WAN side - and outside of consumer Router/AP's, most folks running something like RouterOS, pfSense, or VyOS generally don't need to worry about it - lower BW connections, and some modems/routers - yes, it's a concern...

The challenge is that QoS is sometimes app specific, so what works for one, doesn't work for others - which consumer AP's try to apply as a blanket policy...

Going to something a bit more sophisticated and agile - there's a lot of knobs to turn and levers to pull - and there is no one single way that will meet everyone's needs - but being a bit knowledgable about QoS, and application needs, one can find a reasonable solution...

Folks rave about fq_codel, and rightfully so - but I don't use it,

I'm actually using an alternate approach, and it generally works - it's part of the BSD stack, and FreeBSD has done a fair amount of work there - Hierarchical Fair Service Curve (HFSC) which is a bit on the older side, but if one puts in the effort, it works very well - esp for VOIP and links that have variable quality..

I'll state again however - bufferbloat is vastly overrated - it's a concern, but it's not a red-light we must do everything kind of items...

 

[kvic]

Folks rave about fq_codel, and rightfully so - but I don't use it,

I'm actually using an alternate approach, and it generally works - it's part of the BSD stack, and FreeBSD has done a fair amount of work there - Hierarchical Fair Service Curve (HFSC) which is a bit on the older side, but if one puts in the effort, it works very well - esp for VOIP and links that have variable quality..

I'll state again however - bufferbloat is vastly overrated - it's a concern, but it's not a red-light we must do everything kind of items...

fq_codel just happens to be there on ER-X and ubnt makes it very easy to start using it. Not a checklist requirement for me. But I must say fq_codel performs very well. Regardless of my WAN condition at different hours, buffer bloat consistently has A+ (according to dslreports). Video continues to play smoothly (stutters occasionally but rarely) while up/down are saturated.

Good that we have choices. HFSC should work very well when configured properly. Personally I'm sure to screw up.

Did a little read up on codel guys. I believe they popularised the term "buffer bloat". Dslreports picked up later and started its brain wash campaign through its new test. A decade ago old folks in dslreports started the first (?) speed test and those guys left to start speedtest.net. I think the revenue from online ad shall be very handsome. From that perspective bufferbloat blowing up huge is in their interest.

The codel guys appear like serious ppl. Some contributors use Network Simulator to test the algorithms. Now to me, when ppl deploy ns3 (i played with ns2 in college) for simulation, it's serious science. They make qos easy and free to use for everyone, that's to me a leap in civilisation for mankind.

For EdgeRouter users, a brave soul compiled cake for EdgeOS. I think his binary ko only works on Cavium based models. Little traction at the moment but you may check out details here.

 

[sfx2000]

The codel guys appear like serious ppl.

FWIW - they are serious people - and much of the research has benefited all - primarily those on 3G/4G wireless, where RTT and latency, along with available bandwidth are highly variable... and here fq_codel and codel schemes are extremely effective...

 

[kvic]

Going to share from time to time some of the configs I went through on ER-X, and perhaps later on RT-AC56U.

First, a mini how-to enable SSH auto login in ER-X/EdgeRouter's: http://kazoo.ga/edgerouter-x-ssh-auto-authentication/

TL;DR

Copy your public SSH key file to ER-X:

$ scp ~/.ssh/id_rsa.pub ubnt@erx:/home/ubnt ​

Next login ER-X, then:

ubnt@ER-X~$ configure
ubnt@ER-X~$ loadkey ubnt /home/ubnt/id_rsa.pub
ubnt@ER-X~$ commit
ubnt@ER-X~$ save
ubnt@ER-X~$ rm /home/ubnt/id_rsa.pub​

 

[kvic]

The thread seems so far more focus on buffer bloat (and dslreports' test in particular). Why not let's dig a bit deeper into fq_codel.

I had a closer look into dslreports grading system and underlying measurements. I still have no idea how it reaches its final score. But I found that some of its underlying data are useful tools to gauge how fq_codel performs. Assume its measurements are actually accurate. Here are my data from dslreports:

Idle latency and its variation:

Download latency and its variation under stress:

Upload latency and its variation under stress:

I think fq_codel is doing very well on upload. Good but less so on download. Without fq_codel, download is even worse. Lesson learned.

Now even though fq_codel only has a few knobs (probably not true if including other kernel parameters of the network stack!), the big question is how to improve on download lag?

 

[sfx2000]

If this helps... 150/10 connection over cable - pfSense 2.3.2 with HFSC on Netgate RCE-V-2440 -
Client side is Mac Mini 2012 (OSX 10.11.6) over ethernet with Safari 9.1.3

Idle

Upload

Download

 

[Nullity]

HFSC literally has nothing to do with bufferbloat as it is only a scheduler (it decides the order, not queue depth).
During a fully saturating upload, with my pfSense install, my HFSC traffic-shaping setup will have a 600ms ping with CoDel disabled. With CoDel enabled I get 35ms average and 55ms maximum. Average queue depth (pftop) is 1-3 packets with CoDel.

(Download is a totally separate conversation since AQMs are usually unneeded.)

Bufferbloat is a problem for many consumers, which is why DOCSIS 3.1 includes PIE: http://www.cablelabs.com/how-docsis-3-1-reduces-latency-with-active-queue-management/



PS - I have found that the dslreports bufferbloat test is/was not accurate (~25% off, in my case). I think it's because all tests/measurements are ran in-browser.