ER-X: small is beautiful

[Nullity]

I think fq_codel is doing very well on upload. Good but less so on download. Without fq_codel, download is even worse. Lesson learned.

Now even though fq_codel only has a few knobs (probably not true if including other kernel parameters of the network stack!), the big question is how to improve on download lag?

To improve download bufferbloat, the most common way is to NEVER allow full use of your connection, which means preemptively limiting your max download speed.

It always takes a small amount of time for a remote TCP sender to obey your request to slow down. If your max download rate is reached while the remote TCP sender is still sending faster than you can receive, you experience bufferbloat (likely caused by your ISP's equipment at the network node that rate-limits to whatever bitrate tier you are paying for). So, you need to rate-limit your download to approximately 90% to even 50%, depending on how unpredictable your network is.


Toastman explains it best in his QoS tutorial, IMO: http://www.linksysinfo.org/index.php?threads/qos-tutorial.68795/


There are other factors when discussing download, like the idea that buffering is practically useless when moving from a low-bandwidth node (10Mbit) to a high-bandwidth node (1Gbit) since there is practically no congestion, meaning only traffic-policing (no buffering) should be used rather than traffic-shaping (buffered). This lack of buffering would also limit the need for scheduling (reordering of packets), which would simplify the whole setup even more.

 

[sfx2000]

HFSC literally has nothing to do with bufferbloat as it is only a scheduler (it decides the order, not queue depth).
During a fully saturating upload, with my pfSense install, my HFSC traffic-shaping setup will have a 600ms ping with CoDel disabled. With CoDel enabled I get 35ms average and 55ms maximum. Average queue depth (pftop) is 1-3 packets with CoDel.

@Nullity - go back and learn about HFSC - all QoS schedules, and most do queue management - which is needed obviously for QoS in the first place.

HFSC - It is a traffic shaper (and one of the most comprehensive ones out there) - about the only thing missing from it is Stochastic Fairness Queueing, which can be handled under the hood - not advisable, and as such, it's kind of picking nickels up compared to the primary HFSC implementation...

Yes, on pfSense one can have codel on individual shapers as well if needed - pfSense has a lot of knobs to turn and twist.

Remember, once outside of the gateway, there's nothing that can be done about Bufferbloat in any event, and most any kind of QoS/Shaping will help on that GW..

FWIW - if you truly have a 600mSec ping time without Codel, go back and look at your shaper config, as something is definitely not right...

Without Codel - 10 days worth of pings to www.google.com - pretty consistent at 10.3 ms average, with a min of 8.4, a max of 23.7... see below

So some might not need codel, try it and see - as for me, I'm ok without it.

 

[sfx2000]

Nice writeup here on the pf, hfsc, and bsd tuning - might want to review...

https://calomel.org/pf_hfsc.html

 

[Nullity]

Nice writeup here on the pf, hfsc, and bsd tuning - might want to review...

https://calomel.org/pf_hfsc.html

Actually no, that is not a good writeup. They do not even understand the most novel feature of HFSC which is the decoupling of bandwidth and delay by employing HFSC's m1 & d parameters. Calomel says m1 & d set the "initial bandwidth assignment" which is false. That site is not well respected over on the pfSense forum.




Additionally, you say HFSC does not have Stochastic Fair Queueing which is not accurate. HFSC is "Hierarchical Fair Service Curve" which means that HFSC does employ Fair Queueing as it defines it in the HFSC paper. Fair Queueing has no official definition.

Stochastic Fair Queueing is actually WORSE (less perfect, hash collisions do happen) than HFSC but it's novel improvement over earlier Fair Queueing algos is that it is less processor intensive.

 

[Nullity]

@Nullity - go back and learn about HFSC - all QoS schedules, and most do queue management - which is needed obviously for QoS in the first place.

HFSC - It is a traffic shaper (and one of the most comprehensive ones out there) - about the only thing missing from it is Stochastic Fairness Queueing, which can be handled under the hood - not advisable, and as such, it's kind of picking nickels up compared to the primary HFSC implementation...

I am an HFSC addict. I have read all cited papers from HFSC's paper and even all the paper's cited by those papers...

The HFSC paper is only a dozen or so pages long, so it may be easier for you to cite a passage that proves that HFSC controls queue depth. I never saw such a passage. I could babble more about HFSC but like I said, it is likely more efficient for you to just read the paper yourself and paste a passage disproving my claim that HFSC's algorithm does not concern itself with queue depth.



Maybe we should create a new thread dedicated to this topic (QoS, traffic-shaping, bufferbloat, etc)?

 

[sfx2000]

@Nullity - actually it is - HFSC is fairly complex, and he tries to explain it - to a certain extent he does...

I fully understand the mechanics behind codel, fq_codel, cake and other QoS schemes - and this is why I chose to go down the path that I took - testing with codel on and off yielded zero benefit in my case, so it's off, as it is not needed in my use case and network layout. Doesn't mean it's the best approach, but it's one that works for me - and I've posted the results to confirm that.

Unlike most, I have been deep inside router design, and also have had chances to work with very large networks with much more bandwidth and capability than most - and as a former member of IETF, IEEE, and 3GPP2 - it was my job to grok the intimate details of things like QoS.

In any event - use what works - if codel works better for you - that's pretty awesome.

But again - I'll return to my original statement - Bufferbloat is vastly overrated - it gets a lot of attention, and that's good, but there's little impact once the connections get above 100Mbps.

 

[sfx2000]

Additionally, you say HFSC does not have Stochastic Fair Queueing which is not accurate. HFSC is "Hierarchical Fair Service Curve" which means that HFSC does employ Fair Queueing as it defines it in the HFSC paper. Fair Queueing has no official definition.

Stochastic Fair Queueing is actually WORSE (less perfect, hash collisions do happen) than HFSC but it's novel improvement over earlier Fair Queueing algos is that it is less processor intensive.

If you use this in conjunction with other schemes, it works... and it's there for people to play with in the Linux kernel...

 

[sfx2000]

The HFSC paper is only a dozen or so pages long, so it may be easier for you to cite a passage that proves that HFSC controls queue depth

The kernel defines the queue depth - so the queues are already there - it's up to HFSC on how to manage and shape the queues...

 

[buddyp]

I bought an ER-X based on the rave reviews here and I'm really happy with it. It took over all the stuff I was doing on my ASUS AC-66 running a version of Merlin's code that I hacked up and is now running a real Merlin build in dumb AP mode.

I'd be very interested in seeing more info on hardening the ER-X. I'm using the Emerging Threats blocklist already but I'd like to have something like fail2ban as well.

I'd also really like to know more about your "super" AP. I was a little disappointed in how much functionallity disappeared when I clicked the button for AP mode. Not that I'm a fan of DD-WRT, but it will still act as a VPN server when in AP mode. The ASUS has a pretty decent CPU and plenty of RAM that is pretty much all unused when acting as a simple AP. Could be put to good use doing something else.

 

[sfx2000]

The Ubiquity ER-X and the MicroTik hEX are starting to turn the flow against "big honking" Consumer Router/AP's, as they tend to be very focused - and this is a good thing...

There is a growing collective of SNBForum members that have gone down this route, or taken that next step with a bit more...

Personally - I'd like to see a major consumer vendor - like Linksys, Netgear, D-Link, or Asus bring in an open router type of box - either X86 or ARM, but with enough memory and storage to take things a bit further...

Routers, even in the home context, do not need to have WiFi these days - just give us an open platform and see what we can do with it - the first vendor that gets this item - you might win the jackpot!

 

[System Error Message]

- what she said "small is beautiful"

Havent seen someone make a joke on that yet.

 

[sfx2000]

FWIW - I think the hive mind, at least some, are migrating away from the "big honking routers" into something is that is better suited...

The big honking routers all have the same issue - old code inherited from the WRT54G Linksys drop - and that's ok... the Broadcom SDK has direct DNA back to that code base...

When we look at alternatives - VyOS is a good example, so is FreeBSD and similar BSD's - all of a sudden those retail AC5300 boxes don't fare so well...

And VyOS/FreeBSD - that's the now - what about tomorrow, and how do we scale?

OpenSNB might be an answer there...

 

[Nullity]

@kvic

Have you made use of the CLI yet? I have read good things about VyOS's CLI. (It's modeled after Cisco's IOS CLI, I think, which I really enjoy using.)

One of my biggest complaints with pfSense is the inability to further tweak things via CLI. Practically everything must be done through the GUI.

 

[kvic]

I bought an ER-X based on the rave reviews here and I'm really happy with it. It took over all the stuff I was doing on my ASUS AC-66 running a version of Merlin's code that I hacked up and is now running a real Merlin build in dumb AP mode.

Good to hear from you. We also have kenz71 in process of setting up his. Many other before us like Trip and a few other handles I can recall.

I'd be very interested in seeing more info on hardening the ER-X. I'm using the Emerging Threats blocklist already but I'd like to have something like fail2ban as well.

In my Asus when as router, I ran a small ban list that poked on my three open ports. It's for fun. The rest port scanners I simply ignore and drop. On ER-X, I haven't found a good way to manage such a list.

Might be possible to use hooks that could run user scripts. I saw one for running HE tunnel update. The hook is well designed, unsure from Debian or EdgeOS/Vyatta.

I'm actually more a believer of ignore/drop and whitelisting. Simpler and less work for the router

I'd also really like to know more about your "super" AP. I was a little disappointed in how much functionallity disappeared when I clicked the button for AP mode. Not that I'm a fan of DD-WRT, but it will still act as a VPN server when in AP mode. The ASUS has a pretty decent CPU and plenty of RAM that is pretty much all unused when acting as a simple AP. Could be put to good use doing something else.

We're in agreement here. No one wants to waste a piece of good hardware as a dumb AP. I'm running DNS, OpenVPN and a couple other daemons on my RT-AC56U as AP. Yesterday the idea of running SiLK as NetFlow collector flashed across my mind..

Indeed AP and repeater modes in all-in-one's receive little love from vendors. OpenWRT/DD-WRT may show a bit more care on the 3rd party world. I plan to cover some aspects of my RT-AC56U as AP. Be patient with me..

 

[kvic]

@kvic

Have you made use of the CLI yet? I have read good things about VyOS's CLI. (It's modeled after Cisco's IOS CLI, I think, which I really enjoy using.)

One of my biggest complaints with pfSense is the inability to further tweak things via CLI. Practically everything must be done through the GUI.

Thanks for your inputs and sfx's about QoS. Many food for thought and I need to find time and play around. Will report back on QoS to the thread.

On the CLI, the very first few moments I felt like all limbs being tied up. Something straight forward in Linux becomes a strange place in EdgeOS/Vyatta. I quickly get used to it and I think I start loving it..

The only thing that troubles me for the moment is auto completions, without sudo as root, auto completion only works with Vyatta commands not Linux files/paths/commands.

edit:
Elaborated in a bit more detail my initial impression of the CLI and the Tab completion problem: http://kazoo.ga/edgerouter-x-meeting-the-cli

 

[buddyp]

@kvic

Have you made use of the CLI yet? I have read good things about VyOS's CLI. (It's modeled after Cisco's IOS CLI, I think, which I really enjoy using.)

I believe the CLI is modeled after the JUNOS CLI. Its not a lot like the Cisco CLI, but if you can you use the Cisco CLI you will pick up the VyOS one in a few minutes.

 

[KenZ71]

Finally got my ER X running as well, pretty cool. Literally!

I'll start another post for a question I have so I don't hijack this one.

 

[kvic]

I believe the CLI is modeled after the JUNOS CLI. Its not a lot like the Cisco CLI, but if you can you use the Cisco CLI you will pick up the VyOS one in a few minutes.

Zero Cisco nor Juniper CLI experience here but over 20 years of Linux shell experience. It should also be quick for seasoned Linux users to pick up the CLI in EdgeOS. To be proficient perhaps need a bit more time.

For example, to configure one IPv6 firewall rule, I type:

set firewall ipv6-name WAN_LOCAL-v6 rule 40 action accept
set firewall ipv6-name WAN_LOCAL-v6 rule 40 description "Packet Too Big"
set firewall ipv6-name WAN_LOCAL-v6 rule 40 protocol icmpv6
set firewall ipv6-name WAN_LOCAL-v6 rule 40 icmpv6 type 2

Lots of redundant typing..in bold.

Is there a way to position "myself" at the "node" rule 40 and reduce the verbosity? I could think of such a feature shall exist but I haven't looked into details yet.

Would be good to hear this and other tricks from Edgerouter users

EDIT:

My example maybe too easy. To position myself at "rule 40". Only have to type:

$ configure
$ edit firewall ipv6-name WAN_LOCAL-v6 rule 40​

then I could

$ set action accept
$ set description "Packet Too Big"
$ set protocol icmpv6
$ set icmpv6 type 2​

To go back to previous level, type

$ exit​

..back to the root of the tree in our case. Type exit again will quit Configure mode.

 

[kvic]

The HW crypto in my new toy..

http://kazoo.ga/the-crypto-engine-in-edgerouter-x/

One third the price. x4 the IPsec throughput of a "cisco vpn router".

MediaTek shall receive a better consumer impression IMO.

 

[buddyp]

EDIT:


..back to the root of the tree in our case. Type exit again will quit Configure mode.

If you don't want to exit config mode, you can type 'up' to go out one node, or 'top' to go all the way out to the root