ER-X: small is beautiful

[sfx2000]

The HW crypto in my new toy..

Nice for L2TP/IPSec - not so good for OpenVPN, as OpenVPN is userland still, and the SDK doesn't expose the capability there

But that's an OpenVPN problem, not an EdgeRouter problem...

 

[kvic]

If you don't want to exit config mode, you can type 'up' to go out one node, or 'top' to go all the way out to the root

Works very well. With prompt showing which level I'm at:

ER-X:~ $ configure
ER-X:~ $ edit firewall ipv6-name WAN_LOCAL-v6 rule 40
[edit firewall ipv6-name WAN_LOCAL-v6 rule 40]
ubnt@ER-X# up
[edit firewall ipv6-name WAN_LOCAL-v6]
ubnt@ER-X#
[edit firewall ipv6-name WAN_LOCAL-v6]
ubnt@ER-X# up
[edit firewall]
ubnt@ER-X# edit ipv6-name WAN_LOCAL-v6
[edit firewall ipv6-name WAN_LOCAL-v6]
ubnt@ER-X# up
[edit firewall]
ubnt@ER-X# top
[edit]
ubnt@ER-X# exit
exit
ER-X:~ $​

 

[kvic]

Nice for L2TP/IPSec - not so good for OpenVPN, as OpenVPN is userland still, and the SDK doesn't expose the capability there

But that's an OpenVPN problem, not an EdgeRouter problem...

I've been trying to look for some evidence that MediaTek's SDK does not include support for HW crypto in OpenSSL.

My gut feeling is it's included but is not used for the obvious reason: ER-X platform models can do but Cavium based models can't. Not so nice skew to their product line-up.

 

[kvic]

Riddle on the download lag resolved. CTF made it again. Congrats Broadcom.

It introduces extra lag (contrary to saving it!) in the datapath. With CTF off, about 20ms lag shed off (compared to that in #18 same dslreports test but they changed graph recently):

The reduction in jitters perhaps partly contributed by CTF off and mainly other tunings. I probably won't be able to reconcile exactly what are those as tunings on fq_codel & tcp/ip stack are done over several days & nights.

For gamers, if you land here accidentally, you know what to do on ASUS routers *grin*

 

[sfx2000]

I've been trying to look for some evidence that MediaTek's SDK does not include support for HW crypto in OpenSSL.

Give this a try... this should tell you pretty quick if HW accel for OpenSSL is built or not...

To test to baseline

openssl speed aes-256-cbc

Then specify the engine in envelope mode...

openssl speed -evp aes-256-cbc

compare the numbers

 

[System Error Message]

theoratically cut through forwarding (which is CTF) is supposed to reduce latency since packets are forwarded by their headers rather than entire packet. Just like the difference between store-and-forward switch to CTF switch.

The problem with mediatek and other chinese companies is the documentation. They produce chips that are alright but the documentation tends to be where the issue is. For example the raspberry pi compatibles tend to have problems with documentations, licenses, support.

The fun thing to know would be how the chips compare (power use idle vs load, voltages, clocks like how people compare chips for overclocking with x86).

 

[buddyp]

@kvic, Where did you find the MediaTek SDK?

 

[sfx2000]

theoratically cut through forwarding (which is CTF) is supposed to reduce latency since packets are forwarded by their headers rather than entire packet. Just like the difference between store-and-forward switch to CTF switch.

Keep in mind that Mediatek bought Ralink - so the docs for low level stuff might be there when searching around - also keep in mind that CTF as a concept is very broad...

CTF is an interesting from a traffic as opposed to Store and Forward with regards to routing - CTF is a "short cut", e.g. it makes some assumptions mathematically compared to Store/Forward (which is pretty much absolute)

CTF - it can be a good thing, but it does depend on the system engineering on the back end - and various vendors do their thing, and many times this ends up being "secret sauce", to whit the closed source ctf.ko that we see with Broadcom.

Don't consider Broadcom's CTF implementation with what is happened with the ER-X - might be Ralink's implementation (perhaps) or UBNT's implementation (which is possible).

 

[sfx2000]

The reduction in jitters perhaps partly contributed by CTF off and mainly other tunings. I probably won't be able to reconcile exactly what are those as tunings on fq_codel & tcp/ip stack are done over several days & nights.

Just note that DSL Speedtest, along with others, are depended on the Browser implementation - which makes this benchmark a bit unreliable when compared Apples to Oranges...

I can obtain different results across ARM/X68 on the same platform, and poor correlation across different platforms - e.g. compare i5-4260 on MacOS (MacBook Air) on Safari vs Chrome on Windows10 with an i7-4790, vs N3700 on Debian Jessie with Firefox...

All on the same connection - but each client will have it's own characteristics...

 

[kvic]

I may have written with lack of clarity. So just to clarify.

About HW crypto in OpenSSL

I know ER-X platform does not support it in OpenVPN/OpenSSL. Knew it long ago..I'm interested in proving a point: the lack of support is due to MediaTek SDK not providing a patched OpenSSL or ubnt not using it. I don't have access to the SDK. People may look at ubnt's GPL tarball for traces which I haven't myself. I'm afraid of the GPL tarball being too indulging and killing more of my time.

About CTF and extra lag

It's CTF. Broadcom's Cut Through Forwarding. I should have made it explicit about my connection. Sorry for the confusion. Here it's: WAN <> ER-X <> RT-AC56U (AP) <> PC. So all dslreports tests went through RT-AC56U.

It was when I plugged my PC to ER-X directly. Ran one test. Boom. I knew something not right in RT-AC56U. Turned out to be CTF.

 

[kvic]

Here are my final settings for fq_codel on ER-X, and tuned tcp/ip parameters on RT-AC56U. They seem to work 'great'.

fq_codel on ER-X:

traffic-control {
    smart-queue smart-qos {
        upload {
            ecn enable
            flows 1024
            fq-quantum 1514
            htb-quantum 20000
            limit 1514
            rate 100mbit
        }
        download {
            ecn enable
            flows 1024
            fq-quantum 1514
            htb-quantum 20000
            limit 1514
            rate 100mbit
        }
        wan-interface eth0
    }
}

tcp/ip parameters tuned on RT-AC56U:

echo 2 > /proc/sys/net/ipv4/tcp_ecn
echo -gro 2 > /proc/net/vlan/vlan1
echo 50000    > /proc/sys/net/ipv4/tcp_max_tw_buckets
echo 1048576  > /proc/sys/net/core/wmem_max
echo 1048576  > /proc/sys/net/core/rmem_max
echo 1000     > /proc/sys/net/core/netdev_max_backlog
echo 4096 87380 2038400 > /proc/sys/net/ipv4/tcp_rmem
echo 4096 87380 2038400 > /proc/sys/net/ipv4/tcp_wmem
echo 1024     > /proc/sys/net/ipv4/tcp_max_syn_backlog

I don't understand all of them. But seems to work well. Some said they shall be good for any connection with a Bandwidth Delay Product less than 4MB.

Critics welcome!

 

[Nullity]

You likely need to lower the configured 100Mbit bitrate since it should be set to your real-world lowest maximum bitrate. For functional QoS, the device must be the bitrate bottleneck.

You may consider completely disabling download QoS, especially if your interests are primarily with improving latency (vs sharing bandwidth). With my (not-so-great) ISP the increase in latency during a fully saturating download is only ~30ms, which is barely an increase above my ~10ms during idle.
I can get marginal increases in latency (~15ms) if I rate-limit the download but that also means I decrease my maximum download speed. Ultimately, I choose max speed over a few millisecond latency improvement.


Regarding your TCP tweaks on AsusWRT, I think they only useful when AsusWRT is acting as a network end-point and not when it is acting as an intermediary, so they may be rather unneeded.

 

[kvic]

Nullity, all points good!

About disabling download, you make a good point. And I heard another one specific for EdgeOS. So eventually I'm going to disable it.

About tcp/ip parameters on RT-AC56U, its effect puzzles me indeed. Suppose PC to ER-X through RT-AC56U happens at layer 2. I can't understand why CTF and tcp/ip have any effect. Since I also run OpenVPN server and other daemons on RT-AC56U, I leave it as is. One note: I have both IPv4 & IPv6 forwarding enabled on RT-AC56U. Not sure if this complicates things.

I keep the rates at 100 'cos EdgeOS internally reduces to 95%.

Any thoughts on fq_codel quantum and limit as well as HTB quantum?

The consensus recommendation for HTB quantum seems to be 8000 or greater for a WAN > 100 Mbit/s. No more precise than that. EdgeOS default 200,000 is outrageously high and I think that's a bug. I reduce it to 20,000.

On fq_codel quantum, 1000 for >100 Mbit/s WAN. Limit keeps to be same as quantum. That's only recommendation I came across. EdgeOS defaults quantum to 1514 and limit to 10240 (x10 Flows). Not quite sure the reasoning..

 

[kvic]

The problem with mediatek and other chinese companies is the documentation. They produce chips that are alright but the documentation tends to be where the issue is. For example the raspberry pi compatibles tend to have problems with documentations, licenses, support.

Maybe true 10 or 15 years ago. Quite a different story on today's tier one companies. There was a wake up call for myself a few years ago..

On the SoC, believe it or not, I have the impression MT7621AT is more advanced than BCM4709. Not HW crypto which BCM is lacking. Not HW QoS which BCM also lacks. Not HW NAT which BCM again lacks (it has so called CTF which is a software patch). But the design of the "switch" block. It's actually better not called a switch. But I don't have a better name for it or the paradigm of such a "switch" design.

The fun thing to know would be how the chips compare (power use idle vs load, voltages, clocks like how people compare chips for overclocking with x86).

It's a little device. Power consumption and heat dissipation are both negligible. Borrowing WikiDev's photo (SoC under heatsink):

I think it would be very interesting once you have test bench ready and can benchmark this little guy in future. *grin*

 

[kvic]

Keep in mind that Mediatek bought Ralink - so the docs for low level stuff might be there when searching around - also keep in mind that CTF as a concept is very broad...

No luck there. I'm particularly interested in the design of the "switch" block in MT7621AT. Asus RT-N56U uses a lessor version of MT7621 but seems little fanfare there due to people focus more on wireless performance (?)

CTF is an interesting from a traffic as opposed to Store and Forward with regards to routing - CTF is a "short cut", e.g. it makes some assumptions mathematically compared to Store/Forward (which is pretty much absolute)

I took back my praise to CTF made half year more ago. Please let CTF die. It's not interesting and it's buggy. It's a huge table in kernel space at best, about 4MB!

CTF - it can be a good thing, but it does depend on the system engineering on the back end - and various vendors do their thing, and many times this ends up being "secret sauce", to whit the closed source ctf.ko that we see with Broadcom.

What you said most likely not true. And that's also most likely not what's happening at various vendors. CTF is in Broadcom's full control, and is provided through its SDK as a binary kernel module. I don't see how vendors could enhance in anyway or break it in stupid ways. Like ASUS, for example, their engineers do not dare to simply remove a debug line from the kernel. I would assert CTF and kernel are used as-is by most if not all vendors as a whole from Broadcom's SDK.

CTF works sometimes as Broadcom advertised. On other occasions it brings lots of compatibility, stability and performance issues. Broadcom is the sole one to blame. Personally I think marketing CTF as hardware NAT is a misrepresentation (CTF does not intend to speed up only NAT'ed traffic) and a scam on consumers (CTF is not performed by dedicated & special purpose HW).

 

[Nullity]

I keep the rates at 100 'cos EdgeOS internally reduces to 95%.

I do not like that at all...
A prosumer OS like EdgeOS should not be doing short-cuts like that. If I type in "100Mbit", that's exactly what I expect to be passed to HFSC, HTB, CoDel, etc.
Anyway...

Any thoughts on fq_codel quantum and limit as well as HTB quantum?

The consensus recommendation for HTB quantum seems to be 8000 or greater for a WAN > 100 Mbit/s. No more precise than that. EdgeOS default 200,000 is outrageously high and I think that's a bug. I reduce it to 20,000.

On fq_codel quantum, 1000 for >100 Mbit/s WAN. Limit keeps to be same as quantum. That's only recommendation I came across. EdgeOS defaults quantum to 1514 and limit to 10240 (x10 Flows). Not quite sure the reasoning..

https://tools.ietf.org/html/draft-ietf-aqm-fq-codel-06#section-5.2.4

I would leave it default unless you have a reason to do otherwise. The whole premise of CoDel is to be a "no knobs" setup that does not require tweaking.

I think I played with lowering the quantum when I lowered the MTU of my connection (to improve worst-case delay on my ~768Kbit upload) but I am unaware of any other reason to change it.



IIRC, the most important value is the interval followed by the target, but even when I discovered that the pfSense implementation of CoDel had mistakenly flipped the "target" and "interval" values, it made no measurable difference... CoDel still worked as expected. *shrug*


Regardless of tweaks, CoDel has always worked optimally for me, so I just leave everything default usually. The bufferbloat mailing list encourages people to try random values for the parameters, IIRC, simply to test unknowns.

 

[kvic]

If you have your own server certificate, can use it for the web interface. If you don't, pixelserv-tls can help to generate one quickly. You can of course also use EasyRSA or fall back to openssl CLI.

Assume you already have the server certificate. Here is how to install on ER-X:

Prepare the server certificate in a file named, server.pem which shall contain both the public and private key in that order but not strictly required.

Step 1: Save server.pem to /config/auth/ on ER-X.

Step 2: Go to /etc/lighttpd.

sudo rm server.pem
sudo ln -s /config/auth/server.pem server.pem

Step 3: Bounce lighttpd to take effect.

sudo killall -SIGINT lighttpd
sudo /usr/sbin/lighttpd -f /etc/lighttpd/lighttpd.conf


Benefits of using a server certificate, and a bit more detail to resources: http://kazoo.ga/erx-use-tls-certificate-for-gui/

 

[sfx2000]

What you said most likely not true. And that's also most likely not what's happening at various vendors. CTF is in Broadcom's full control, and is provided through its SDK as a binary kernel module. I don't see how vendors could enhance in anyway or break it in stupid ways. Like ASUS, for example, their engineers do not dare to simply remove a debug line from the kernel. I would assert CTF and kernel are used as-is by most if not all vendors as a whole from Broadcom's SDK.

CTF works sometimes as Broadcom advertised. On other occasions it brings lots of compatibility, stability and performance issues. Broadcom is the sole one to blame. Personally I think marketing CTF as hardware NAT is a misrepresentation (CTF does not intend to speed up only NAT'ed traffic) and a scam on consumers (CTF is not performed by dedicated & special purpose HW).

I'm not a big fan of how CTF has been implemented on the Broadcom HDK/SDK - and that it is implemented on the SW side as a blob without much documentation...

That all being said - cut through forwarding can be efficient... many of the big-iron routers use a variation of this in certain predictable paths in conjunction with the more traditional store/forward implementation.

 

[kvic]

Did some quick tests on forwarding performance on ER-X. Indeed as advertised by some online discussions. It's a little beast!

Switching: 937Mbit/s
Routing: 924Mbit/s with IP offload on. No QoS. No firewall. No NAT.


For more details on the tests: http://kazoo.ga/er-x-forwarding-speed/

 

[sfx2000]

Maybe true 10 or 15 years ago. Quite a different story on today's tier one companies. There was a wake up call for myself a few years ago..

On the SoC, believe it or not, I have the impression MT7621AT is more advanced than BCM4709. Not HW crypto which BCM is lacking. Not HW QoS which BCM also lacks. Not HW NAT which BCM again lacks (it has so called CTF which is a software patch). But the design of the "switch" block. It's actually better not called a switch. But I don't have a better name for it or the paradigm of such a "switch" design.

It's a nice little design - the MediaTek/Ralink chip isn't exceptional, but it is capable - not better or worse than the BCM equivalent - it's an older design, and the newer ARMv7 chips do have more horsepower and better internal bandwidth (then you have Marvell, with both the Armada XP and Armada 38* series, used in conjunction with the LinkStreet SOHO - those are beasts...)

The real advantage of the ER-X, and I'm sure one would agreed is the software - EdgeOS is a fork of VyOS, and it is a real network based OS, and extremely scalable, flexible, and functionality that is hard to beat...

(in my designs back in the past life - we looked at several solutions, Ralink was one of them, we ended up going with Marvell as they were already inside our Vendor ecosystem, whereas Ralink/Realtek and the link were not - Broadcom, while in our ecosystem, did not have a part that met our needs at the time, whereas Marvell did, so we went that route)