Voxel Custom firmware build for Orbi RBK50/RBK53 (RBR50, RBS50) v. 9.2.5.2.8SF-HW

[Voxel]

Continuation of

https://www.snbforums.com/threads/custom-firmware-build-for-orbi-rbk50-v-2-5-0-42sf-hw.60308/
. . .
https://www.snbforums.com/threads/c...50-v-9-2-5-2-5sf-hw-v-9-2-5-2-5-1sf-hw.66034/
https://www.snbforums.com/threads/c...k50-rbk53-rbr50-rbs50-v-9-2-5-2-6sf-hw.67085/

New version of my custom firmware build: 9.2.5.2.8SF-HW.

Changes (vs 9.2.5.2.7.1SF-HW):

1. Toolchain: Go is upgraded 1.15.5->1.15.6.
2. OpenSSL v. 1.1.1 package is upgraded 1.1.1h->1.1.1i (fixing CVE-2020-1971).
3. curl package is upgraded 7.73.0->7.74.0 (fixing CVE-2020-8284, CVE-2020-8285, CVE-2020-8286).
4. wireguard package is upgraded 1.0.20201112->1.0.20201221.
5. OpenVPN is upgraded 2.4.9->2.5.0.
6. ipset package is upgraded 7.9->7.10.
7. lighttpd package is upgraded 1.4.56->1.4.57.
8. proftpd package is upgraded 1.3.6e->1.3.7a.
9. unbound package (used in stubby) is upgraded 1.12.0->1.13.0.
10. elfutils package is upgraded 0.179->0.180.
11. nano package is upgraded 5.3->5.4.
12. libusb package is upgraded 1.0.23->1.0.24.
13. logrotate package is upgraded 3.16.0->3.17.0.
14. tar package: optimize for a size.
15. Add procps-ng package utilities ('ps', 'top').
(run '/usr/bin/top-procps-ng' or '/usr/bin/ps-procps-ng -aux' from console to check them).
16. OpenVPN server: add 'CHACHA20-POLY1305' cipher to 'ncp-ciphers' option and change the cipher of downloaded config for Windows clients to 'CHACHA20-POLY1305'.
(Important: it is highly recommended to use 'CHACHA20-POLY1305' if your client is based on v. 2.5.x, much faster,
change your non-Windows client config if possible).


The link is:

https://www.voxel-firmware.com (thanks to vladlenas for his help with hosting).

Voxel.

 

[spinkick]

Thank you!

 

[digital10]

thanks Voxel. I see a lot of upgrades, as an end user will this improve the router performance anyhow?

 

[Voxel]

thanks Voxel. I see a lot of upgrades, as an end user will this improve the router performance anyhow?

General performance was improved in 9.2.5.2.6SF-HW:

9.2.5.2.6SF-HW
. . .
3. Toolchain: -finline-functions compiler option is added to GCC.
4. Kernel level acceleration (-O2, -march=armv7-a -> -mcpu=cortex-a7, -finline-functions, etc.).
. . .

This release is mostly important fixing various CVEs (your safety). Plus upgraded packages: bug fixing, stability, reliability.

Additional improvement of performance could be expected only in OpenVPN server if you are using it to access your LAN externally:

9.2.5.2.8SF-HW
. . .
16. OpenVPN server: add 'CHACHA20-POLY1305' cipher to 'ncp-ciphers' option and change the cipher of downloaded config for Windows clients to 'CHACHA20-POLY1305'.
(Important: it is highly recommended to use 'CHACHA20-POLY1305' if your client is based on v. 2.5.x, much faster,
change your non-Windows client config if possible).

Voxel.

 

[Skippy Bosco]

Merry Christmas Voxel. Merry Christmas everyone.

 

[digital10]

This release is mostly important fixing various CVEs (your safety). Plus upgraded packages: bug fixing, stability, reliability.

bug fixes, stability, and reliability is a performance improvement.

Thanks for your work, could not use the toxic stock firmware.

 

[mith_y2k]

Have you looked at AirVPN’s OpenVPN upgrades and optimizations for ARM (specifically on the CHACHA implementation)?

GitHub - AirVPN/openvpn3-airvpn: OpenVPN 3 is a C++ class library that implements the functionality of an OpenVPN client, and is protocol-compatible with the OpenVPN 2.x branch.

OpenVPN 3 is a C++ class library that implements the functionality of an OpenVPN client, and is protocol-compatible with the OpenVPN 2.x branch. - AirVPN/openvpn3-airvpn

github.com

 

[Voxel]

Have you looked at AirVPN’s OpenVPN upgrades and optimizations for ARM (specifically on the CHACHA implementation)?

It has no sense to compare OpenVPN with any kind of its clone. All of such clones are user-space VPN implementations. The only serious real competitor could be kernel-space VPN implementation (e.g. famous WireGuard included into linux kernel).

Bottleneck is the speed of encryption/decryption of VPN channel. AirVPN you pointed vs OpenVPN:

(1) Is using anyway OpenSSL, but not own algorithms of encryption/decryption (the same as OpenVPN). And ARM acceleration of CHACHA20-POLY1305 is provided by OpenSSL. Namely assembler codes for ARM with e.g. NEON (SIMD) parallelization.

(2) OpenVPN 2.5.x already allows to use CHACHA20-POLY1305 cypher. I.e. since this version of my build. I.e. previous versions of OpenVPN most probably were slower vs AirVPN client (because of lacking support of CHACHA/POLY in OpenVPN 2.4.x). Now: at least not slower.

(3) Well, (just IMO) any C++ class library will be slower vs pure C library. Object-oriented SDK is (sometimes) more convenient for developers to use but there is some "overflow" in resulting binary codes because of such an abstraction. So as a rule C++ class library is not faster C library. And C library is not faster than library created with pure assembler.


Voxel.

 

[Voxel]

Bottleneck is the speed of encryption/decryption of VPN channel.

Some benchmarks to show what your could expect from OpenVPN 2.5.x based on OpenSSL if properly assembled (executed on ORBI RBR50). Just OpenSSL test of the performance (encryption/decryption):

Stock version of firmware, AES-256-CBC cypher:

type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-256 cbc       7052.33k     7076.09k     7229.18k     7222.48k     6976.13k

My version of firmware, AES-256-CBC cypher:

type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
aes-256-cbc      10921.20k    12733.17k    13484.26k    13805.51k    13893.30k    13832.93k

My version of firmware, CHACHA20-POLY1305 cypher:

type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
chacha20-poly1305    14532.29k    25564.49k    38060.48k    40880.28k    41798.57k    42345.49k

Voxel.

 

[Loof]

I'm having some throughput issues with the latest couple of Voxel releases.
I have a 1G connection and I was testing with the latest stock Netgear firmware... and getting 600-900Mbps.
I am getting 150-300Mbps with the Voxel release... No clue why yet.

I installed netdata in order to better monitor the system, I've run into a few minor issues that perhaps someone has ideas for:
1) The fping that is available seems broken in netdata (-N) mode as it always returns 0 for the RTT
2) The github repo seems out of date... Is there canonical repo I can contribute PRs to as I find issues?
3) Is there a viable port of iprange for the Orbi release?
a) I think using FireQOS to set up a baseline QOS makes sense, is there a better tool that's already supported?
b) If it is not available, do you have any recommendations on cross compiling, packaging it and providing the package for the entware repo?

I bumped up the net.core.netdev_max_backlog and net.core.netdev_budget, but even a 4x increase seemed to make little difference.
I tried switching the WAN interface (eth0, if I recall) from hyfi_pfifo_fast to fq_codel... no real difference there.
Is there anything I should be looking at to try to find the root cause?

I do notice a spike in softirqs (%40-50) during speed tests... but no clear indication of if that is normal or something that can be tuned.

I'd appreciate any help...

 

[Skippy Bosco]

I have a 1G connection and I was testing with the latest stock Netgear firmware... and getting 600-900Mbps. I am getting 150-300Mbps with the Voxel release... No clue why yet.

How/Where are you speed testing? From the Speed Test in the Orbi Web Admin?

 

[Loof]

How/Where are you speed testing? From the Speed Test in the Orbi Web Admin?

I get similar, though slightly... say 20% faster results than an ethernet connected PC via ookla & fast.com... the xfinity speedtest is a little faster (though I'm not sure that's worth much).
I am most concerned about optimizing the speed to a hardwired PC... but each step is important.

I'm currently using an Xfinity g/w in bridging mode... I have my own modem, but some of my earlier issues were caused by a connectivity issue... So, I'm making sure they don't have any equipment to blame.

 

[Skippy Bosco]

What do you get when you test via the Orbi web admin Speed Test?

Orbi Web Admin

Advanced -> Setup -> Speed Test

 

[Loof]

That's what I was responding to... I get a bit faster results than via a connected PC.

Thanks,
Max

 

[Loof]

One interesting thing... I did the most simplistic network performance test I could think of... LanBench on windows to an ncat echo server.
From Windows -> Linux server I get 930Mbps (not bad for 1G Ethernet).
From Windows -> Orbi I get 160Mbps - Same as I'm seeing on speed tests.

How can I diagnose the root cause of this?

 

[LowDelay]

How can I diagnose the root cause of this?

I think you already found the reason with the following:

I do notice a spike in softirqs (%40-50) during speed tests

That is likely because one of the cores is saturating performing the routing (50% of all SIRQ = one core at 100% CPU)

So, there must be something going on, as most SoC's of this capability can do plain routing at well over 600Mbps, heck, even the MT7621 platform I'm most familiar with routes at 900+Mbps on a good day. But turn on any decent QoS, and boom, less than 200Mbps, and SIRQ is pegged on the processor with the QoS process.

You sure you have QoS off?

 

[Loof]

The WAN & LAN ports are set to a variant of the pfifo_fast queue... I also tried the default (which is prio + 3 fq_codel lanes) and just fq_codel...

I just took a peek and the Orbi has 4 cores... So, 50% = 2 cores at 100%

I have switched to using iperf3 in hopes of more consistent results. I'm using a Linux server, the Orbi and a Windows 10 workstation.
I have a 1G unmanaged switch, this has been tested with similar results both via the switch and directly to the Orbi.
In this situation, the WIndows workstation is attached to the Orbi, the Orbi to the switch and the Linux server to the Switch...

Either Windows or Linux to the Orbi is around 1/2 of the theoretical bandwidth.
However, look at Orbi to Linux...

A few odd things...
Windows -> Linux 948Mbps
Linux -> Windows 949Mbps
Linux -> Orbi 510Mbps
Orbi -> Linux 752Mbps
Windows -> Orbi 511Mbps
Orbi -> Windows 431Mbps

Any ideas where to start looking?

 

[LowDelay]

Orbi -> Linux 752Mbps

Aha, the issue looks like the inbound dev interface (the LAN) must have a rate-limiter on it.

post the output of "tc -s qdisc"

I'm guessing there might be an HTB instance there.

What I'd do is scrub the LAN interface down to just a single fq_codel qdisc on the device node itself. Test performance and build up from there.
IIRC one can do that with a tc replace targeting the root of the device (ethN)

 

[Loof]

Here you go...

I don't think the ifb 'devices' are actually being used.
The built in switch appears to have QOS support too, maybe I need to disable that 1st.

qdisc hyfi_pfifo_fast 0: dev ifb0 root refcnt 2 [Unknown qdisc, optlen=20]
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc hyfi_pfifo_fast 0: dev ifb1 root refcnt 2 [Unknown qdisc, optlen=20]
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
[B]qdisc hyfi_pfifo_fast 8006: dev eth0 root refcnt 5 [Unknown qdisc, optlen=20]
Sent 255365888 bytes 1405511 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0[/B]
qdisc hyfi_pfifo_fast 8007: dev eth1 root refcnt 5 [Unknown qdisc, optlen=20]
Sent 3921082025 bytes 2948091 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc hyfi_pfifo_fast 0: dev teql0 root refcnt 2 [Unknown qdisc, optlen=20]
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc hyfi_pfifo_fast 0: dev pas0 root refcnt 2 [Unknown qdisc, optlen=20]
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc fq_codel 0: dev wifi0 root refcnt 2 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
Sent 220 bytes 2 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
  maxpacket 256 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev wifi1 root refcnt 2 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
Sent 220 bytes 2 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
  maxpacket 256 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev wifi2 root refcnt 2 limit 1024p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn
Sent 220 bytes 2 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
  maxpacket 256 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0

 

[digital10]

Any one noticed lately that there are Wifi drops?

I am not sure if its my devices, the Netgear hardware/antenna, or the firmware. Maybe just coincidence. I am on latest Voxel firmware.