Voxel Custom firmware build for Orbi RBK50/RBK53 (RBR50, RBS50) v. 9.2.5.2.8SF-HW

[Skippy Bosco]

Any one noticed lately that there are Wifi drops?

I am not sure if its my devices, the Netgear hardware/antenna, or the firmware. Maybe just coincidence. I am on latest Voxel firmware.

Do you have Hue Lighting, LED lighting, Sonos speakers, etc?

 

[GrundleJuice]

After a couple days of casual seraching/reading/etc and talking with people that have network infrastructure backgrounds, I think I have a very basic idea of what I would like to do with my upcoming home network build. I should preface this with the fact that I am very much a novice with IT overall, and have essentially zero skills for coding of any kind. However, I see that this firmware maintains the GUI for the most part, and that interests me. I liked the idea in Skynet of blocking in/out traffic by IP/country and also having a whitelist capability to allow as necessary. I would use that, along with the log to see what/where traffic is going. I'm not some paranoid, tinfoil hat kind of person (I dont think!), just don't feel it necessary to have to compromise data privacy with technological convenience if I can avoid that.

So, my ultimate questions are:
Can I run Skynet script on this firmware and would it require anything more that using the router GUI?

If ssh is required, where is a good place to learn what I need to know to understand how it works and do what is needed to use Skynet (via USB i assume)? I found a couple things on how to use ssh, but they were written for someone with a general knowledge of the software, which I do not have.

If not ssh, how would I go about using Skynet on the RBR50? Do I need to also run it on the satellite units?

Should I not mess with this, I am likely going to get in over my head?

I have done some homework on how to unbrick if I softbrick using the Windows TFTP client, so hopefully I can restore to factory FW if I need to. I also have a backup router for worst case.


Lastly, Thank you much to Voxel for the time you have put in to make this FW, offer it publicly and maybe most of all, support it by being actively involved on here! That is hugely generous IMO. Also, thank you to anyone and everyone who have helped along the way and who may have info/advice for me on this. Happy 2021 everyone.

 

[LowDelay]

Here you go...

I don't think the ifb 'devices' are actually being used.
The built in switch appears to have QOS support too, maybe I need to disable that 1st.

...

Thanks, so it does not look like a shaper is there, but this command will nuke anything on eth1 and leave a simple fq_codel running.

tc qdisc replace dev eth1 root fq_codel

Switch QoS enabled usually just means the switch will respect DSCP, so a VoIP EF packet will have precedence over CS0, in the switch.

 

[gssjshark]

Do you have Hue Lighting, LED lighting, Sonos speakers, etc?

Just curious, What issues could arise with Sonos speakers?

 

[digital10]

Do you have Hue Lighting, LED lighting, Sonos speakers, etc?

no to all

 

[Skippy Bosco]

Just curious, What issues could arise with Sonos speakers?

Channel overlap and wifi saturation.

 

[Loof]

This is definitely a TCP issue, not an ethernet issue. I tested using UDP rather than TCP and get a very consistent 1.05 Mbits/sec... which is pretty much perfect.

My current setup has my windows PC hooked directly to the router, my Linux server hooked via a 1G switch that is connected to the router.
So, since I'm getting good performance to the Linux server... and 1/2 of the performance to the Orbi... The switch is clearly fine.

I've tried everything I can think of... adjusting the various buffers... different qdisc schedulers & settings... disablign nagle.
I even tried old school 'pfifo', but I don't see any drops or other issues presented there.

Thanks, so it does not look like a shaper is there, but this command will nuke anything on eth1 and leave a simple fq_codel running.

tc qdisc replace dev eth1 root fq_codel

Switch QoS enabled usually just means the switch will respect DSCP, so a VoIP EF packet will have precedence over CS0, in the switch.

 

[Loof]

I'm still struggling with this issue, I'm just not certain where on the network stack to look...

• It affects TCP only, not UDP
• It affects both IPv4 and IPv6
• If affects both Linux & Windows clients
• It definitely affects Ethernet connected clients, but appears to affect WiFi as well (that is more variable, so it's unclear)
• Download speed tested at the router (via the UI, ooklah exec) 836.85Mbps, download speed tested (ooklah web) from a client 199.3Mbps

Is there some setting that could limit throughput to 200Mbps by default?

 

[Voxel]

Is there some setting that could limit throughput to 200Mbps by default?

No. Your case is something specific. I'd suggest to (temporary) flash the stock, then reset and back to my version. Maybe check your cables: it happens sometime...

Voxel.

 

[Loof]

Thank you, I appreciate you chiming in...

I had just done a stock, reset and reflash before reporting this issue.
It didn't appear to happen with the stock firmware, but I rely on some of the expanded capabilities that your release offers.
I've checked the cables and it also works at full speed through the cables + switch (Windows -> Orbi (switch) -> NG Switch -> Linux = 900+Mbps)
Also, UDP is working at full speed (this is based on iperf3 testing in TCP vs UDP) which wouldn't be the case if it were a hardware or firmware issue.

Does anyone have any other ideas for what I can look for in the stack?
I seem to consistently be getting 200Mbps from various devices and media... So, it feels like there's some sort of cap/throttling going on.

Max

 

[Loof]

For some reason the 2.7.x releases have much better network performance than the 2.5.2.x releases.

A bit more information on the issue...
I could not get the voxel release to budge beyond 400Mps to the router via iperf and ~200Mbps (out of 1g!) on a speed test over the Internet.
I went through every guide and scanned through source code... nothing made the performance budge.
Class based QOS, classless, pfifo... huge memory buffers... longer queues... shorter queues...
I had netdata spitting output on every statistic it has... no major issues...

I decided today to take a step backwards... I first installed 2.5.2.4. Speed test showed no major difference, so no regression between Voxel and Stock in the 2.5.2.x line.
However, I'd had perfectly fine performance when I reset to factory... but I'd done that on 2.7.x.
I flashed 2.7.2.102... and my Internet speed is back up... immediately returning to 600Mps (which is not unusual after the Comcast modem has been running for a week, I bet once I reset that it will be up to ~800).

I did not reset the configuration... So, there's something actually different in 2.7.x.

I just determined this, so I'll be seeing if I can spot something... maybe diff the trees if I have time.

Updated...
There are definitely some configuration differences (at least, based on the defconfig-orbi-desktop settings) that could affect networking.
However, I didn't notice anything big in terms of changes.

I used telnet and pulled in iperf3 onto the stock rom, and the tcp speeds are abysmal! I'm seeing barely 200Mbps!
Perversely... the Internet speeds are much greater! Several different speed tests consistently get ~600Mbps+

I'm now even more confused

Max

 

[digital10]

I would wait for Voxel to comment on any updates so far the US has 2.7 and the rest of the world 2.6 not sure why the US is different. Release notes says nothing more than "Fixes security issues" so I don't expect to see improvements on any performance.

 

[Loof]

I had asked Voxel that previously... He indicated that it is something related to Parental Control due to license limitations.
Since I don't use parental controls... Perhaps I should test 2.6.x and see if it works the same way.

The main difference is that a number of different kernel modules are enabled in the newer release...
I really don't understand what's going on with the networking on these routers... I've never had a reason to dig very deeply into Linux's stack... Just a few common tweaks... and only expand queues/backlogs/duty cycle/memory if something runs out.
It's almost always worked very well for my needs.

Max

 

[Voxel]

I would wait for Voxel to comment on any updates so far the US has 2.7 and the rest of the world 2.6 not sure why the US is different.

Look (screenshot from my browser, today is 14th Jan 2021):

Latest GPL codes from NG/DNI available for International version is 2.5.2.4. They (codes) are included into my build. Now, regarding 2.7.xx:

US Region Only. I am not from US. I myself cannot use this 2.7.xx version. So what? Because of this I prefer to release the version which could be used in all regions including mine. Not US specific. Not Chinese specific such as 2.6.1.52 (see the first screenshot). I am not from China too.

I cannot say concretely because of lacking GPL source of 2.6.1.40 but most of the differences are in ARMOR and Disney Circle. I guess it related to license or so. Region specific. No differences in e.g. QCA firmware (Wi-Fi drivers and NAT acceleration). But here are differences in Circle firmware and in ARMOR packages. They are in binary form. So, repeating: I think it is restriction of license. Depends on country and region to use.

I do not think that 2.7.x is better vs 2.5.x And I do not think that problems reported by @Loof are 2.5.x specific and 2.7.x is the remedy. I do not see reports from others re: 2.5.x or my build speed degradation. Something related to his environment, RBR<->modem or so...

P.S.
In general, lack of GPL sources for the latest version of the stock firmware (2.6.1.40) from NG is violation of GPL/GNU license... But it is not my business to push NG to keep license. I am not lawyer. They do not touch me, I do not touch them.

Voxel.

 

[Loof]

Voxel,
I completely understand where you're coming from... However, I the difference in throughput via the router is consistent and testable.
I did a factory reset with the 2.5.x stock... so it was a very clean installation of your release.

I have no clue what is causing it... yet... and don't suggest that you need to build from 2.7.x to get the benefit nor that there's anything wrong with your release.
The changes you've made a solid improvement and should function much faster for all the right reasons.
I don't currently have a cross build environment set up... I'm debating if it will be better/quicker to set up a build so I can experiment with these options to see what makes a difference...
Or just create a VM to act as a router and put the Orbi in AP only mode

I did -not- see any major difference that should make any difference... So, I agree that there should not be a major difference, but there is.
I started by looking at the sysctl network settings between 2.7.x and 2.5.x voxel... There was nothing there that seemed particularly different.

I took some time over my lunch break to do a cursory comparison between 2.7, 2.6 (china) and your release.
I did not have a chance to see what some of these options actually do, but perhaps they can be the start of a viable discussion.

I don't see how they affect anything, but probably worth porting:
* Some new crypto kmods are enabled: aes, gf128, xts
Doubtful, nothing I'm doing should really use encryption
* The 2.6 & 2.7 releases add a new set of offsets for HW_STATUS, etc.
These only get used in the artmtd config and appear to only be informational
* The following network kmods are enabled: ipt-nat6, nf-nat6, gre6, ip6-tunnel, iptunnel6
This shouldn't make a difference, since most of my tests have been IPv4 based
* nls-utf8 is enabled
I did notice some locale errors in the current release (with bash & tmux) but didn't bother to investigate...
* There are a number of minor changes in qsdk-ipq4019-linux.git (which is annoying, since they didn't change the version/patchlevel
Mostly enabling IPV6 NF, some bitmap changes on FS and a few optimizations (i.e. adding const)

Not relevant, but curious
* Only the china version 2.6.x has any of the hardening options enabled.
* Only the china version specifically enables quagga, but I think it is on by default
* For some odd reason the NG configs enable alsa... but there's no audio that I know of

What I'm looking for here is something I can do to nail down what changes between my 'working' and 'slow' builds. How I can isolate the root cause of this performance issue.

Max

 

[Voxel]

Well, Max, OK. I try to reproduce your environment and your problem. I have now some time to play with. I do not promise but let us hope.

I need some clarifications:

(1) You wrote that:

I'm having some throughput issues with the latest couple of Voxel releases.

Does it mean that there were my releases working fine for you and your environment? And only latest couple of Voxel releases are problematic. I.e. after my integration of 2.5.2.4? Or?

(2) You wrote that:

I first installed 2.5.2.4. Speed test showed no major difference, so no regression between Voxel and Stock in the 2.5.2.x line.

So does it mean that you have the same problem (speed) with this stock 2.5.2.4?

(3) You tried to use stock 2.7.x and it is good for you (2.7.x==US Region only). Does it mean that your ORBI is locked to NA?

artmtd -r region

Not WW (WorldWide)?

Voxel.

 

[Loof]

In terms of going back... I had a 250Mps service and everything was 'fine' for quite a while. I upgraded to 1Gbps service and, initially, with an older voxel release. Everything looked ok, but soon the speed degraded. The issue was with the service, because I tested it directly at the cable modem... I even let them send me their integrated gateway so they would stop blaming my equipment.

They fixed the issue, but I kept seeing ~200Mbps speeds on my side... and then realized they really HAD fixed it and it was now my router that was limiting the speed.
I did a factory reset and installed the latest (this was a couple of months ago, so an earlier 2.7.x build) which cleared up the problem... I saw 800-900Mbps speeds through the Orbi.

I left it that way for a bit, just to make sure everything was stable. After a couple of gotcha, I rolled back to a compatible release and installed your release... to find that the speeds were back to 200Mbps.
However, this time it was definitely the router... Connecting directly to the modem gave me solid speeds. I've been trying to troubleshoot that and switched back to the stock 2.5.x release...
Which also gives me ~200Mbps. Then I upgraded from 2.5.x stock to 2.7.x stock and the speed was in the ~600Mbps range. I have not had a chance to bypass the router and see if I'm still speed limited or if the service has degraded slightly. In either case, 600Mbps should be plenty for what we're doing.

My router reports the region is NA, so it's likely to be locked. I purchased it in the USA at Costco as a bundle with 2 satellites.

 

[Skippy Bosco]

For what it is worth, I c

Which also gives me ~200Mbps. Then I upgraded from 2.5.x stock to 2.7.x stock and the speed was in the ~600Mbps range. I have not had a chance to bypass the router and see if I'm still speed limited or if the service has degraded slightly. In either case, 600Mbps should be plenty for what we're doing.

What is the real world problem you are trying to solve? Are you seeing buffering on your clients when streaming? Are you seeing slowdown or stalls on file transfers?

I suspect you may be chasing a ghost in terms of that metric.

What do you see on a wired client connection when doing a speed test?

What do you see when two clients with wired connections do a speed test?

I'm on an RBR50 with 2 RBS50 running V9.2.5.2.8SF-HW with default settings, Armor disabled, Disney circle disabled, parental controls and device access disabled. I see full bandwidth on the router and clients.

That said, you've indicated that you checked your cables, but have you actually replaced them with a cat 5+ cable between your modem and your router just to rule it out?

Also, have you disconnected any other devices that are wired to the modem, router or satellite to rule them out?

 

[Loof]

That's fair...

So, I have 4 kids in virtual schooling and I work from home. We usually have at least 2-3 devices streaming video and/or music at any one time as well.
When there's a severe network issue, the zoom calls (rarely the vpn) usually drop out.
This is a mix of directly wired, fully wireless and 'wired' to the sats.

When it's less severe (200Mbps is about this point), I can notice it in latency on my ssh connections through my vpn. And, of course, in a slight lag on web page hits.
Additionally, if one of the kids finishes work early and is trying to play a game (i.e. Fornite) you can notice spikes of lag (and complaining children)

Part of the reason I have a 1Gbps connection is that it is asymmetric and anything lower than that won't give me sufficient upload to be functional with all of that traffic.

I've verified the speed by using netdata with everything going and multiple computers doing speed tests or similar, it tops out at close to 200Mpbs.

The majority of the time my testing is on a wired connection, once in a while I'll test wireless just to see if there's an issue with the link.
I've tried 3 different network cables to my computer, going through the switch and not. I briefly tested jumbo packets, but I can't run with jumbo across my network because some of my devices flake out.
I've tried 3 other different network cables and 2 different cable modems... So, all hardware has been ruled out (except the router, but I doubt it is a hardware problem if switching versions changes the behavior).

At 600Mps things are smooth, though I do sometimes see latency spikes when I'm monitoring carefully (or when we're all playing an online game together).
I have not bothered to determine (yet) if that latency comes from my provider or is caused by a bottleneck somewhere in my network.

Part of the problem I have is that, while I can monitor for issues at any time... If I don't wait for -everyone- to be asleep, I have someone yelling at me that the Internet is down.
I intend to start working my way backwards through voxel releases when I have the chance, perhaps I can reproduce the issue between 2 voxel releases which would be ideal.
For example, tonight, my wife is relaxing in the tub and watching her favorite shows... It is already 11p here and I'll likely go to bed before she does...
I'll hope for trying it out tomorrow... It's very rare that I have any chance to affect the router over the weekend, so it's most likely Monday evening before I'll have more data.

Beyond that, I'll likely work on finding my own solution at some point... It just takes time and energy I do not have right at the moment.

What is your total bandwidth?

For what it is worth, I c


What is the real world problem you are trying to solve? Are you seeing buffering on your clients when streaming? Are you seeing slowdown or stalls on file transfers?

I suspect you may be chasing a ghost in terms of that metric.

What do you see on a wired client connection when doing a speed test?

What do you see when two clients with wired connections do a speed test?

I'm on an RBR50 with 2 RBS50 running V9.2.5.2.8SF-HW with default settings, Armor disabled, Disney circle disabled, parental controls and device access disabled. I see full bandwidth on the router and clients.

That said, you've indicated that you checked your cables, but have you actually replaced them with a cat 5+ cable between your modem and your router just to rule it out?

Also, have you disconnected any other devices that are wired to the modem, router or satellite to rule them out?

 

[Voxel]

I intend to start working my way backwards through voxel releases when I have the chance, perhaps I can reproduce the issue between 2 voxel releases which would be ideal.

It would be good if you do that...

It is difficult to reproduce your problem, my connection provided by my ISP is different and slower.

So let us try to talk in the terms of stock firmware. 2.5.2.4 stock is bad for you (as far as I understand). 2.7.x stock: solid speed. Main changes 2.5.4.x vs 2.7.x are in Circle and Armor. Do you use these add-ons and did not you try to disable them when using my build? Maybe one of them or both are the reason? I have them disabled. @Skippy Bosco too.

Also (to say true I do not think it could help) but did not you try to change the congestion control algorithm? My version is using yeah by default. Stock: cubic as far as I remember.

And did not you try to test by dslreports speed test. I mean: bufferbloat(?).

Voxel.