Real life utility of SoC speed and RAM size in routers

[cookies]

I've always been thinking that a more recent wireless router is a good thing by itself, because of better hardware. But obviously, lots of people seem to get by fine running very old routers with 40+ clients connected on various protocols, generations, and bands, tons of network activity both high speeds and with lots of connections. I am talking in a consumer context here, home routers, but in the possibly most demanding home use cases that can be imagined.

So I guess my question is, what is the practical extent of an SoC with more cores and higher frequency, and larger amounts of RAM clocked at higher speeds? Did everything get 'good enough' at some point (2016, 2020? 2024)? Can some types of common network activities overload a relatively modern router? 50+ wireless clients? 1000 torrents? Several PC's syncing backups of thousands of small files at the same time? Would latency suffer? Reading the CPU and RAM utilization graphs in my Asus routers' interface never really seemed to give me any insights as they always seemed low no matter what I was doing.

 

[tgl]

Here's the thing: in most routers of this class, performance-critical stuff happens in custom chips and the data never passes through the CPU at all. So traditional specs like CPU clock rate and RAM size don't matter a bit, at least not for any case that the hardware acceleration logic can handle. That hardware is all proprietary and the manufacturers don't want to talk about it, even if there were any standardized performance metrics for it, which there aren't. So you're really flying pretty blind if you're hoping that published specs will tell you much.

I have seen some router makers quote throughput on quasi-standard test cases like iperf3, but they're invariably talking about wired performance. Wireless performance has so many variables besides the router itself that it'd be hard to quote reproducible numbers anyway.

 

[RMerlin]

In general, the main CPU in the SoC will have little to no impact on LAN/wifi network performance. The heavy lifting is handled by the wifi chip itself. Higher end wireless chips even have their own dedicated CPU most of the time, to better handle things like encryption or MIMO.

The main CPU will mostly impact WAN performance (as it will handle NAT), or other services you run on the router (like VPN, SMB sharing, QoS, traffic analysis). NAT is generally offloaded to a separate network packet processor, but it will still require some amount of CPU usage - and that amount will explode if you run any feature that prevents the use of the packet accelerator, requiring the CPU to do the full heavy lifting of processing these packets.

The RAM mostly affect these running services as well, as very little memory is needed per connected client. That's why you can have a router working just fine with 128 MB of RAM, with higher end routers having up to 2 GB. Having hundreds of simultaneous NAT entries (for instance from running a heavy torrent) may require more RAM, but I doubt any of the modern routers currently on the market would run out of RAM in these scenarios.

 

[cookies]

Thanks, that was educational! I guess the wifi/custom hardware also get better with time, but if I understand it correctly that is independent of the main CPU and we can hardly know the secret sauce without benchmarking, or even just observing it in very complex and hard to reproduce workloads. This is much harder to keep updated on spec wise than PC hardware then

 

[coxhaus]

When it comes to routing clock rate is the most important. Once you add applications then it becomes a balancing act to running applications and routing. Routing is just moving data in and out so, a fast clock is the most important, as you move data quicker at a higher clock rate. I would like at least 3Ghz for clock speed. It seems snapper to me.

Applications like more cores and cache. 64 bit works better than 32 bit. And of course you need enough RAM for applications.

 

[tiddlywink]

..

 

[sfx2000]

When it comes to routing clock rate is the most important. Once you add applications then it becomes a balancing act to running applications and routing. Routing is just moving data in and out so, a fast clock is the most important, as you move data quicker at a higher clock rate. I would like at least 3Ghz for clock speed. It seems snapper to me.

Applications like more cores and cache. 64 bit works better than 32 bit. And of course you need enough RAM for applications.

In SW based routing distributions like pfSense, clock rates are everything, esp if doing NAT - and it doesn't matter what chip/core type is used, as it's not very process intense...

In Linux software only - yes, clocks and cores matter as well, but not as much as it does with BSD, and much of this is architecture of the network stack - don't get me wrong, as BSD's pf is great, but Netfilter under Linux has been under very heavy development, and there's been a lot of optimization there - there's also more support for HW acceleration, and we have the Fast Path in SW for NAT that BSD doesn't have.

When looking at Vendor BSP's from Qualcomm, Broadcom, Mediatek - these are all heavily optimized for the platform SoC's, and include closed source drivers for everything from interfaces (wired and wireless) to crypto to flow offloading (NSS or Runner).

 

[sfx2000]

In general, the main CPU in the SoC will have little to no impact on LAN/wifi network performance. The heavy lifting is handled by the wifi chip itself. Higher end wireless chips even have their own dedicated CPU most of the time, to better handle things like encryption or MIMO.

Absolutely - looking at 11ax/11be chipsets, the Wireless chips themselves has resource availability similar to what we saw with entire router/AP's from 10 years ago...

 

[sfx2000]

BTW - this is always a fun exercise...

Consider a 1Gigabit connection - this is 83.3K packets per second with a packet size of 1500 - which is optimal, but when we look at iMix, many of the packets are much smaller to model things like VoIP and IOT, along with gaming...

Let's assume all of those packets are ethernet frames with TCP/IP...

Every packet needs to be ack'ed, then tossed thru the NAT and iptables rules, perhaps do some treatment based on QoS rules as well...

Some of this works on the internal registers, some of this works in L1/L2/L3 cache, and some of this has to go out to RAM...

Anyways - let's go back to that gigabit connection - 83.3K packets per second, this is roughly 12000 cycles per packet - sounds like a lot, but it really isn't...

a 1.2GHz processor will get best results around 500Mbit/Sec because of everything the SW based routing with NAT has to do. Let's not forget that TCP has to be acked...

On a full 1Gbit symmetric connection - to get best results without any accelerators, one needs about 2.4GHz - it's the NAT, the Packet In, Packet out, look at things for the Firewall, apply QoS, decide if the packet is slow path or fast path, etc...

And it doesn't matter whether it's ARM, X86, MIPS, RISC-V, whatever - it's just down to clocks...

Doesn't matter how big the core is - Cortex-A53, x86 Airmont to name a couple - the pfSense folks tend to migrate towards the big cores with Core-i3/i5, which is ok, as turbo gets them up there, but it's not really needed.

That being said - going back to the Router SoC vendors like Qualcomm, Broadcom, Mediatek - to hit benchmarks, they have more than a few magic tricks they do in HW - think CTF, Runner (Broadcom), NSS (Qualcomm, ex Ubicom), and others - it's like the Fuel Economy Ratings for your car - fixed parameters and there they can focus on specific thing to make the best numbers.

Again - once one gets to a PHY rate, bits/sec isn't an important benchmark any more - it's packets per second that is much more relevant...

@thiggins - this goes back to my other comment earlier today about WiFi testing and the device/traffic mix...

What would be a better solution?

Define what? The Key Performance Indicators - and how to apply test cases that are consistent and repeatable. In the past, you've done great work there, but the Wireless Environment we see today is very different than it was 10 years ago when 802.11ac was in their first gen of devices. The...

www.snbforums.com

Again, things like this could be a fun research project, but I would guess folks like Spirent, Keysight, etc have this kind of figured out...

 

[coxhaus]

If you look at the rating on the CISCO switches it is based on packet size as it makes a difference.

Routing is definitely clock as I said above but once you add SNORT or an IPS/IDS software package clock is not the most important as if you have a weak CPU with high clock, it can be killed in performance using the application. You need a performance CPU at that point with a high clock. At this point I would prefer a 64bit CPU over a 32bit.

 

[sfx2000]

Routing is definitely clock as I said above but once you add SNORT or an IPS/IDS software package clock is not the most important as if you have a weak CPU with high clock, it can be killed in performance using the application. You need a performance CPU at that point with a high clock. At this point I would prefer a 64bit CPU over a 32bit.

SNORT doesn't add any value to a home network, or even a small business network...

Times have changed, and if one needs that level of security, you have to look at the endpoints directly as managed clients.

Same goes with ad-blocking - used to be that things like pfblocker had some level of relevance, with https and ipv6, it's kind of a moot point - need to consider using an end-point solution there...

For the edge - the router needs to keep things as simple as possible - netfilter/nftables, these are fairly efficient, as is BSD and pf - some basic rules is all one really needs for a firewall...

Let's review a basic firewall rule...

Status: active

To Action From
-- ------ ----
22 LIMIT Anywhere
Anywhere ALLOW 192.168.1.0/24
22 (v6) LIMIT Anywhere (v6)

Now let's add a few rules - they don't actually add much, but they're reasonable..

sudo ufw default deny incoming
sudo ufw default allow outgoing

sudo ufw limit ssh

sudo ufw allow from 192.168.1.0/24
sudo ufw allow from fe80::/64
# done for most purposes

# if you want to be more specific on the LAN side - we limit ports to link-local
sudo ufw allow proto tcp to any port 135 from 192.168.1.0/24
sudo ufw allow proto udp to any port 137 from 192.168.1.0/24
sudo ufw allow proto udp to any port 138 from 192.168.1.0/24
sudo ufw allow proto tcp to any port 139 from 192.168.1.0/24
sudo ufw allow proto tcp to any port 445 from 192.168.1.0/24

sudo ufw allow proto tcp to any port 135 from fe80::/64
sudo ufw allow proto udp to any port 137 from fe80::/64
sudo ufw allow proto udp to any port 138 from fe80::/64
sudo ufw allow proto tcp to any port 139 from fe80::/64
sudo ufw allow proto tcp to any port 445 from fe80::/64

sudo ufw allow proto udp to any port 5353 from 192.168.1.0/24
sudo ufw allow proto udp to any port 5353 from fe80::/64

sudo ufw allow proto tcp to any port 80 from 192.168.1.0/24
sudo ufw allow proto tcp to any port 443 from 192.168.1.0/24
sudo ufw allow proto tcp to any port 80 from fe80::/64
sudo ufw allow proto tcp to any port 443 from fe80::/64

Yes, that now means every packet needs to go thru that ruleset... and this one is a fairly simple one...

ufw and pf rules as essentially the same - pf is slightly more efficient, but linux has the upside here with the overall NAT implementation...

Recall, this is just ports, add things like pfblocker, now one has to look at every TCP packet - yeah, that becomes a problem, eh?

I really like to keep my edge as simple as fracking possible, as that is performance, and as I mentioned, gettng a gigabit connection isn't that hard - clocks do matter - it could be a Celeron on X86 even...

 

[coxhaus]

Well, I want to try to watch IOT devices on my LAN. So, I am thinking SNORT as I have run it before. With my Dell Intel i7 gen 8 PC I will have enough horse power to run SNORT. I might think about SNORT as a separate PC but I don't really want to run a second PC. I would run real SNORT without Pfsense.

I am like you in that I want my firewall to run without lag. I do not tolerate lag on my firewall.

 

[TheLostSwede]

In general, the main CPU in the SoC will have little to no impact on LAN/wifi network performance. The heavy lifting is handled by the wifi chip itself. Higher end wireless chips even have their own dedicated CPU most of the time, to better handle things like encryption or MIMO.

This is mostly true for Broadcom based hardware, whereas MTK and QCA have at least traditionally, done the WiFi offloading on the main SoC. This might well have changed with more recent WiFi standards, but Broadcom appears to have gone for the beefiest hardware in the WiFi chips compared to its competitors. It also seem to depend if the SoC has integrated WiFi or not.

The main CPU will mostly impact WAN performance (as it will handle NAT), or other services you run on the router (like VPN, SMB sharing, QoS, traffic analysis). NAT is generally offloaded to a separate network packet processor, but it will still require some amount of CPU usage - and that amount will explode if you run any feature that prevents the use of the packet accelerator, requiring the CPU to do the full heavy lifting of processing these packets.

The RAM mostly affect these running services as well, as very little memory is needed per connected client. That's why you can have a router working just fine with 128 MB of RAM, with higher end routers having up to 2 GB. Having hundreds of simultaneous NAT entries (for instance from running a heavy torrent) may require more RAM, but I doubt any of the modern routers currently on the market would run out of RAM in these scenarios.

Don't forget that the OS is also running in RAM on most routers. This is why often half of the RAM or more isn't available to the user.

 

[TheLostSwede]

Thanks, that was educational! I guess the wifi/custom hardware also get better with time, but if I understand it correctly that is independent of the main CPU and we can hardly know the secret sauce without benchmarking, or even just observing it in very complex and hard to reproduce workloads. This is much harder to keep updated on spec wise than PC hardware then

Keep in mind that WiFi has become a lot more complex with things like MU-MIMO, MLO, more advanced encryption and a bunch of other things that has lead to more complex router hardware as a whole. However, if you look at the development of the CPU in the router, we haven't moved very far in the past decade, especially not if you compare with something like smartphones that are also based on Arm CPU cores.
Most routers are still using two or four Arm Cortex-A53 cores, which was launched in 2012 by Arm, so first products about a year later. Broadcom even has some current router SoCs on the lower-end based on the Cortex-A7, although it's merely a year older, but a much weaker core than the Cortex-A53.

As mentioned above, it's all about the offloading magic that's going on that routers work as well as they do and this is why most of the random xinese SBCs are terrible replacements for router SoCs, since the xinese chips (as well as the RPi) weren't designed to provide the best possible network throughput.

On top of that, the WiFi components in a routers are very different from client WiFi components, least not due to them either having integrated or external PA/LNA's for increased transmission and reception range, something client WiFi chips don't have and why you never get the same range out of them if you configure them as an access point.

More advanced operating systems with more features also require more RAM, as we've gone from super basic operating systems in routers, to fairly complex things that can handle things only much more expensive devices could do a mere decade ago, at least at the speeds we can do it today.

Personally I would like to see a move to the Cortex-A5xx series and a more advanced production node for the SoC, so we can get down the power usage of the router SoC, since it's unlikely that we're going to see more power efficient WiFi. since that's always going to be the most power hungry part in a WiFi router. Power efficiency is kind of getting out of hand with some of the higher-end routers, especially the ones with 10 Gbps ports.

 

[sfx2000]

This is mostly true for Broadcom based hardware, whereas MTK and QCA have at least traditionally, done the WiFi offloading on the main SoC. This might well have changed with more recent WiFi standards, but Broadcom appars to have gone for the beefiest hardware in the WiFi chips compared to its competitors. It also seem to depend if the SoC has integrated WiFi or not.

Broadcom typically is FullMAC - e.g. everything runs on the WiFi chip, so it don't really need to use the mac80211 layer which runs in kernel space - ath11k, and mt76 are SoftMAC drivers, so the workload is split, basically following the OSI layer concept - e.g. things air-side, e.g. radio items along with baseband specifics, including scheduling, config, etc are handled on the WiFi chip, and network facing items are handled by the mac80211 layer...

I'm checking to see what ath12k does (802.11be), but as of now, it's a fork of ath11k, so I suspect it follows the same path...

This pic might actually help here - again nl80211, cfg80211 are common - mac80211 says there is a softMAC in play...

It's always fun to look back at this architecture, as this is Linux, and how do we do MESH kind of things - with Qualcomm's Mesh thing, it's pretty straight forward, and MTK favors OpenMESH - both of these approaches are vendor specific.

Asus with AIMesh is interesting, as they support things across different vendors - some do FullMAC, some do SoftMAC - but yet it works...

Kudos to them...

 

[sfx2000]

Most routers are still using two or four Arm Cortex-A53 cores, which was launched in 2012 by Arm, so first products about a year later. Broadcom even has some current router SoCs on the lower-end based on the Cortex-A7, although it's merely a year older, but a much weaker core than the Cortex-A53.

It's control plane vs data plane...

control plane you don't need so much if it's just a basic GW - Cortex-A7 is fine here, and see see a lot of them - interesting to note that a cortex-A7 core at 28nm is less than 1mm squared - they're tiny...

cortex-A53 is Cortex-A7 on steroids - it's a bit wider, and as a ARMv8A, it can do things that A7 or A9 can't do - it does replace A9 and earlier, but can run ARMv7 code just fine..

Anyways - these cores, along with MIPS for example, are all control plane - most of the magic for routing and switching is a layer down, and much of that is HW for regular Router/AP's

(FWIW - yes, on switches, maanged or not, there is firmware that runs, and it generally runs on an RTOS - for Broadcom solutions, it's generally eCOS, and others it's ether ThreadX or FreeRTOS

What I'm getting at - don't worry about what's under the hood - platform engineers have thought this thru given the constraints of the product - sometimes it's choices that they would not prefer...

 

[sfx2000]

Personally I would like to see a move to the Cortex-A5xx series and a more advanced production node for the SoC, so we can get down the power usage of the router SoC, since it's unlikely that we're going to see more power efficient WiFi. since that's always going to be the most power hungry part in a WiFi router. Power efficiency is kind of getting out of hand with some of the higher-end routers, especially the ones with 10 Gbps ports.

I'm just curious as to when (and who) will do a RISC-V based solution...

Who will take the first step there - Qualcomm has signed on for RISC-V, and there's a shedload of vendors in the PRC that are developing SoC's around the ISA...

IIRC, MediaTek is along for the ride on RISC-V...

 

[TheLostSwede]

What I'm getting at - don't worry about what's under the hood - platform engineers have thought this thru given the constraints of the product - sometimes it's choices that they would not prefer...

It does matter though, at least when you step away from that "basic" router you mention, which most users here do. Once you want to do VPN or storage or anything outside of just routing data, the CPU core(s) really matter.

 

[TheLostSwede]

I'm just curious as to when (and who) will do a RISC-V based solution...

Who will take the first step there - Qualcomm has signed on for RISC-V, and there's a shedload of vendors in the PRC that are developing SoC's around the ISA...

IIRC, MediaTek is along for the ride on RISC-V...

I'd say that RISC-V isn't quite there today, but give it a year or two. I would say Realtek might be the first to try this, as they've always been about cost and Arm cores are costly in terms of licensing.

 

[coxhaus]

I'd say that RISC-V isn't quite there today, but give it a year or two. I would say Realtek might be the first to try this, as they've always been about cost and Arm cores are costly in terms of licensing.

We used RISC in our old network Sun machines way back. Intel finally did them in. It is a way to get high clock, but you pay for it in the reduced instruction set.