Forwarding Performance and Switching Capacity of RT-AC68U

[tgl]

LAN to LAN should not cause any CPU utilization at all.

Well, all I can say is the web GUI showed CPU utilization. I did misspeak above --- I said that it was 20%-of-one-core per iperf3 stream, but it seemed more like it was 20%-of-one-core for any number of streams. Go figure. In any case, I agree that the CPU probably isn't directly involved in shoving bits from port X to port Y. It's doing *something* during the test, though.

I wonder if in AP mode, the WAN port is set to standard 1G for some reason.

As I said, both ends of the connection agreed that the on-the-wire data rate was 2500MHz. The symptoms I saw looked like there was only about 1GHz worth of switching fabric behind the WAN port, which'd be a pretty serious design fail if true. Could be just a bug though, ie firmware running the chips in the wrong mode or something like that.

I'd love to have someone else try to reproduce these results, either with 42095 or some other firmware.

 

[drinkingbird]

Well, all I can say is the web GUI showed CPU utilization. I did misspeak above --- I said that it was 20%-of-one-core per iperf3 stream, but it seemed more like it was 20%-of-one-core for any number of streams. Go figure. In any case, I agree that the CPU probably isn't directly involved in shoving bits from port X to port Y. It's doing *something* during the test, though.




As I said, both ends of the connection agreed that the on-the-wire data rate was 2500MHz. The symptoms I saw looked like there was only about 1GHz worth of switching fabric behind the WAN port, which'd be a pretty serious design fail if true. Could be just a bug though, ie firmware running the chips in the wrong mode or something like that.

I'd love to have someone else try to reproduce these results, either with 42095 or some other firmware.

I edited my post above after thinking about it, my guess is the LAN ports share a 1G backplane connection, which your traffic has to pass through in order to reach the 2.5G WAN port. Its possible that it is a software virtual interface limitation (which could be corrected) but less likely.

Try running a 1G wired IPERF and a wireless IPERF at the same time (to your 2.5G WAN device). I suspect you'll be able to exceed 1G total then. Sounds like your environment won't support >1G on wifi due to channel limitations so that's the next best test.

As far as CPU, who knows, maybe bandwidth monitoring, QOS (even if not active, it is analyzing something) or AIProtection (again, not active but getting mirrored to CPU when it doesn't need to be). The fact that it does not increase with multiple streams tells me it isn't related to amount of traffic.

 

[ColinTaylor]

FYI Unlike the OP's RT-AC68U the XT8 uses a separate single port chip for the WAN socket. The three (not four) LAN sockets are connected to a different switch chip.

 

[tgl]

Try running a 1G wired IPERF and a wireless IPERF at the same time (to your 2.5G WAN device). I suspect you'll be able to exceed 1G total then.

Ah-hah. I'd not thought of it that way, but it appears you're right: the total traffic from WAN to all 3 LAN ports is capped at ~1Gbps, but there's additional bandwidth available between WAN and wireless. I'm not able to drive the wireless side hard enough to see what the saturation point is there.

So my original thought that ASUS might have skimped a bit on the LAN-port provisioning was correct, though not in the way I expected.

 

[drinkingbird]

FYI Unlike the OP's RT-AC68U the XT8 uses a separate single port chip for the WAN socket. The three (not four) LAN sockets are connected to a different switch chip.

Makes sense, the 2.5G ASIC is likely more expensive so having one that supports 3 ports on the LAN would raise the price of the router. Soon enough they'll all be multi-gig capable as it gets more popular and prices drop.

If a single ASIC is supporting all 3 LAN ports it is odd that @tgl is seeing some speed variation between ports but it could be as simple as one port being closer to a noisy IC or something.

 

[drinkingbird]

Ah-hah. I'd not thought of it that way, but it appears you're right: the total traffic from WAN to all 3 LAN ports is capped at ~1Gbps, but there's additional bandwidth available between WAN and wireless. I'm not able to drive the wireless side hard enough to see what the saturation point is there.

So my original thought that ASUS might have skimped a bit on the LAN-port provisioning was correct, though not in the way I expected.

My guess would be it is possible to get 2.5G (or a bit less with overhead) total between LAN/wireless and WAN, assuming you can drive the wireless hard enough which many probably won't be able to anyway.

 

[tgl]

If a single ASIC is supporting all 3 LAN ports it is odd that @tgl is seeing some speed variation between ports but it could be as simple as one port being closer to a noisy IC or something.

I'm confused about that too. My results pretty clearly indicated that LAN2 and LAN3 are somehow more closely associated with each other than they are with LAN1. Maybe the ASIC supporting them is really a four-port chip with one port unconnected? But you wouldn't think a four-port chip would have much asymmetry in it.

 

[RMerlin]

I'm confused about that too. My results pretty clearly indicated that LAN2 and LAN3 are somehow more closely associated with each other than they are with LAN1. Maybe the ASIC supporting them is really a four-port chip with one port unconnected? But you wouldn't think a four-port chip would have much asymmetry in it.

Maybe some ports are from the BCM6755, while others (like the 2.5 Gbps port) are from the BCM53134. I'm not familiar enough with the XT8 architecture.

 

[tgl]

My guess would be it is possible to get 2.5G (or a bit less with overhead) total between LAN/wireless and WAN, assuming you can drive the wireless hard enough which many probably won't be able to anyway.

Yeah. Looking at the spec sheet, the XT8's 5GHz-1 radio is just 2x2, so max PHY rate of 867Mbps in an 80MHz channel. So I've got no chance of saturating its WAN port with one wireless client under local conditions. However, the 5GHz-2 (backhaul) radio is 4x4, so in principle with a clear 160MHz channel you could find yourself wanting 3+GHz worth of WAN capacity for that, plus the 5GHz-1 and 2.4GHz channels would need even more.

It does seem like ASUS made some odd choices here --- why didn't they spring for a 5GHz or 10GHz WAN port? Maybe they know something I don't about what the internal data paths can manage.

 

[drinkingbird]

Yeah. Looking at the spec sheet, the XT8's 5GHz-1 radio is just 2x2, so max PHY rate of 867Mbps in an 80MHz channel. So I've got no chance of saturating its WAN port with one wireless client under local conditions. However, the 5GHz-2 (backhaul) radio is 4x4, so in principle with a clear 160MHz channel you could find yourself wanting 3+GHz worth of WAN capacity for that, plus the 5GHz-1 and 2.4GHz channels would need even more.

It does seem like ASUS made some odd choices here --- why didn't they spring for a 5GHz or 10GHz WAN port? Maybe they know something I don't about what the internal data paths can manage.

I'm sure the answer is cost on all accounts. Right now multigig is pretty new and those chips are going to be more expensive.

 

[drinkingbird]

I'm confused about that too. My results pretty clearly indicated that LAN2 and LAN3 are somehow more closely associated with each other than they are with LAN1. Maybe the ASIC supporting them is really a four-port chip with one port unconnected? But you wouldn't think a four-port chip would have much asymmetry in it.

Or two groups of 2 ports with one not connected (didn't realize it only had 3 LAN ports, most are an even number). It is likely one physical ASIC but internally divided into two dual port or something. ASICs are like CPUs now they'll put several "cores" into one package.

The variation you're seeing isn't really enough to worry about though. Like I said, it could be as simple as a little more noise on one port, its path to ground is slightly longer or goes by something noisy etc.

Even enterprise hardware has limitations like this, however it is clearly listed in the documentation when it does. Some of the original Cisco 10G cards (4 port was fine, but 8 and 16 port had lots of limits) had very specific limitations and guidelines, use this pair of ports if you want full 10G in both directions (20G) between two boxes (since those ports shared an ASIC and didn't have to go through the backplane), or only use 4 out of 8 ports total since the backplane connection was only 40G full duplex, etc. Even when they went to 80G backplane connections the 16 port card was really a 16 port 5G card effectively.

48 port 1G cards when their switches had 40G backplane per card, you basically just didn't use 8 of the ports if you wanted to guarantee line rate on all ports.

Even now that you can get 1Tb and 2Tb backplane connections, some 40 and 100G cards still can't have all ports run at full capacity.

 

[tgl]

The variation you're seeing isn't really enough to worry about though.

No, of course not; you'd never notice it in real-world usage. I'm just curious about what it tells us about the architecture of this router.

 

[drinkingbird]

No, of course not; you'd never notice it in real-world usage. I'm just curious about what it tells us about the architecture of this router.

Tear off the covers and trace out the paths between the chips, look up the pinouts on Broadcom's site, let us know

 

[tgl]

Tear off the covers and trace out the paths between the chips, look up the pinouts on Broadcom's site, let us know

ifixit.com, I'm not.

 

[drinkingbird]

No, of course not; you'd never notice it in real-world usage. I'm just curious about what it tells us about the architecture of this router.

If you really wanted to deep dive, and have 4 NICs (configured to make sure each IPERF stream uses a separate NIC) you could do a LAN to LAN transfer on port 1 and 2, while also doing a port 3 to WAN transfer. Then do the same but with LAN on 2 and 3 and WAN on port 1. Then LAN on 1 and 3 and WAN on 2. If no scenario lets you get an aggregate 3G on the lan ports (ignoring the WAN for now) then it is a limitation of the switching chip, but it is unlikely it can't handle 3 gigs (or 6 gigs if you do full duplex IPERF tests). Even $10 desktop switches have full wire speed between all switch ports. If one or two scenarios net less than 3G (or 6G full duplex) then that would point to what @RMerlin said, they may have 2 LAN ports running off one chip and the 3rd going to through the 1G backplane to another chip. Even when only doing a 1G transfer, that backplane connection adds a tiny bit of latency and processing and will affect speed slightly.