RT-N66U - Less performance than expected on high speed fibre connection

[rotor]

Hello all,

I live in the UK and I’ve recently purchased an RT-N66U to connect to my BT Infinity fibre broadband. This is true fibre, the fibre cable enters my house and connects to a fibre modem provided by BT, and gives me ~300 Mbps down and ~20 Mbps up. My new router connects to the fibre modem via Gigabit Ethernet.

The problem is that the RT-N66U isn’t giving me the full 300 Mbps, it’s only giving me around 210 Mbps.

Background: In addition to the fibre modem, BT (the ISP) also provided me with a router known as the “Home Hub 3” (or HH3), which has a Gigabit Ethernet WAN port, and 4 Ethernet LAN ports, of which a single port is Gigabit; it also has Wifi. Using the HH3 I can consistently get just over 310 Mbps by visiting speedtest.net, whereas using the RT-N66U the highest I’ve ever seen is 210 Mbps. Given that SmallNetBuilder benchmarked the WAN to LAN speed of the RT-N66U at 732 Mbps, something clearly isn’t right.

All my tests are from a GigE wired PC (core i7, 8 GB RAM, Windows 8), have been repeated numerous times over a period of almost two weeks, and are very consistent: slightly over 310 Mbps with the HH3, slightly over 210 Mbps with the RT-N66U.

This is my setup:
Fibre --- Fibre Modem --- GigE --- Router --- GigE --- Gigabit Switch --- GigE --- PC

Swapping out the router between the Home Hub 3 and the RT-N66U produces very consistent results. Nothing else changes; the cables are the same, the LAN IP on both routers is set the same. Literally I disconnect the two GigE cables from one router and connect them to the other router, power the router on, wait for things to boot and connect (1 – 2 minutes in both router’s cases), and test.

QoS and DoS protection are both disabled on the RT-N66U. Until last night I was running firmware 260, and last night I upgraded to beta 321 -- same test results.

The Internet connection requires PPPoE (with a generic username and password), and I wonder if this is where the bottleneck is occurring? I tested plugging my 3-year-old Core 2 Duo laptop directly into the fibre modem and configured a PPPoE connection in Windows 8, and it connects fine, but I only achieved ~60 Mbps while the laptop's CPU rocketed to almost 100%. So it appears that the PPPoE connection is very CPU intensive, and perhaps that is what the Asus isn't handling very well. I suspect the Home Hub 3 has a flag on its PPPoE client that disables something (perhaps encryption) that makes it a lot less CPU intensive, allowing it to achieve the full 300 Mbps -- the HH3 is a small/cheap router, and I refuse to believe it has more processing power than the RT-N66U. Now I just need to know what PPPoE tweak is needed on the RT-N66U (there is a field for "additional parameters).

I’m hoping somebody has come across this issue and will be able to help me.

Everything else about the router is awesome, it seems stable, the Wifi performance is fantastic, and generally I’m very happy with it. 210 Mbps is an amazing speed to have, but given I can achieve 300 Mbps with the cheap ISP router, surely the RT-N66U can achieve this also!

Enquiring minds want to know.

Thanks for taking the time to read this, and thanks for your help!

 

[SoCalReviews]

Have you asked your ISP for suggestions? They might have some settings changes you can make on the modem (bridge mode, etc.) that can help. It sounds a bit odd but as you stated there might be some auto sensing feature in the ISP provided router that changes the settings in the modem (turns off DHCP and NAT in one of the devices to prevent double NAT, etc.) that you can manually make to optimize the connection when you use the RT-N66U instead of the ISP router. Your ISP may be able to help you understand the detailed function of the equipment they provided so you can determine how to narrow down the source of the bottleneck.

There might be some changes in the RT-N66U settings that could help eliminate the bottlenecks. I would recommend mostly keeping the router's default settings...for example don't enable QoS or other firmware settings that could add additional unnecessary load on the router's processing. Asus routers are already optimized for maximum throughput using mostly the default settings. I would also re-examine the quality of the interconnect Ethernet cable you are using. If you are using Cat5e then try Cat6. I use shielded/shielded twisted pair rated as Cat7 spec for all my interconnect cable. It may not make sense that there should be any differences if you swap out equipment using the same cables but there could be sensitivity differences that effect gigabit Ethernet equipment differently.

When you run your tests try connecting your computer that is running the speedtest.net test directly to the Asus router as opposed to down the chain. This sounds strange but you may also consider keeping the ISP router in the chain and connect the Asus router immediately after it (double NAT if necessary) and then run the speedtests. That might give you some idea whether the ISP router and modem combination in the chain are providing some unknown benefit working together. It could help you further narrow down the source of the bottleneck when you use the Asus.

 

[rotor]

First of all, thanks for taking the time to respond.

BT is one of the biggest (if not the biggest) ISP in the UK, and unfortunately are notoriously terrible at customer service (I only signed up with them because they are literally the only ISP serving my address with the fibre service). This is the disheartening part, that I know I have zero chance of actually getting in touch with someone at BT that even knows what I'm talking about, let alone being able to offer any help. If I call BT, the conversation will go along these lines: please plug in the Home Hub 3, power it on, and run your tests. Are you getting 300 Mbps? Yes? Ok, thanks for calling, glad we could help, buhbye.

I agree about the default settings, and that is what I have. QoS is disabled, and so is the DoS setting on the firewall settings page. The cable is fine, I have tried a couple of different cables, and I get consistent results regardless of what cable I use. The cable I'm currently using is Cat6 and brand new.

I am definitely going to test plugging my PC straight into the Asus to see if that makes a difference, although I can't see how (maybe some incompatibility between the Asus and my 3Com switch). Plugging the Asus into the back of the ISP's router will possibly prove that without PPPoE the ASUS is capable of more than 200 Mbps (which is what the SmallNetBuilder review already proves) -- but unfortunately that doesn't help me.

 

[got_milk]

The BT Home Hub uses a Broadcom BCM6361 SoC, which has a dual-core 400MHz processor. That would make it fairly more powerful than the RT-N66U, so the problem could be a CPU bottleneck overall.

You could see if this is the case by enabling Telnet/SSH access to the router, then logging in and running the top command. That'll show you load averages and CPU usage, which you can monitor while performing a speedtest. If the CPU usage jumps up to near 100% and hovers there while the speedtest runs, that'll likely be your reason why.

 

[SoCalReviews]

First of all, thanks for taking the time to respond.

BT is one of the biggest (if not the biggest) ISP in the UK, and unfortunately are notoriously terrible at customer service (I only signed up with them because they are literally the only ISP serving my address with the fibre service). This is the disheartening part, that I know I have zero chance of actually getting in touch with someone at BT that even knows what I'm talking about, let alone being able to offer any help. If I call BT, the conversation will go along these lines: please plug in the Home Hub 3, power it on, and run your tests. Are you getting 300 Mbps? Yes? Ok, thanks for calling, glad we could help, buhbye.

I agree about the default settings, and that is what I have. QoS is disabled, and so is the DoS setting on the firewall settings page. The cable is fine, I have tried a couple of different cables, and I get consistent results regardless of what cable I use. The cable I'm currently using is Cat6 and brand new.

I am definitely going to test plugging my PC straight into the Asus to see if that makes a difference, although I can't see how (maybe some incompatibility between the Asus and my 3Com switch). Plugging the Asus into the back of the ISP's router will possibly prove that without PPPoE the ASUS is capable of more than 200 Mbps (which is what the SmallNetBuilder review already proves) -- but unfortunately that doesn't help me.

What you really need to do is find out if the Asus router is the source of your bottleneck or if for some reason the ISP provided modem just works better when it is connected to the ISP modem. After you determine what the problem is or whether the Asus is the problem then you can decide how to arrange your network. As I suggested before this is the way I would try to debug the problem...

For testing purposes

ISP modem - ISP router - Asus router - computer running speed test

If you still experience a severe bottleneck problem then you know the problem doesn't just involve the ISP modem - Asus router combination. If you get better speeds...up to 300mbps with the dual router combination then you can suspect that the ISP modem and ISP router combo has some kind of advantage. It could be that for some reason the ISP router is handling and buffering the throughput from the modem's gigabit port better.

I understand that it seems you are defeating the purpose of eliminating the ISP router with the Asus but once you determine if the Asus router is causing the bottleneck then you can decide on your next course of action.

I actually have one of my networks set up with two gigabit routers...double NAT...and it works well...I can stream HD videos no problem and I get the full 100mbps throughput from my cable modem. I haven't tried more latency sensitive applications...such as online gaming or VOIP through double NAT but for most computer and streaming applications it works fine.

 

[SoCalReviews]

The BT Home Hub uses a Broadcom BCM6361 SoC, which has a dual-core 400MHz processor. That would make it fairly more powerful than the RT-N66U, so the problem could be a CPU bottleneck overall.

You could see if this is the case by enabling Telnet/SSH access to the router, then logging in and running the top command. That'll show you load averages and CPU usage, which you can monitor while performing a speedtest. If the CPU usage jumps up to near 100% and hovers there while the speedtest runs, that'll likely be your reason why.

Good point...The Asus CPU could indeed be the source of the bottleneck. If the connection is a real world 300mbps down and 20mbps up then that might be pushing both the Asus gigabit WAN port and/or the Asus single CPU. The problem is that review tests show that the RT-N66U has been able to handle that kind of throughput.

 

[rotor]

The BT Home Hub uses a Broadcom BCM6361 SoC, which has a dual-core 400MHz processor. That would make it fairly more powerful than the RT-N66U, so the problem could be a CPU bottleneck overall.

You could see if this is the case by enabling Telnet/SSH access to the router, then logging in and running the top command. That'll show you load averages and CPU usage, which you can monitor while performing a speedtest. If the CPU usage jumps up to near 100% and hovers there while the speedtest runs, that'll likely be your reason why.

I think you've possibly nailed it. The CPU in the RT-N66U is 600 MHz single-core, while the CPU in the HH3 as you pointed out is 400 MHz dual-core, which I guess adds up to 800 MHz of total power.

So I've telnetted to the N66U, and while transferring a large file, sirq jumps from 0% to 97%. Here is the output from top:

At load (speedtest.net showing a result of 203 Mbps):
Mem: 46896K used, 192964K free, 0K shrd, 6288K buff, 21656K cached
CPU: 0% usr 1% sys 0% nic 0% idle 0% io 0% irq 97% sirq

And at idle:
Mem: 46896K used, 192964K free, 0K shrd, 6288K buff, 21656K cached
CPU: 0% usr 0% sys 0% nic 99% idle 0% io 0% irq 0% sirq

So if the bottleneck is caused by interrupts, according to a brief googling there's a way of enabling interrupt coalescing (in Linux in general, not the N66U specifically).

 

[rotor]

Good point...The Asus CPU could indeed be the source of the bottleneck. If the connection is a real world 300mbps down and 20mbps up then that might be pushing both the Asus gigabit WAN port and/or the Asus single CPU. The problem is that review tests show that the RT-N66U has been able to handle that kind of throughput.

How come SmallNetBuilder can benchmark it at 732 Mbps then?

 

[bluepoint]

Have you played with the mtu values? Most PPPOE mtu implementations are optimized at 1492, try other values and see if speed changes.

 

[rotor]

Have you played with the mtu values? Most PPPOE mtu implementations are optimized at 1492, try other values and see if speed changes.

I compared the ISP's router and the RT-N66U, and they both have a 1492 MTU (tested by pinging out with the no-fragment flag and trying different sizes).

I just set it to 1454 on the RT-N66U and tested, and got virtually the same results (198.5 Mbps).

 

[SoCalReviews]

How come SmallNetBuilder can benchmark it at 732 Mbps then?

Maybe the PPPoE requires additional processing overhead that the dual core ISP provided router can handle better. Did you get any results from testing the setup I recommended where you simply add the Asus router after the ISP router and then run the same speedtest? If you got a better throughput of closer to 300mbps when plugged into the Asus in this way then that would possibly show that the ISP router has an advantage when paired with the ISP modem but also that the Asus router is capable of higher throughput than 210mbps when it's not paired to the ISP modem. I would think that would be kind of an important result to narrow down the problem. It would also support a reason for the SNB benchmark of more than 700mbps. That's just how I would debug it instead of spending any more time guessing.

 

[got_milk]

If the PPPoE connection uses encryption/compression, that would definitely explain why it's so CPU intensive.

You could use the BT Home Hub as a dedicated router, and let the RT-N66U be a wireless access point? That'll allow you to reach your maximum speed while retaining the excellent wireless capabilities of the Asus router.

I can't think of any other ideas.

 

[rotor]

If the PPPoE connection uses encryption/compression, that would definitely explain why it's so CPU intensive.

You could use the BT Home Hub as a dedicated router, and let the RT-N66U be a wireless access point? That'll allow you to reach your maximum speed while retaining the excellent wireless capabilities of the Asus router.

I can't think of any other ideas.

The one feature I want is the PPTP server so I can VPN into my router from my iPhone and/or laptop when I'm out and about. I would lose the ability to do that.

 

[rotor]

Maybe the PPoE requires additional processing overhead that the dual core ISP provided router can handle better. Did you get any results from testing the setup I recommended where you simply add the Asus router after the ISP router and then run the same speedtest? If you got a better throughput of closer to 300mbps when plugged into the Asus in this way then that would possibly show that the ISP router has an advantage when paired with the ISP modem but also that the Asus router is capable of higher throughput than 210mbps when it's not paired to the ISP modem. I would think that would be kind of an important result to narrow down the problem. It would also support a reason for the SMB benchmark of more than 700mbps. That's just how I would debug it instead of spending any more time guessing.

I will run that test tonight. I'm 99% certain the problem is CPU related, but I still reckon there is some tweak that can be made to the PPPoE client in the RT-N66U to make it less CPU intensive.

 

[rotor]

Last night I built a Mikrotik RouterOS VM and gave it a dedicated physical NIC on my ESXi host (for the WAN port), and I achieved 300+ Mbps no problem. When I started throttling the CPU resources granted to the VM, it only started dropping below the full 300 Mbps when I went down to 400 MHz (at 500 MHz it was still capable of routing the full speed). I guess this only proves that a 600 MHz processor (what the RT-N66U has) isn't as vastly powerful as I thought it was (before I bought it).

I'm going to try and source one of these:
http://routerboard.com/RB951G-2HnD

 

[got_milk]

Remember that the CPU architecture of a desktop-grade machine is different and likely provides better overall performance than the MIPS processor that's in the RT-N66U.

That doesn't make the router's processor incapable by any stretch of the means. It's easy to understate how difficult and processor intensive routing encrypted/compressed bandwidth at 300Mbps is. I'm actually quite surprised you get the speeds you do with the RT-N66U.

The RB951G-2HnD you linked to will likely suffer the same problem as the RT-N66U, as it's using an Atheros processor, which uses the same architecture as the Broadcom CPU in the RT-N66U.

 

[rotor]

Remember that the CPU architecture of a desktop-grade machine is different and likely provides better overall performance than the MIPS processor that's in the RT-N66U.

That doesn't make the router's processor incapable by any stretch of the means. It's easy to understate how difficult and processor intensive routing encrypted/compressed bandwidth at 300Mbps is. I'm actually quite surprised you get the speeds you do with the RT-N66U.

The RB951G-2HnD you linked to will likely suffer the same problem as the RT-N66U, as it's using an Atheros processor, which uses the same architecture as the Broadcom CPU in the RT-N66U.

Excellent points. I completely agree that the processor architectures are completely different, for me it was more to prove that it is indeed a very CPU intensive task.

The RB951G-2HnD isn't available for another 2-3 weeks (it's new), so in the meantime I've asked their pre-sales whether they think it can handle 300 Mbps over PPPoE.

I've been looking at the Intel Next Unit of Computing (NUC), but the bare unit (without RAM or mSATA disk) is £220 -- so quite a big step up from the RT-N66U at around £110. It would be super powerful though, I'm tempted from the "just because you can" factor.

 

[got_milk]

You'd need a second network interface on the Intel NUC though (one for a WAN connection and at least one to go to a switch or directly to a machine). I think you can get Thunderbolt NICs but those are probably pricey and the NUC is pretty expensive as it is anyway.

 

[rotor]

You'd need a second network interface on the Intel NUC though (one for a WAN connection and at least one to go to a switch or directly to a machine). I think you can get Thunderbolt NICs but those are probably pricey and the NUC is pretty expensive as it is anyway.

There are two versions of the NUC. One has dual GigE. The model number is DC3217IYE.

 

[got_milk]

There are two versions of the NUC. One has dual GigE. The model number is DC3217IYE.

http://www.intel.com/content/www/us...oards/desktop-boards-d33217gke-dc3217iye.html

It has one Ethernet interface. The other model doesn't even have one specified in either the box or the motherboard itself.