RT-N66 performance: Asus Stock firmware vs Tomato

[jyavenard]

There's been lots of talk about how much better the Asus RT-N66 stock firmware performed over Tomato.

Being an irregular developer on the tomato project, and having recently acquired a RT-N66U, I wanted to identify the issue considering that:

- Tomato now uses the exact same broadcom drivers as Asus latest firmware
- The Tomato kernel is slightly more recent (note: the newer Asus 144 firmware has caught most of those changes up)
- Tomato is leaner, with far less junk running...

So, with the same hardware and drivers, how could tomato be slower?

So I wanted to quantify the difference in a particular use case: my typical usage.

Speed over a 5Ghz link between a macbook air (dual channel so 300Mbit/s max) located in another room than the RT-N66, about 12m away.

The test consisted of varying the transmit power, then transfer a 512MB file of random data using rsync over a ssh connection.

Peak transfer speed was looked at the hardware interface level
Average transfer speed is the average speed reported by rsync at the end of the transfer.
Repeat three times.

When I first started my test, the Asus was crushing tomato big time. It was significantly faster. But very irregular too.

Trying to investigate the differences between tomato and the Asus stock firmware, I noticed that Asus by default uses the hardware accelerated NAT driver (ctl kernel module, it's not just for NAT but that's what Asus calls it).
Tomato doesn't use this module.
This module is incompatible with netfilter and qos ; the second you enable QoS with the asus firmware, the ctl module is unloaded.

Here are the results:
https://docs.google.com/spreadsheet/ccc?key=0ArnXIvHcJfCzdERlRzd5TUR1YndnRmJhU09SU0toVFE

Conclusion: Asus stock firmware is only faster when no QoS is in use...
also tomato provides more consistent throughput.

If you want to use QoS, you're better off using Tomato.

There's some other differences in regards to signal level, the Asus stock firmware provides a more linear power output... I need to sort this out on Tomato which shouldn't be too complicated

I also tested router transfer speed over gigabit ethernet.
The results were also very interesting.
Asus with ctl: 66.4MB/s (this is in line with SNB's own test)
Asus without ctl: 45MB/s
Asus with QoS on: 20MB/s
Tomato: 24.1MB/s

So once again, tomato is faster with QoS active.

Summary: don't use QoS and want best performance: use Asus firmware
Use QoS? use tomato a

 

[Bitgod]

Of course, now I have to decide which way to go. I've never had a need for QoS, so wouldn't have bothered, but just bought an Ooma telephony device and thinking I might need to QoS for that. Though, all things considered, the phone call seems to only take 6 KB both ways and I don't do tons of calls, so just as easy to slightly throttle when doing big downloads.

Well, except I'm also reminded now that one of the things I liked using DD-WRT previously was keeping track of monthly bandwidth useage, which the stock firmware doesn't do as well....so that might still make me go Tomato.

 

[RMerlin]

The results were also very interesting.
Asus with ctl: 66.4MB/s (this is in line with SNB's own test)
Asus without ctl: 45MB/s
Asus with QoS on: 20MB/s
Tomato: 24.1MB/s

One suspicion is that when QoS is enabled, it throttles transfers between vlan1 and eth1/eth2, not just vlan1/eth1/eth2 and br0. If you repeat your test with QoS enabled but with your max down/upload set to, say, 300 Mbits, do you still get only 20 MB/s?

Recent changes by Asus now use bridge_netfilter in an attempt to identify from which interface the traffic comes - which I assumed was their fix to this issue. At least one person reported that it did not fix that issue, and that changing his down/up limits was still affecting his wireless performance.

 

[jyavenard]

One suspicion is that when QoS is enabled, it throttles transfers between vlan1 and eth1/eth2, not just vlan1/eth1/eth2 and br0. If you repeat your test with QoS enabled but with your max down/upload set to, say, 300 Mbits, do you still get only 20 MB/s?

Actually, QoS being on was only set as a way to disable CTF.

The limits set in QoS were for each gigabit (1000Mbit/s for both upload and download)...

Something I posted elsewhere, but probably more relevant here.
hardware acceleration is disabled not just when QoS is active but also when you enable port forwarding, DMZ and URL blocker.

You can also disable CTF directly, (LAN -> Switch Control -> Disable HW acceleration).
Disabling CTF alone, and not activating QoS, saw the speed drop from 66.4MB/s to 45MB.s

I think CTF was only used to make the RT-N66U rates well in reviews and benchmark like the one done by SNB: they do their tests with everything disabled but routing.. As such, you get artificially high results that are in no way representative of what you'll see in the real world.


I'd love to see how much they get on the RT-N56, with port forwarding or QoS turned on (even if not active by way of setting very high limit). I doubt you would see 1.2Gbit/s like they get now

 

[RMerlin]

Asus are also using GRO by default, which is supposed to improve performance, but was however the source of crashes with VPN and QOS. I'd be curious to see if disabling GRO (which I have done in my test 144.10 build) has any actual impact on performance.

 

[Gitsum]

Asus are also using GRO by default, which is supposed to improve performance, but was however the source of crashes with VPN and QOS. I'd be curious to see if disabling GRO (which I have done in my test 144.10 build) has any actual impact on performance.

What is GRO?

 

[RMerlin]

Google is your friend: http://vger.kernel.org/~davem/cgi-bin/blog.cgi/2010/08/30

 

[icemankent]

I am one who definitely needs QoS to work.
(VoIP, torrenting, streaming - you name it - but need to protect those Voice Calls !!)

I am a fan of how D-link routers let you configure QoS.

--Q1--
Regarding Tomato (I am assuming Shibby here), does anyone have a link to either (hi-res aka "readable") screen shots, or a tutorial, or other type of instructional manual ?

I bought (and returned) the RT-n66u before trying Tomato-Shibby - but if I am comfortable with the QoS functionality on Tomato, I may go get the router again and give it a try.

PS - the only walkthrough I saw was on youtube, and what threw me off was when he was configuring the wan interface, the drop down list did NOT have a 1000 Mbps (GigE) option.
I thought that was a huge bug for Tomato, since all of the rt-n66u's interfaces are GigE.
http://www.youtube.com/watch?v=FnfdmN-Oq58

--Q2--
Can you also confirm if the latest Shibby build has fixed that as well ?

 

[sovereignty68]

Google is your friend: http://vger.kernel.org/~davem/cgi-bin/blog.cgi/2010/08/30

really. General Register Office? Girls Riders Organization? Greater Rochester Orthopaedics? That's what google came up. Google isn't always your friend.

What is GRO?

GRO = Generic Receive Offload

 

[RMerlin]

really. General Register Office? Girls Riders Organization? Greater Rochester Orthopaedics? That's what google came up. Google isn't always your friend.


GRO = Generic Receive Offload

I didn't know what GRO was until I googled it a few nights ago. "GRO Linux" or "GRO Networking", forgot which one I used.