SNB Routing Performance in reviews

[cruiser]

Hi Guys

I was wondering what you knew about the tests done on SNB. I've read multiple reviews to different routers over the years now, but I never seem to reach those performances (with stock firmware) in my setup. The author has explained his newer setup here:

http://www.smallnetbuilder.com/lanwan/lanwan-howto/31103-how-we-test-hardware-routers-revision-3

When looking at the review from the Netgear R7000 we see a Lan-WAN Performance of 941.7 Mbps
http://www.smallnetbuilder.com/wire...etgear-r7000-a-asus-rt-ac68u?showall=&start=1

But when visiting the dd-wrt site on the R700, they mention that it has 450Mbit/s IPv4 NAT performance on stock firmware.
http://dd-wrt.com/wiki/index.php/DD-WRT_on_R7000

How to you explain those discrepancies? When I read the authors explanation I understand that he is testing NAT performance, because who has a full routed network in his "small" network.

fyi: This is in no way an attack on SNB. I just think I'm misunderstanding something. I really like the site

 

[thiggins]

Without knowing the test method the DD-WRT article author used, I can't explain the difference.

IxChariot uses a very efficient test method that uses very little CPU. iPerf/Jperf can be tricky to configure and will report lower throughput than IxChariot with default settings.

 

[cruiser]

Hi Tim,

Thanks for your reply. You're right, it's hard to estimate how they do their testing. But is it correct that you test NAT performance in your setup? I was not sure on that.

 

[thiggins]

Thanks for your reply. You're right, it's hard to estimate how they do their testing. But is it correct that you test NAT performance in your setup? I was not sure on that.

I don't know how I would do WAN > LAN and LAN > WAN tests without going through the NAT firewall.

 

[cruiser]

I don't know how I would do WAN > LAN and LAN > WAN tests without going through the NAT firewall.

Ok thanks.

You could set the routers in "Routing mode" and route the traffic. That way the router doesn't have to do any NATing. In the real world this happens when you have access to an AS and can announce your own subnet to the Internet. (All IPs are public)

Natting is pretty intensive for the router, as it must change every packet and keep track of that, whereas if you route, you just have to adjust the TTL. So without nat, you get different numbers.

 

[thiggins]

I suppose I could test straight routing. But most consumer router users use NAT, so that's the way I test 'em.

 

[Mordred]

The 500mbps is with pppoe.

I have seen multiple reports from users on amazon and netgear forum that mentioned this.

Thus most likely the author did a real world test, not a synthetic test like SNB.

The >900mbps can only be reached with a standard config and test tools, but in a lot of cases CTF is disabled and then routing comes to a crawl.

I'm still surprised Tim hasn't done any researched on this and considers this in his tests. That's one of the reasons why I don't trust reviews anymore, they don't reflect real world.

 

[cruiser]

The 500mbps is with pppoe.

I have seen multiple reports from users on amazon and netgear forum that mentioned this.

Thus most likely the author did a real world test, not a synthetic test like SNB.

The >900mbps can only be reached with a standard config and test tools, but in a lot of cases CTF is disabled and then routing comes to a crawl.

I'm still surprised Tim hasn't done any researched on this and considers this in his tests. That's one of the reasons why I don't trust reviews anymore, they don't reflect real world.

I hadn't thought about the pppoe part. I use cablemodem, so I'm not affected by that. I've made the experience that iperf seams to reflect performance pretty good. Btw, what does CTF stand for?

 

[thiggins]

Thus most likely the author did a real world test, not a synthetic test like SNB.

We can't really know that, can we, since the test itself and test conditions are not described anywhere?

SNB tests on a private LAN so that variables are controlled. We also test with router defaults since that is how most people use the routers. All this and other test methods are always described in detail in our "How We Test" articles. Those are rarely found in other reviews.

I know PPPoE, PPTP and L2TP throughput is different. But testing with those on a private LAN would require equipment we don't have access to.

 

[cruiser]

That's absolutely understandable. You guys test so many routers, you would never be done.

 

[Mordred]

We can't really know that, can we, since the test itself and test conditions are not described anywhere?


SNB tests on a private LAN so that variables are controlled. We also test with router defaults since that is how most people use the routers. All this and other test methods are always described in detail in our "How We Test" articles. Those are rarely found in other reviews.

I know PPPoE, PPTP and L2TP throughput is different. But testing with those on a private LAN would require equipment we don't have access to.

No you can't know how the test was done, but you see I'm just an advanced user that sometimes reads user comments, this way I kind of know these things.

I really like your tests, but I'm missing a few tests that will show some of their weaknesses. E.g. CTF does not work with all features thus it is turned of in some cases, in asus firmwares there are different levels for hw acceleration.

Then a test with a single wireless adapter is ok, but I'm sure it would be interesting nowadays to see a simultaneous test where a few clients transfer at the same time, e.g. two on 2.4G and two on 5G. I mean how many people only have one wireless device at home?

Because your tests are so simple it is easy for manufacturers to tune the default config in order to make them look good at benchmarks.

 

[thiggins]

I agree with your points, Mordred. But we do more tests than any other site and can't test everything. I occasionally disable Cut Through Forwarding/ NAT acceleration to see the difference for those products that allow it.

For wireless (and any other benchmarks), yes, it's possible that manufacturers tune configurations to do well on SNB tests. But how about tests on other sites? Can't tune defaults to ace everyone's tests.

You will see some multi-client tests introduced with Wireless Test Process V8, which is coming in a few weeks. They will probably still be too simple for you.

 

[RMerlin]

Hi Guys

I was wondering what you knew about the tests done on SNB. I've read multiple reviews to different routers over the years now, but I never seem to reach those performances (with stock firmware) in my setup. The author has explained his newer setup here:

http://www.smallnetbuilder.com/lanwan/lanwan-howto/31103-how-we-test-hardware-routers-revision-3

When looking at the review from the Netgear R7000 we see a Lan-WAN Performance of 941.7 Mbps
http://www.smallnetbuilder.com/wire...etgear-r7000-a-asus-rt-ac68u?showall=&start=1

But when visiting the dd-wrt site on the R700, they mention that it has 450Mbit/s IPv4 NAT performance on stock firmware.
http://dd-wrt.com/wiki/index.php/DD-WRT_on_R7000

How to you explain those discrepancies? When I read the authors explanation I understand that he is testing NAT performance, because who has a full routed network in his "small" network.

fyi: This is in no way an attack on SNB. I just think I'm misunderstanding something. I really like the site

I suspect that NAT performance might be with hardware acceleration disabled. In my own tests with a slower RT-AC56U (only 800 MHz), I get around 350 Mbits of WAN to LAN throughput when HW acceleration is disabled. Once I enable HW acceleration, I get 750+ Mbps of throughput. My tests were done with iperf.

My tests are done with NAT enabled, so that would be inline with the 450 Mbps result you are mentioning.

 

[chadster766]

We can't really know that, can we, since the test itself and test conditions are not described anywhere?

SNB tests on a private LAN so that variables are controlled. We also test with router defaults since that is how most people use the routers. All this and other test methods are always described in detail in our "How We Test" articles. Those are rarely found in other reviews.

I know PPPoE, PPTP and L2TP throughput is different. But testing with those on a private LAN would require equipment we don't have access to.

Hi Tim,

PPPoe, PPTP, L2TP and IPv6 testing you could probably do with equipment you have in stock.

All you need is a Intel i5 Desktop with a good amount of memory and two PCI-E Gigabit Ethernet cards. On this machine run MikroTik RouterOS. This setup will give testing ability for PPPoe, PPTP and L2TP. PPPoe testing on a system like this can get results +900MBit/s. Combine that with your xChariots testing process the sky's the limit. MikroTik RouterOS can also do the IPv6.

For IPv6 testing another way I recommend is using Hurricane Electric's Tunnel Broker. This will get the routable IPv6 Prefixes to hand off to the routers you want to test and also provide routable /64's to test platforms for local WAN throughput testing.