How We Test Hardware Routers - Version 4

[thiggins]

The long overdue update of our Router test process brings powerful new hardware to bear and adds functional tests.

Read on SmallNetBuilder

 

[CDROUTER Jason]

Well, I guess that means we should make a forum account Great working with you, Tim! Hopefully your readers will find this useful.

 

[System Error Message]

Well, I guess that means we should make a forum account Great working with you, Tim! Hopefully your readers will find this useful.

How do i get one without losing an arm or leg and is it powerful enough to test multi gigabit routers like the mikrotik CCR line?

Is it possible to not allow the router to recover from DoS attack? My router is pretty fast and im pretty sure it doesnt need time to recover.

 

[CDROUTER Jason]

How do i get one without losing an arm or leg and is it powerful enough to test multi gigabit routers like the mikrotik CCR line.

Well, it's an enterprise level test product, for sure. Our main customers are the vendors themselves and service providers. Tim worked with us on getting his new test procedure in place since the good work he does here was very much like the work CDRouter does for lab-grade situations.

 

[thiggins]

Is it possible to not allow the router to recover from DoS attack? My router is pretty fast and im pretty sure it doesnt need time to recover.

Yes. Pretty much everything is programmable.

As Jason said, CDRouter is not aimed at or priced for individual consumers. Like Ixia and octoScope, QACafe is partnering with SmallNetBuilder to help us provide the most comprehensive test processes you'll find from any publication.

 

[Cloud200]

Are there going to be any devices you will go back to and run the new testing process on?

p.s.
Thanks for all the great work over the years!

 

[thiggins]

Are there going to be any devices you will go back to and run the new testing process on?

Yes. I'll be retesting a selection of AC1900 and up routers. Might do a few AC1200 and AC1750, too.

 

[Razor512]

So I guess we really won't be seeing these devices at out local electronic stores for ~$300...

Anyway, for the functional tests, for some of the latest routers, are there many differences in how many tests they pass? Can the NTA1000 automatically change settings on a consumer router?
I have noticed that on some routers tend to have some of the less common functions disabled by default, and all of them cannot really be enabled at the same time. For example, on many TP-Link routers, the IPv6 stuff is disabled by default.

http://i.imgur.com/x295HOV.jpg
http://i.imgur.com/nOGEvDy.jpg

 

[thiggins]

CDRouter can't change router configurations. That's why I use a standard configuration for testing.

I'm not testing IPv6, IPv4 only. Reasons are here. Maybe in a year or so things will be better.

 

[System Error Message]

Well i was hoping to perform the functional test on the CCR while it has traffic so that if say you had gigabit level traffic and had a DoS attack at the same time or testing how fast will it perform functionally other than NAT and how secure it is.

@thiggins i know consumer routers have flaky ipv6 or that not all 3rd party firmware have very good ipv6 implementations but for routers that arent in the consumer space or using 3rd party firmware could you perform a full security and functionality test? I think it would be good to know where a router fails.

Just out of curiousity how much do one of these units cost?

 

[RMerlin]

Nice To this date, one of my favorite reviews you've posted remain this one. It actually made me go back to look at a few things, I remember fixing one or two of the issues reported there.

 

[vnangia]

Would you be able to retest some hardware that you have tested in the past (for example, the EdgeRouter?) so that we have some idea of how much more demanding this test regimen is? Cheers.

 

[thiggins]

I'll be retesting some products, yes. EdgeRouter is not likely to be one. The performance tests don't significantly push the products harder. The functional tests are what's really new.

 

[vnangia]

Fair enough!

Second question, having had time to look through that long PDF: is there scope now for you to test security gateways (I'm thinking of, for example, ZyXel or SonicWall) to get a sense of what penalty there is for activating the UTM features? That would be a really helpful thing for those of us considering devices that can do some amount of threat protection and detection.

 

[System Error Message]

@thiggins could you do a functional test on a mikrotik hEX lite and also test it for reliability under load? Preferably it would be better to test their MIPS with gigabit ports and PPC based ones like the RB850gx2. what i mean by reliability is that do a functional test but at much higher speeds such as how well would it cope with many upnp sessions popping up simultaneously and so on while it is under load.

 

[thiggins]

Second question, having had time to look through that long PDF: is there scope now for you to test security gateways (I'm thinking of, for example, ZyXel or SonicWall) to get a sense of what penalty there is for activating the UTM features? That would be a really helpful thing for those of us considering devices that can do some amount of threat protection and detection.

UTM testing is beyond the scope of what I or CDRouter can do. You need a threat/malware "zoo" and very different test processes. This is best left to the labs that focus on this work.

 

[thiggins]

@thiggins could you do a functional test on a mikrotik hEX lite and also test it for reliability under load? Preferably it would be better to test their MIPS with gigabit ports and PPC based ones like the RB850gx2. what i mean by reliability is that do a functional test but at much higher speeds such as how well would it cope with many upnp sessions popping up simultaneously and so on while it is under load.

Not likely anytime soon. There are too many other process and site changes I am working on now.

 

[sfx2000]

UTM testing is beyond the scope of what I or CDRouter can do. You need a threat/malware "zoo" and very different test processes. This is best left to the labs that focus on this work.

I think the test cases are more than enough - and to SEM's question/comment regarding the uTik - my best guess would be that the test equipment vendors used it, along with others to validate the test scripts.

Would be interesting to see the results, but I expect NDA's would perhaps prevent disclosure in case anything went wrong with the device during test case/test plan development.

This is all good stuff...

 

[System Error Message]

But i wanted to buy one, or an x86 software version i could use.

 

[thiggins]

But i wanted to buy one, or an x86 software version i could use.

I'm sure if you contact the QACafe folks, they'll be happy to quote you one. But expect at least a five figure price. As has been said, this is test equipment for manufacturers and labs and is priced accordingly.