SNB's Router Test Gets Tougher - A Preview

[thiggins]

Here's a sneak peek at the second part of our upcoming router test process.

Read on SmallNetBuilder

 

[bompus]

Tim,

I found out, after much painful testing, that ApacheBench will treat a response as an error if all of the responses don't match the same exact content length. This could be the reason why your upload tests were failing when testing some routers. It's not a guaranteed explanation, but always good information to have on hand when using AB for testing. I actually ended up going from ab, then tried siege, then finally ended up on my own app written in NodeJS for load testing, so I was aware of what it was actually doing.

Hopefully the info helps

 

[David Thatcher]

You also need to test Fast Ethernet (100Mb) devices connected directly to the router. I found out recently that the Archer C2600 only gives 40-60Mbps with Fast Ethernet devices (they ran full speed when separated by a gigabit switch). This is very much important because a lot of smart TVs, smart players, powerline adapters, etc. only have Fast Ethernet. It's an easy way for them to save money. We need to know that the router can handle Fast Ethernet connections at full speed, in addition to any gigabit capabilities it may have.

 

[thiggins]

I found out, after much painful testing, that ApacheBench will treat a response as an error if all of the responses don't match the same exact content length.

I was aware of that through my Googling. But the problem was usually attributed to dynamic content. I'm just downloading static JPG and PNG files. So if the content length changes, I'd say the router isn't working properly, right?

 

[thiggins]

You also need to test Fast Ethernet (100Mb) devices connected directly to the router. I found out recently that the Archer C2600 only gives 40-60Mbps with Fast Ethernet devices (they ran full speed when separated by a gigabit switch). This is very much important because a lot of smart TVs, smart players, powerline adapters, etc. only have Fast Ethernet. It's an easy way for them to save money. We need to know that the router can handle Fast Ethernet connections at full speed, in addition to any gigabit capabilities it may have.

Interesting. But the problem isn't necessarily due only to the router. The NIC I test with could work fine, but another may not.

 

[sfx2000]

Hmmm... just wondering - one of the issues that routers have to deal with is packet loss and wide differences in latency on the WAN vs. LAN side... so one might see 600-1000 microseconds from LAN host to Router, but it might be an order of 10 larger on the first hop to WAN host on the WAN side of the router - and each hop, let's say to Facebook or Netflix might add yet another 10-15 milliseconds...

netem can do some of this modeling... and this can be run on the WAN host simulator...

https://wiki.linuxfoundation.org/networking/netem

Latency handling is a big deal for some of the forum members here, and it can stress lower end devices moreso than higher end devices... esp. when Quality of Service comes into play...

Thoughts here?

 

[sfx2000]

We need to know that the router can handle Fast Ethernet connections at full speed, in addition to any gigabit capabilities it may have

When one looks at the typical BHR, the internal switch element of the System on Chip should have no issues with a mix of fast ethernet on some ports and gigabit on others.

It's not the same as WiFi, where the medium is shared across all associated stations, so a slow client impacts the link - in switched ethernet, each attached client has it's own dedicated link.

 

[thiggins]

Hmmm... just wondering - one of the issues that routers have to deal with is packet loss and wide differences in latency on the WAN vs. LAN side...Thoughts here?

This is way into the weeds for little ol' SmallNetBuilder.

 

[sfx2000]

Without turning this into a linux kernel tuning discussion...

Ubuntu, out of the box, is pretty conservative with socket buffers, even Ubuntu Server...

For high speed links (gigabit or faster) on high performance servers/services, I've found the following to be helpful, as we don't need to be blocking transactions...

Add/edit the sysctl.conf values over in /etc...

# Some kernel tweaks for Network performance
# aggressive, but good for gigabit
#
# these might not work for everyone
#
# socket buffer space - sounds like a lota at  16MB per socket. it isn't
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
# These are the corresponding settings for the IP protocol, in the format
# (min, default, max) bytes. The max value can’t be larger than the
# equivalent net.core.{r,w}mem_max.
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
# Increase the number of outstanding syn requests allowed.
net.ipv4.tcp_max_syn_backlog = 4096

 

[sfx2000]

This is way into the weeds for little ol' SmallNetBuilder.

I like where you're going with this update to the test plan, and it's definitely a good addition..

 

[James5mith]

Would be interested in something like the pfsense/Netgate SG-2220 appliance as a baseline. It should in theory provide true wire speed performance, even with a lot of inspection happening.

https://www.netgate.com/products/sg-2220.html

 

[thiggins]

Kernel tweaks were done as described in the Ars article. ("So about that testing section").

 

[David Thatcher]

Interesting. But the problem isn't necessarily due only to the router. The NIC I test with could work fine, but another may not.

In my case, two different Fast Ethernet NICs, as well as a Gigabit NIC set for Fast Ethernet, all exhibited reduced speeds with the Archer C2600.

 

[sfx2000]

Kernel tweaks were done as described in the Ars article. ("So about that testing section").

Give my numbers a try

 

[Jim Salter]

The new process will still have the weakness of not checking performance with encrypted WAN connection types, unfortunately.

Hi Tim. What are you referring to here? Router-managed VPN and forwarding most LAN-to-WAN traffic down a tunnel to the endpoint?

If that is what you're referring to, you can test it the same way you did here... just set up your "local" router inside your testing LAN, set up a "remote" router connected to its WAN port, and put your WAN-side testing server on the other side of your "remote" router (and its tunnel endpoint).

FWIW, I did some testing of this recently for a forthcoming article, and got about 30 Mbps for a Netgear R8000 running Kong DD-WRT, with an endpoint in DigitalOcean. The throughput varies depending on your choice of cipher/digest pair, with the bottleneck being the R8000 (not the endpoint in the $5/mo droplet at DigitalOcean, which is capable of pushing 200+ Mbps).

 

[thiggins]

What are you referring to here? Router-managed VPN and forwarding most LAN-to-WAN traffic down a tunnel to the endpoint?

Hi Jim. Thanks for checking in. I'm referring to the WAN ISP connection type, such as PPPoE, L2TP, etc. I'm told the encryption required in these connection types can reduce router throughput.

As for VPN tunnel throughput, Doug Reid usually does that testing. He usually does a client-to-gateway test and uses another faster router for gateway-to-gateway tests. But we don't use a consistent method, so can't publish charts for VPN performance. It's on the list.....

There are some other threads with users reporting VPN tunnel performance. Lots of folks looking to at least investigate going full VPN these days.

Keep up the great work on Ars, BTW. I enjoyed your latest run at mesh testing.

 

[Jim Salter]

Thanks Tim! I always check your take on any hardware I'm looking at. Not much point in trying to do my own hardware teardown if you've been there before me. =)

Is PPPoE encrypted? I thought it was authentication-only? I have to admit I've paid as little attention to it as possible, since I try to avoid any DSL-based connection types.

I'm not sure off the top of my head how you'd go about setting up the server side of a PPPoE link, but L2TP and PPTP are easy enough to set up on either Windows or Linux, if you're willing to put in the effort. Are they common as plain old WAN connection types, as opposed to actual VPNs though? I've only ever seen them used for small or mid size business VPN connections.

 

[thiggins]

Is PPPoE encrypted? I thought it was authentication-only? I have to admit I've paid as little attention to it as possible, since I try to avoid any DSL-based connection types.

I'm about in the same boat as you. Haven't paid much attention because it's not really that feasible to test on an ongoing basis. Or at least I don't think it's feasible. I usually get pinged from test gear suppliers and have never heard from anyone peddling a DSLAM in a box.

 

[Jim Salter]

I'm about in the same boat as you. Haven't paid much attention because it's not really that feasible to test on an ongoing basis. Or at least I don't think it's feasible. I usually get pinged from test gear suppliers and have never heard from anyone peddling a DSLAM in a box.

Some quick googling comes up with this: https://linux.die.net/man/8/pppoe-server

I also found a how-to with someone who got it working on Ubuntu: https://poundcomment.wordpress.com/2011/03/30/pppoe-server-on-ubuntu/

As far as I can tell, though... there really shouldn't be any encryption overhead? Looking at this stuff, all I'm seeing is CHAP authentication https://en.wikipedia.org/wiki/Challenge-Handshake_Authentication_Protocol which occurs at random intervals, but I wouldn't THINK they'd be frequent enough to pose any serious overhead - it's not like the per-packet "authentication" digest in something like OpenVPN, much less an actual encryption of the payload in each packet. This should be more along the lines of passing a password hash once every few hundred? thousand? packets, AFAICT.

I'll also note that this whole freaking concept of PPPoE and CHAP strikes me as insanely broken, not a whole hell of a lot more secure than ROT13, and I cannot for the life of me think what DSL providers believe they're gaining from using it.

 

[Alwyn]

This new router test is a great improvement over previous tests because it can better expose weaknesses in routers.

I would like to make 3 suggestions:

1. Test for bufferbloat and jitter (perhaps by pinging the host every second). These are important, but often overlooked or neglected, performance metrics.
2. Conduct a full-duplex test (IE, 1 Gbps upload and download for a combined 2 Gbps) rather than just a unidirectional test. This is important to determine how well a router handles the upstream and downstream connection when it is pushed to its limits.
3. Log the number of dropped packets and disconnections. This will determine how well the router handles flow control.

Please let me know what you think of including these suggestions and thank you for all the work you have done to help consumers like me decide what router to buy.