How We Test Wi-Fi - Revision 11

[thiggins]

Our Revision 11 process brings multiple clients and a change to WAN-based testing.

Continue reading on SmallNetBuilder

 

[thiggins]

This was a tough birthing. The system can do so much more. Boiling things down to chartable and digestable benchmarks was a real struggle!

 

[sfx2000]

This was a tough birthing. The system can do so much more. Boiling things down to chartable and digestable benchmarks was a real struggle!

It's a significant effort, no doubt...

I see that the OctoScope Stack-Max includes the MPE2-38 faders...

Are you including the NLOS dynamic fading models where multipath is available?

This comes into play with the different Rate Adaptation algorithms that the chipset vendors use - QC-Atheros has their own, as does Broadcom and Intel.

This is important for beamforming, DL/UL MU-MIMO, OFDMA, etc...

What happened to flent? I know you were looking at it some time back...

Interesting that the PAL's moved from QC-Atheros over to Intel for 11ax - ath10k, with the right agreements with Qualcomm, was pretty good - CandelaTech did a lot of work there on those 11ac Wave 1/2 drivers, and that ended up being the basis for community efforts with the ath10k-ct drivers...

 

[thiggins]

I'm not using the MPE2's for this round of benchmarks. The attenuators are only 60 dB, which limits test range for 2.4 and 60 GHz (@ 160 MHz B/W).
I can still see MU-MIMO gain even without using them. OFDMA; don't really care at this point. From everything I've seen, it's relevant only with many more STAs than I'm testing with.

Flent's main contribution was that it made traffic + ping testing easy. I'm doing essentially the same thing with octoScope's multiperf, using iperf3 TCP/IP traffic and ping.

The AX200 and now AX210 are pretty much it when it comes to M.2 format STAs. Or at least they were at the point where the STApals were being developed. From what I can see, Qualcomm has chosen not to pursue that market.

 

[mccanntech]

Tim,

Just want to say thanks for everything. I'm so glad you're back to reviewing and writing more. I know it's a ton of work to do these tests and produce these articles. From the outside at least, it all seems well worth it. I've been coming to SNB for years. I know for me and many others SNB is an invaluable resource, and an example of how testing should be done. I appreciate your attention to detail, and I'm wildly jealous of the gear you get to use.

Thanks for all the work it takes to make that happen, and for detailing out the benchmarks and testing methods you're using. It's been a while since I had eyes on the revision 10 method, and this gave me a few ideas for how I can improve my own. All I need is to find a friendly octoScope rep...

 

[thiggins]

Thanks for the kind words @mccanntech. The restart has taken much more time than I had expected. Websites get crusty when ignored for a few years and I had a lot of patching to do behind the scenes to get the Charts back in shape.

The octoScope stack *is* powerful and much more than I need for my relatively simple testing. But the plus side is I've gotten pretty good at python.

Take a look at Jim Salter's network testing tools. This blog post provides an overview. You can see it in action in the 2x2 AP roundup he did for me a few years ago.

2x2 AC Access Point Roundup - Part 2 - SmallNetBuilder

Updated - Jim Salter takes another run at our collection of 2x2 AC APs, this time testing multi-client load handling in open air, with roaming too!

www.smallnetbuilder.com

You have to bring your own clients, but it's pretty powerful.

It's a shame he isn't reviewing Wi-Fi stuff any more.

 

[mccanntech]

Thanks I'll check that out, and agreed. I follow Jim's work pretty closely, and still refer people to his guide on access point placement. I wish Ars would assign more Wi-Fi reviews to him.

There's far too many people trying to promote a YouTube channel, and not enough people who understand beyond the surface level of single-client speed tests and marketing hype. I get frustrated trying to search for accurate information, let alone nuanced and in-depth reviews.

There's a lot of room between white papers and "like-and-subscribe" journalism, and not enough sites like SNB filling the gap.

 

[thiggins]

Unfortunately, you get what you pay for with content that's paid for by advertising. The only way to make a decent living doing reviews is to be an "influencer" on one of the social media platforms. Fortunately, I never had to do that. But it's the reason you don't find many sites like SNB.

 

[LanyardNL]

Tim,

First of all : your website is amazing, and it's a blessing to have such a thing available on the web. Thank you so much!

Noticed that in your new setup you test WAN -> WLAN traffic, given the increasing bandwith of wireless standards ofcourse the most reasonable thing to do since many devices only have a WAN port that supports speeds higher than 1Gbit. However, in an earlier draft you stated that for WAN-LAN testing 'wireless won't work'. How did you set this up in the end to make it work out?

I also noticed that you're configuring the router for 2.4GHz at 40MHz channel bandwith, while in earlier drafts this was 20MHz. The client seems to still be configured for 20MHz. Is there a reason for this change?

 

[thiggins]

@LanyardNL Thanks for your comments and questions.

If you can point me to the "Earlier draft" where I said wireless won't work with WAN-LAN testing, I'll correct it. I worked with octoScope to develop the technique for testing through the router. It's not as easy as it should be, but it works.

I set the router to 40 MHz and control channel bandwidth on the STA. I still do RvR with 20 MHz bandwidth, but use 40 MHz for the Multiband test.

 

[LanyardNL]

@thiggins When I said 'earlier' I actually meant waaay back haha. This is the page is was referring to. I was recently doing some tests of my own, and couldn't get any communications from the WAN side to the WLAN PC, while this worked fine when using a physical LAN port. Since there is not a lot of information on this subject, I simply figured your 'wireless won't work' comment in this ancient document meant there was some fundemental reason this isn't possible.

 

[thiggins]

Yikes that's an oldie! Since that comment was in the Wired routing test section, I just mean using a wireless connection to test wired throughput wouldn't work.

If you follow the steps for the wired connection, but use a wireless client (STA) on the LAN side, things should work. Use iperf3 and put the client on the STA and server on the WAN side machine. You can run an uplink test without having to forward ports.

If you want to run downlink, you'll need to forward the default iper3 port 5201 to the STA's IP address, run the the iperf3 client on the WAN machine and use the router WAN IP address as the -c parameter.

 

[sfx2000]

Our Revision 11 process

Having seen the first formal review based on the process - nice work!

 

[thiggins]

Having seen the first formal review based on the process - nice work!

Thanks, @sfx2000 . As I said, getting the list of potential benchmarks winnowed down was tough. You should see the tests left on the cutting room floor.

I originally was going to do RvR with three STAs, each at different DSCP priority, best effort, voice and video. I got some pretty interesting results with that one.

 

[RMerlin]

You should see the tests left on the cutting room floor.

SNB: The Snyder Cut, coming up soon?

 

[thiggins]

SNB: The Snyder Cut, coming up soon?