Looking for Wireless Stress Test Cases

[thiggins]

I'm in the process of exploring expanding SNB's Wi-Fi testing and need some feedback.
The testing would involve multiple clients of different types running a mix of different application traffic
that represent typical home uses.

I know what happens in my household, but it's not very stressful. So I'd like some tougher use cases.

So here's what I'd like from you:

1) How many wireless devices and what types are connected to your home network?
2) Which of those devices are in simultaneous use, worst case?
3) What are those devices doing (apps in use simultaneously, worst case)?

Here's an example of what I'm looking for in the desired format.
(* = device in simultaneous use)

*, 1x1 AC , tablet , Netflix
*, 1x1 N, phone , Facebook
, 2x2 AC, notebook , Web, Email
, 2x2 N, notebook , Skype, Pandora
, 1x1 N, notebook, Hulu, Web
* , 1x1 AC, phone , FaceTime
, 3x3 N, notebook , Torrenting
*, 1x1 N, Chromecast, YouTube
* ,1x1 N, Roku, Amazon Prime Video

You can write a story if you like to accompany the above. But PLEASE also include the info in the requested format. You can identify exact devices if you like, i.e. iPhone 6s, Roku 3, etc. but it's not necessary. The more important information is the Wi-Fi type.

You can attach spreadsheets if that's more convenient.

Thanks in advance for the help.

Tim

 

[sfx2000]

PM me - fair amount of experience here

 

[System Error Message]

while i dont use that many wifi devices as im still by myself there are a few stress test cases you can organise for example
Mixing N and AC on the same band
Mixing single channel and triple channel wifi clients on the same AP
pushing packets from wifi to wire and vice versa or from one radio to another in the case of the AC3200 and measuring what it is capable of
Finding practical bandwidths by measuring common applications such as file transfers and direct file video streaming
Stress testing by filling the router with as much throughput as possible and keeping the CPU maxed out. In the case of a typical AC1900 router this would be 2Gb/s between WAN and LAN and 1900Mb/s between WLAN and LAN. If the router is good it would not freeze or stop forwarding anymore packets or shut itself off.

In my opinion the type of clients used doesnt matter, only some hardware stuff (wifi protocol, channels, count) with some software (bandwidth testers, stress testers, common applications). Seeing how a wifi AP/router copes under enormous stress is important since it tells how often you would have to restart it or how well it works under strain. Any wifi router would work well under normal use even the shoddy ones that ISPs give but what really matters is if it can deliver what it is meant for so if it is an AC1900 router with wirespeed NAT than it should be able to deliver that which none actually can.

Im just waiting for my 10Gb/s NICs to arrive so that i can stress test really fast routers. It'd probably at the end of October since they are imported so i should than hopefully be able to test more than 40Gb/s of forwarding throughput if i can get at least 4 systems up and running. I already found either 2 motherboards/CPUs not working already and they are lga 1366 boards + xeons. Wasnt an issue with RAM because i have plenty different sizes and frequencies.

 

[azazel1024]

Current setup is 3 base stations.

N600 Channel 11 and Channel 149+153.

AC1200 Channel 1+5 and Channel 36+40+44+48

AC2600 Channel 11 and Channel 149+153+157+161.

Facing the front of my house from the outside, the N600 is on the left in the garage with the antennas run to the outside rear of the garage. The AC1200 is on the left side of the interior of the house in the center on the main level. The AC2600 is in the basement on the far right in the front corner of the house high up near the ceiling.

The house is typical drywall and 2x4 stud construction, but I do have a 4ft thick masonry fireplace right next to that AC1200 base station, but the fireplace doesn't obstruct the LoS path for most of the main level from the AC1200 (it used to until I moved the base station). The house is 65ft wide, 24ft deep rancher, plus a 25ft wide garage, 1 acre property with the house pretty much dead center on it.

For clients I have

iPad (iOS7) 2 1:1N dual band
2xAsus Memo Pad HD7 (android 4.4) 1:1N dual band
iPhone 6 (iOS9) 1:1AC dual band
iPhone 6s (iOS9) 2:2AC dual band
Asus T100 (windows 10) 1:1N dual band
HP Envy 4t with Intel 7260ac (windows 8.1, soon to be 10) 2:2AC dual band
Amazon Fire HD stick 1:1N dual band

I also have a wireless printer, airport express (First gen N) and AppleTV 3, but all of them are wired and not wireless currently.

This does not include a few 2.4GHz devices (a couple of Bluetooth as well as a wireless mouse on 2.4GHz). House phone is DAC.

A fairly typical heavy use case would be the Fire TV stick streaming Netflix/Amazon prime, both my sons on their Memo pads playing Minecraft and my daughter on the iPad 2 also playing minecraft (WLAN hosted game), my wife on her phone browsing the internet/facebook and me possibly downloading a game/file/browsing the internet on the laptop or my tablet. Frequently this is all on the AC1200 base station, occasionally you might find 1-3 of the users connected to the AC2600, the others on the AC1200 or every once in awhile it might be someone outside on the N600 and the others on the AC1200 or AC2600.

This is for the heavy use cases.

More typical would probably be the fire TV stick streaming and 1-2 users either streaming on one of the mobile devices or browsing the internet.

From a lot of testing, 90% of the time 90% of the devices are connected on 5GHz.

If you want to throw in routing testing, it isn't that uncommon that I'll be downloading a large file on my desktop or server at the same time there are 1+ devices streaming video and someone else browsing the internet all at the same time (symmetrical 75Mbps connection run through the AC2600 router).

At some point when I take my lazy pants off, the AC1200 will be replaced with the Archer C8 I have laying around and then the AC1200 (C5) might be moved to replace the N600 (WDR3600)).

 

[thiggins]

I appreciate the detailed responses. But I asked for stories OPTIONALLY to accompany the simple table I requested.

It will really help if you boil it down.

Thanks

 

[azazel1024]

*, 1:1n tablet, online gaming, Netflix
*, 1:1n tablet, online gaming, Netflix
*, 1:1n tablet, online gaming, Netflix
*, 1:1n streamer, Netflix, Amazon Prime
*, 1:1ac phone, web, facetime, streaming audio
*, 2:2ac laptop, downloading large files, web, online gaming
, 2:2ac phone, web, facetime, streaming audio
,1:1n tablet, Netflix, online gaming, downloading large files

That would be my setup for a worst case. 8 devices, 5 users. Worst case is 6 devices simultaneously, most of them streaming video or online gaming with at least one large file download or upload occurring (not bittorrent, SFTP/HTTP). Could be multiple activities at once on some of those devices (like stream audio and web browsing at once, or Netflix and downloading a large file).

 

[System Error Message]

Although you may not likely test this but my reasoning for pushing the limits on testing is because in the future we will have compute programs and i am also building some for use over multiple systems over the network. You could try out p2p based multiplayer gaming on wifi as well that are bandwidth heavy. Some games on LAN will act differently than if it were an internet session. Cluster based rendering would be an interesting test since manufacturers are trying to beat gigabit ethernet with wifi.

If we dont test devices to the limit than when we will never know how reliable and dependable they are compared to x86 based solutions. In the past whenever you pushed a router to the limit it will always freeze and you would have to restart it and this is still possible even on expensive routers.

 

[thiggins]

). Could be multiple activities at once on some of those devices (like stream audio and web browsing at once, or Netflix and downloading a large file).

Thanks fir the table.

With limited 1x1 bandwidth, why would a user purposely degrade video performance by simultaneously downloading a large file?

 

[System Error Message]

Thanks fir the table.

With limited 1x1 bandwidth, why would a user purposely degrade video performance by simultaneously downloading a large file?

You shouldnt downgrade video performance rather you should be able to stream normally and download in the background. Part of this is related to client QoS setting rather than the router itself as this is usually on layer 2 but on layer 2 you can apply priority based QoS in which recent switches and layer 2 bridges should know already however both the client and/or the server must apply this although technically you could apply this on bridges as i do on mikrotik.

Think of it as downloading a game while watching a movie.

 

[stevech]

Thanks fir the table.

With limited 1x1 bandwidth, why would a user purposely degrade video performance by simultaneously downloading a large file?

Family.
Kids.
or
fellow students in dorm or frat house.

Bandwidth allocation fairness is not the feeble QoS in consumer products.

 

[sfx2000]

The big challenge is how to characterize and then model that traffic into something that is consistently repeatable given known endpoints vs. something in the middle that can affect the model...

In 802.11 - we have four basic traffic types - Best Effort, Background, Video, and Voice.

Applications can tag those frames with these QoS levels, otherwise they all become Best Effort, and the frames are intermixed as they are delivered from the source.

Very few applications mentioned so far do any kind of tagging to trigger QoS scheduling (this might change in the world of 802.11ac Wave 2 with MU-MIMO), so it's very hard to even try to model this traffic in an objective context...

So a lot of this becomes subjective - observations based on end-point performance - e.g. Qualcomm Snapdragon Quad Core on Android vs. x86 on windows with some application (just using this as an example) - and there, since everything is best effort, it comes down to client performance, and not the actual AP sending those packets over...

Some AP's attempt to do some packet inspection based on source addr and perhaps watermarks based on amount of traffic to/from source to sink, but again, this is pretty primitive if everything is best effort...

So the challenge is to get something that can be scripted and consistent - all the recommendations are good so far, but if we're measuring the Router/AP under test, then the external variables must be controlled.

 

[azazel1024]

Thanks fir the table.

With limited 1x1 bandwidth, why would a user purposely degrade video performance by simultaneously downloading a large file?

I've generally noticed it doesn't. The 1:1n tablet mentioned is an Asus T100 running Windows 10 (formerly 8.1). It isn't the fastest 1:1 solution out there, but it'll regularly hit 80Mbps on 5GHz 40MHz and 56Mbps 2.4GHz 20MHz. Netflix streaming typically takes 6-8Mbps of that and whether it is the windows net stack, the router in question or at the application level, it manages it well.

I've never seen a quality reduction if streaming and I start a large file download on my tablet. Not a common occurrence, but I certainly do it from time to time.

To add, even when I see several other people streaming video, web, email, facebook and online gaming mixed in, I can generally get an easy 40-50Mbps of bandwidth to my tablet still with a bit shaved off for Netflix sometimes.

Of course hopefully more manufacturers will move to using at least 11ac solutions and hopefully more 2:2. Also different interface, my 11n in my tablet is SDIO, so hard cap of 100Mbps bus, which is why the best it can see is about 90Mbps on 2.4GHz 40MHz, that 100Mbps minus internal protocol overheads and TCP/IP. HSI (effectively is USB2, so 480mbps, minus overheads down to about 350-380Mbps max) or better PCIe would be nice.

 

[sfx2000]

Note - Netflix has a test channel - "Example Short 23.976" that shows bit rates, and this can be very handy in the real world when planning and validating your home network.

 

[thiggins]

Thanks guys.

Ok, folks, we've heard from the usual gang. How about you lurkers out there? Here's your chance to influence how we test.

I have only one usable test case from Azazel so far....

 

[sfx2000]

Thanks fir the table.

With limited 1x1 bandwidth, why would a user purposely degrade video performance by simultaneously downloading a large file?

This is a good test of application flow management across both the router SoC and the priority assignments in the WiFi chipset...

 

[sfx2000]

*, 1:1n tablet, online gaming, Netflix
*, 1:1n tablet, online gaming, Netflix
*, 1:1n tablet, online gaming, Netflix
*, 1:1n streamer, Netflix, Amazon Prime
*, 1:1ac phone, web, facetime, streaming audio
*, 2:2ac laptop, downloading large files, web, online gaming
, 2:2ac phone, web, facetime, streaming audio
,1:1n tablet, Netflix, online gaming, downloading large files

That would be my setup for a worst case. 8 devices, 5 users. Worst case is 6 devices simultaneously, most of them streaming video or online gaming with at least one large file download or upload occurring (not bittorrent, SFTP/HTTP). Could be multiple activities at once on some of those devices (like stream audio and web browsing at once, or Netflix and downloading a large file).

These are all pretty good - I would add for the streaming audio test cases - stream from a device to the AP - e.g. AirPlay streaming, as this is a common thread of interest/concern for many in the forums.

And same would go for Video - use case here is the WiFi security camera (maybe use two, as this does related back to some of the articles on the main site)

Comment about Netflix - they use an adaptive rate codec, so it's hard to tell at what bitrate the content is being delivered at - however, one can have some reference video streams and use VLC to serve and playback the content there.

With regards to Voice Testing - both subjective tests using real world clients (e.g. Skype as an example) as well as objective test cases using deterministic traffic flows - Harvard Sentences is the golden bench here for codec evaluation - interesting to note that with Voice calls - they tend to follow Markov distributions across different frame sizes within most common codecs, even the newer "high definition" voice modes.

Since the Wireless Operators are now launching Voice of WiFI calling, this is an area where different AP's may be challenged.

ixChariot may actually have modeled traffic scenarios - as the Wireless Operators have been pushing hard here on device/AP testing due to the mobile convergence. If Chariot has these, it's a big plus, as now testing can be scripted/automated across these different test scenarios for consistent and reproducible results on both the test bench (RF chamber) as well as Free Space testing at the different test locations.

For real world load testing - something to consider is that NPD did a reference survey and found that there are roughly 5.7 clients per person - so taking the average 4 person household - that's roughly 23 clients - this includes laptops, phones/tablets, media streams/game boxes - which is concurrent with my real world experience - e.g. I consider three screens/clients per person, plus 5 to account for the non-user dedicated devices - and in the house, I would suggest that the client mix is probably 40 percent portables (phone/tablet) to 40 percent fixed devices (laptops), and the remainder being the Roku's, AppleTV's, XBox/PS4 type of clients.

When looking at the client mix from a 802.11 context - I would suggest that about half are 1*1:1 config (single or dual band), 40 percent are 2*2:2, and the remaining 10 percent might be 3*3:3 at this point in time (Q4-2015).

 

[azazel1024]

Actually that streaming audio is exactly that use case. Airplay from Apple "radio" on iPhone 6 to access point and then over the wire to the airport extreme.

I'd add Airplay screen mirroring, but that is an almost never occasion (2, 3 times a year? to my Apple TV).

 

[azazel1024]

Jesus, that many? Has to be including typical non-owned wireless devices.

Maybe with the exception of my brother-in-law, I have more wireless devices in my household than anyone I know. Granted, the number will likely only increase as my kids get older, so my numbers are probably diluted by young children.

I have 8 on that list, with another 4 that are used wired only (Xbox One, Wii, network printer and Apple TV). If you include base stations, that would be another 4 (with 1 currently used wired only for airplay audio).

Though, depends on how you define clients here. I also have a wireless mouse (which isn't used often), my wife a BT exercise band, Xbox One controllers are Wifi direct and the remote for my FireTV stick is 2.4GHz wireless (but not Wifi).

So you wanted to expand the definition, I could have 21 devices, but 5 are strictly used as wired only and several of them are not always in use/powered.

 

[sfx2000]

The device mix is changing - we're going from Desktops/Laptops into a more Mobile Environment, and this changes the game for the AP vendors...

 

[bman212121]

I don't have a specific scenario that involves multiple devices trying to use wifi because I hardwire everything, but there are a couple cases where even a single client performance can be not optimal. (Thus I have to plug in for it to work)

2x2 N, Laptop, Lightroom
2X2 N, Laptop, WMC recordings

To detail what I mean about lightroom, I keep the pictures on a storage server. The first step is importing photos from a flash card to the server. This requires copying hundreds of 20MB raw files from a laptop to storage unit on the network. Once the files have been copied over, it need to import those into the catalog, then create thumbnails and add metadata. The catalog is on the client, so it takes some time to run through this process. After that is done each time you want to edit a picture, it has to download the picture into memory so it can work with it. All of this goes smoothly on a wired connection, it's a difficult experience on N wifi because even trying to flip between photos there can be a significant delay if you want to view them.

The other scenario is shows are recorded from a network tuner (HDHR Prime) to a WMC server. These are 20mbit streams recorded in mpeg2. If you watch the recording on another device you need to maintain at least 20mbps to not have buffering. This can be difficult to do with N if you don't have a high signal strength. I think if you use DLNA streaming for live TV with a prime to a device this would also occur.