Open Mesh OM5P-AC Dual Band 1.17 Gbps Access Point Reviewed

[thiggins]

Open Mesh's OM5P-AC is not the way to go for an inexpensive AC1200 access point.

Read on SmallNetBuilder

 

[RAVI RADHAKRISHNAN]

why the switch chip named as Atheros AR8033 anymore, shouldnt it be QCA ?

 

[thiggins]

AR is Atheros. All are manufactured by Qualcomm now.

 

[GregD]

Fun article, big thanks to Jim for writing this up.

I installed 8 of these AP's in a very large home last year but I never got to perform as much testing as I wanted to. The client hasn't had any complaints but I'm surprised to see such poor 2.4ghz performance. I designed the network for 5ghz so it doesn't matter much (and is probably why there haven't been any complaints) but I'm trying to confirm the maximum single STA throughput on 2.4ghz that Jim saw was only 5.7mbps?? That seems hard to believe even for a cheap little AP like the OM5P-AC. If that's the speed at location D, what was the throughput at location A (worst location)?

I'm interested to know if Jim was using OpenMesh's recommended settings for maximum performance since options like DPI and DNS intercept add additional overhead that affect performance of their cheaper AP's disproportionately.

I'd also like to know if Jim has access to MetaGeek's spectrum analyzer to see what the 2.4ghz spectrum baseline looked like at the time of the testing. Is this testing being done in an urban/suburban house or a rural home?

Lastly, I'm confused by some of the graphics in the article. The article mentions that the single STA best location (location D) performance on 2.4ghz is 5.7mbps yet the stacked throughput comparison chart seems to show location D at somewhere around 25mbps. It seems to me like 5.7mbps would make sense for the location A (worst location). See graphics below.

Single STA Throughput:

Single STA Stacked Throughput:

 

[sfx2000]

I installed 8 of these AP's in a very large home last year but I never got to perform as much testing as I wanted to. The client hasn't had any complaints but I'm surprised to see such poor 2.4ghz performance. I designed the network for 5ghz so it doesn't matter much (and is probably why there haven't been any complaints) but I'm trying to confirm the maximum single STA throughput on 2.4ghz that Jim saw was only 5.7mbps?? That seems hard to believe even for a cheap little AP like the OM5P-AC. If that's the speed at location D, what was the throughput at location A (worst location)?

I found the numbers to be a bit odd... I'm wondering if this is a Client STA issue...

Jim used similar methodology for the 2*2 dedicated AP roundup - and this is indicated for the OpenMesh review...

Going into the 2*2 Roundup - here's his description of the client STA setup...

The stations themselves are four identical Samsung Chromebooks running GalliumOS equipped with a built-in Intel Dual Band Wireless-AC 7265 and Linksys WUSB-6300 external wireless adapters. The WUSB-6300 is my reference adapter and used for most performance testing, because there's less variation between the four NICs than there is between the internal NICs in the Chromebooks, and also because they have exceptional TX performance.

The current Linux driver for the WUSB-6300 does not support 802.11k or v, though, so we shift back to the onboard Intel 7265 for testing of roaming and band steering behavior. I have not been able to find definitive information from Intel regarding the technologies supported in their Linux driver. However, Microsoft says Windows 10 supports 802.11k,v and r and Intel says the Wireless-AC 7265 (Rev. D) also supports all three roaming assistance standards. Empirically, it is very clear that AP-assisted roaming works on the AC 7265 under Linux, due to the dramatic differences in behavior when connected to different models of AP.
​

The concern here is that GalliumOS and the driver support there for both the Intel 7265 (which I've found works well on ChromeOS, but not very well on Ubuntu) and the Realtek based WUSB-6300 uses a driver that has always been a bit sketchy in their linux support...

I'd like to see some numbers with something more relevant - MacBook Air 13 is a great reference device for assessing 2.4 and 5GHz performance due to a solid OS driver and very good antenna design - it's a 2-stream device, if one wants three streams, look at the MacBook Pro line. For Windows, the Dell XPS 13 is a good choice, and ThinkPad T-series is a great exemplar for the corporate space.

For handheld/mobiles - iPhone 6s is a good start, and with Android, consider the Galaxy S8 - for a couple of reasons - first, is that they perform well with WiFi in both bands, and second, they're strongly representative of the current market.

@Jim Salter and @thiggins

Might be good to do a part II retest with something other than a hacked/jailbroken Chromebook with sketchy drivers that do not represent a significant portion of the fielded equipment. Or more clearly define the client station config and back it up on the SNB test chamber.

I always appreciate SNB's editorial independence, and Jim's insight is good, but there's something wrong here, hopefully OpenMesh has a chance to review and respond.

 

[Jim Salter]

I'll pull the raw data back up later today to address some of these concerns. This was not a hasty one and done test; I had a thirty-AP project relying on the outcome.

Sfx, I'm not sure where you get the idea that Intel 7265 driver support is poor in Ubuntu... Or why you think it would be any different in chromeos. It's an in kernel driver, and will usually be the exact same code in either OS - only differences potentially being newer kernel versions on the normal Linux side versus the chromeos linux side.

I have not found the Intel 7265 in these Chromebooks particularly to be much different in performance from Intel 7265 in other brands of laptop. There's nothing special about a Chromebook other than it being cheap. That's why I use them. If you'd like to buy me four MacBooks, I'll be happy to use them as test stations. At least sometimes.

The reason I use those wusb6300 usb3 nics for the majority of the testing is two fold. One, I can ensure an identical chipset and antenna layout even if the model of station needs to change at some point. Two, they outperformed every other 2x2:2 NIC I had access to - INCLUDING onboard Intel NICs in multiple models of laptop. Sure, I'd love for there to be somewhat better driver support; but it's not that bad; there is an rtl8812au-dkms package in the main Ubuntu repository now.

My other choices are to use Windows subsystem for Linux - which can be done with reasonable results, assuming minimum of Fall 2017 Creator's Update. I needed to do that to test one of those KillerAC nics in a gaming laptop once. (Spoiler: it was not particularly killer.) But it doesn't actually gain me anything, so why would I put myself through that?

Mobile testing is even worse. I'm a sysadmin, not an Android developer, let alone an iPhone developer. It's awkward enough trying to run a simple iperf3 test from a phone, let alone get a real scripting platform and run custom tests.

Anyway I'll come back later today with answers about the raw numbers; that 2.4 GHz made me shake my head several times too.

 

[sfx2000]

Anyway I'll come back later today with answers about the raw numbers; that 2.4 GHz made me shake my head several times too.

That's the one thing we can all agree on

Has anybody reached out to OpenMesh for comment?

 

[Jim Salter]

That's the one thing we can all agree on

Has anybody reached out to OpenMesh for comment?

I haven't talked to OpenMesh about the OM5P-AC, no.

For 2.4 GHz single-client (HTTP 1MB repeated download), the raw numbers are 26.3 Mbps for STA D, and 13.2 Mbps for STA A.

This matches up with what you see on the graph; note the light blue (2.4 GHz STA D) is about triple the size of the light magenta (5 GHz STA D), and the dark blue (2.4 GHz STA A) is about double the size of the light blue.

I don't really have numbers to show you for it, but I doubted myself hard on that 23.6 Mbps top speed 2.4 GHz at close range (around 10 feet) and checked it up, down, left, right, and sideways before accepting the run as okay.

Here are the raw numbers for multi-client on 2.4 GHz.

Re earlier questions about tuning: OpenMesh doesn't really lend itself to tuning individual APs. The same settings were in use as were in use when testing the OpenMesh A60s earlier this year - like literally; I just opened up the same Cloudtrax account and adopted the OMP5-AC and got to work. There wasn't much tuning done on the A60 either - I try to test in as close to out of the box default configuration as possible in the vast majority of cases, as my goal is generally not to go "hey look how amazingly I improved this thing twiddling with tons of knobs for hours", it's more "this is how well anyone can expect this device to work."

I am a big, big believer in sane out-of-box configurations... and if a vendor wants to do well in my testing, they'd better ship with a sane config.

 

[GregD]

For 2.4 GHz single-client (HTTP 1MB repeated download), the raw numbers are 26.3 Mbps for STA D, and 13.2 Mbps for STA A.

Jim, thanks for digging into this for us.

Does this mean the article should be edited to show that the OMP5-AC has a maximum throughput of 26.3Mbps not 5.7 Mbps at location D? This is how the article reads to me at least.

Based on all the info on hand, I also don't see where 5.7Mbps came from except perhaps from the 5Ghz speed tests at location A? Could you post the raw numbers for 5Ghz as well?

I am a big, big believer in sane out-of-box configurations... and if a vendor wants to do well in my testing, they'd better ship with a sane config.

Totally agree when it comes to home routers but OpenMesh is more business-oriented IMHO. As a result, some might consider shipping with Application Reporting (DPI) on by default as a "sane config" while others might consider shipping with that feature turned off to improve throughput speeds to be the "sane config" - it all depends on what you value more as an administrator.

Cloudtrax is very simple to use and checking/changing these global settings takes all of 2 minutes which is why I posed the question. As you said, you don't tweak OpenMesh AP's individually per se and that's expected when you're managing environments with many AP's. The Cloudtrax options that I asked to check are applied globally to all the AP's in the network; by tuning the options in Cloudtrax you're tuning the options on the AP.

I'm pretty sure Cloudtrax default network config has DPI turned off, but I can't confirm if some of the other options like DNS intercept, bridge to lan, etc. are configured out of the box for maximum throughput; even if the defaults are set out of the box for maximum throughput there is always a chance some option got changed by accident during initial configuration. If AP firmware or settings aren't the cause of the poor 2.4ghz performance, it may be a client driver issue as others have suggested.

 

[Jim Salter]

Jim, thanks for digging into this for us.

Does this mean the article should be edited to show that the OMP5-AC has a maximum throughput of 26.3Mbps not 5.7 Mbps at location D?

Yes. The conclusion as stated is correct, but the number is wrong. It should read:

So when we see an abysmal 23.6 Mbps of traffic to Station D on 2.4 GHz here, this really is the death knell for the little OM5P-AC's 2.4 GHz performance.

Based on all the info on hand, I also don't see where 5.7Mbps came from except perhaps from the 5Ghz speed tests at location A? Could you post the raw numbers for 5Ghz as well?

That's exactly where it came from. 5.7 Mbps is the value for 5 GHz / A; the charts are correct.

Totally agree when it comes to home routers but OpenMesh is more business-oriented IMHO. As a result, some might consider shipping with Application Reporting (DPI) on by default as a "sane config" while others might consider shipping with that feature turned off to improve throughput speeds to be the "sane config" - it all depends on what you value more as an administrator.

I hope we can both agree that there's nothing "sane" about speeds this abysmal from an access point at close range. If we hypothesize that their A40/A60 line can handle those settings and get decent performance but the OM5P-AC chokes and dies on them, then "sanity" would be handling the setting defaults differently for the OM5P-AC line - or flagging the administrator.

Anyway, I checked the settings you're asking about. DPI is off; throttling is off; the SSID was bridged to the LAN (that I knew about; it needed to be for testing); DNS intercept was on.

It's completely unreasonable to blame DNS intercept for poor short-range 2.4 GHz performance when the short-range 5 GHz performance was perfectly cromulent, though. 140.9 Mbps at STA D on 5 GHz. The OM5P-AC was clearly not suffering from CPU issues to cause that poor showing on 2.4 GHz.

 

[sfx2000]

I hope we can both agree that there's nothing "sane" about speeds this abysmal from an access point at close range

I agree here - and this does look like a firmware issue at the QCA level with the 9557 perhaps - someone at OpenMesh should look into the version and config items... wouldn't be the first time something like this has happened with ath9k - so I'm ruling out the iwlwifi and rt* drivers for now, since performance is similar.

If you have time, you can look at the MCS rates in the wireless PCAP - I'm thinking they're pretty low - minstrel is working, but it needs good data from the baseband.

Hard to tell what's going on there without PCAP's and debug on the AP, but I've seen things like this before - that's why I've repeatedly asked about the ongoing conversation with the team there at OpenMesh - based on other experience here in the forums, they do respond... Don't get me wrong - I'm a systems design/product development engineer with over 20 years in the wireless industry at the OEM/ODM/Carrier and Standards level...

I agree there is an anomaly here with the results. I do take some exception with the terms "death knell" and "abysmal" in a public review without have had a chance to talk with the OEM - if they don't respond, then fine, let it ride, but give them a chance... if they care about their product, they'll fix it.

 

[sfx2000]

It's completely unreasonable to blame DNS intercept for poor short-range 2.4 GHz performance when the short-range 5 GHz performance was perfectly cromulent, though. 140.9 Mbps at STA D on 5 GHz. The OM5P-AC was clearly not suffering from CPU issues to cause that poor showing on 2.4 GHz.

Yep... hence the emphasis on the vendor outreach - something's wrong with the driver IMHO...

 

[thiggins]

Review has been updated with the correct 2.4 GHz throughput. Sorry for the error.

 

[Jim Salter]

I apologize for the error too, since I'm pretty sure it was mine in the first place! =)

 

[GregD]

Thanks again for the comprehensive replies.

Since I'm obsessive, I dug through the firmware release notes and discovered this:

It's important to note that the latest stable release for the OMP5-AC is 6.3.16 and that when using Cloudtrax's defaults, it will not install beta firmware. Below I'm showing the menu option in Cloudtrax that allows you to install the latest beta firmware which is now 6.4.8.

Any interest in trying to update the firmware and see if it helps at all? I know this sort of takes us deeper into that realm of tuning knobs for hours just to make things work correctly...

Anyway, this may not end up doing anything for the single STA throughput numbers but I wondered if it would help improve the multi-client latency test results...