Problem with Apple WiFi stability

[thorel]

Back in January I helped a friend set up a larger single SSID wireless network with 6 wireless nodes. It's an all Apple network (computers too) consisting mostly of a roaming (Apple's term for units set up as a bridge) configuration with a Time Capsule and Airport Extreme on one floor and two Airport Expresses on another floor with channel 1 on the first floor and 11 on the second. Elements on the same floor are far enough apart electrically to not interfere. I added one Airport Express to extend the signal (wireless repeater) in one area of the first floor and used another as a printer connection node. Setting all this up was easy with the latest software and all units had firmware brought current as well.

All wired connections (including the two Macs), go back through a single switch layer to a firewall/router at the root of the network which also is the only in network DHCP source.

No nearby WiFi networks exist except a very weak signal on channel 6.

Now the problem, about once a week there is a cluster of failures in sourcing the WiFi signal. Each unit fails in isolation of all others (not simultaneous or even overlapping) and the cascade can take place over up to 3 days (or all in one day, usually 2 days). Multiple days to a week or more may pass between failure clusters. The sequence of fails is random (no specific source unit). Each failure symptom is that a base station signal vanishes and bringing it back requires a power cycle. A simple power cycle for the failed unit always brings it back to life with a clean connection and configuration. None of the units have required a factory reset and reconfiguration, it just falls over for no apparent reason and wakes up again working properly, (no fail logs though, so no idea what the hardware is saying). The wired portion of the connection remains live, just the the wireless signal stops.

I had a look at the power lines along with the power company and found no issues with high levels of noise any different from intervals when failures don't occur.

I could really use some pointers!

Thanks,
Tim

 

[overdrive31]

Many routers have wireless woes, try setting them all up on a light timer to reboot each day. It may not be the answer you're looking for, but it is a simple solution. What does Apple have to say about the problem?

 

[stevech]

Try eliminating the repeater. At least temporarily, by using cat5 cable on the floor or semi-permanently via a power line IP bridge, or a MoCA bridge. WiFi repeaters (WDS) are notoriously unreliable.

In my experience, most WiFi routers from reputable companies (that excludes Netgear and D-Link), with a few exceptions, run 24/7/365. The ones I choose for home and professional use do so. But my experiences don't include Apple products.

 

[thorel]

The timed reboot may be worthwhile pursuing, but I'll reserve that for a last resort.

Unfortunately Apple and the Apple user forums have been less than useful on this. Most have no idea what the set up really does nor how to debug it...

Thanks though.

 

[thorel]

We tried eliminating the repeater, in fact they attempted to live without one for the first month. The remaining WAPs had the same failure signatures any way.

I'm beginning to think it's an Apple specific software issue and that the ultimate solution is to use a more robust set of hardware. Unfortunately there's been no help from Apple on this...

 

[wfines]

Having trouble understanding one part of your post: You said they fail independently, with no overlap. But you mention that there is a "cascade" failure. When one unit goes down, you're saying that within 2 days, all the others will have failed? I don't THINK this is what you're saying, I'm just trying to understand what takes place in "usually 2 days".

Rereading it, maybe you meant that it's totally stable during the week, but then over a course of usually 2 days, they fail at a more frequent rate? IE: Days 1-6, no issues, then day 7, you have 4 failures over the course of 1-3 days?

Is there ANY correlation with which unit is failing most? Are you saying the Time Machine, Extreme, and Express' are all failing at roughly the same rate? This would surprise me quite a bit.

 

[thorel]

By independently, I mean than a given unit might be the only fail at a given time, all others working properly. Then power cycled, the network is good again, then another (random) unit fails at some random time interval later (bimodal long or short depending on inter or intra cluster interval).

By cascade, I mean that the isolated (as above) failures do seem grouped in that (usually) the entire set of units will need to be reset within the course of one to three days, then nothing for ~5 days, then starting to fail again. This is why I suspected a noise source, but found no evident triggers.

That next failure sequence of units being in a new random order.

Within the sample size I have, all unit types are failing at about the same frequency. Nothing sticks out as significantly lower, but the repeater failures may be (very) slightly higher. You are correct in that this is counter intuitive, but it is the data...

 

[wfines]

EDIT: Still not really sinking in what might be going on. Going to sleep on it. Does, after all, seem like a firmware bug of some sort. I doubt configurations beyond 2-3 Apple WAP units is tested very often.

 

[sfx2000]

Without getting too deep into this, can you spend some time to graphically map out the current layout and connectivity?

WDS can collapse in a heartbeat if too much traffic...

I've got a bit of experience at dealing with Airport and the various iterations... but having the layout would be helpful in making the right recommendations.

 

[wfines]

Yes, having slept on it, please provide an exact layout. I'm not super clear on the network design from your posts.

Is there a lower bound for when they stop failing? When one unit bites dust, have you ever tried leaving that unit off, and seeing at which point the other units stop failing? It seems that if it were a firmware glitch, it would fail down to the lower bound (presumably 2 AP's can safely be considered a very standard, highly tested configuration aka lower bound).

I'd wager that if the repeater fails, then another unit fails... the network would be stable. Or you'd fail down to, say, 3 WAP's, then it would stabilize. Verifying this would help eliminate environmental interference. Perhaps this is already obvious to you, but for me, it would follow that if this is the case, you're most certainly dealing with unstable firmware in multiple WAP WDS configurations.

One quick question: I've always setup these networks, IIRC, as all units sharing the same wireless channel. I vaguely recall this being the recommended configuration. Have you done any channel adjustments? You'd think with a manual config, they'd either work consistently or be flaky consistently, and that AUTO would result in fighting between WAP's, but it's perhaps worth a look.

Smart money here is on a firmware bug related to the number of WAPs in WDS, or some freakish bug with like... DHCP nuking the wireless radios of the devices.

 

[thorel]

Hum, I'll dig out the diagram, I've got it charted some where, now to locate it...

I haven't tried finding a lower bound, customer pressure to be operational has kept that level of experimentation at a minimum. It's a good idea and worth following through. It may take another week or so to get there though.

I did adjust the channel usage a bit and don't let them auto-select. I separated the signals by floor, using channel 1 on all first floor units and 11 on all the second floor units. I did that thinking the signal overlap areas would cause a client stability issue (iPhones, iPads, an AirBook and even a Lenovo Windows laptop). Perhaps that was a mistake and pushing them all to the same channel would improve the WAP stability.

At least now there's a few things to look into - thanks!

 

[thorel]

I found the network diagram, fairly simplistic to keep objects obvious.

More details:
Objects #7 & #8 are upstairs (Channel 11), all other access points are on the first floor and on channel 1.

The clients are limited to 10 addresses total, normally only 2 iPhones, an AirBook, an iPad, and another laptop. Guests, up to 4, usually iPhone type loads.

It usually only has 2 wifi clients not in the chart when failures start occurring, an iPhone and an iPad.

Without getting too deep into this, can you spend some time to graphically map out the current layout and connectivity?

WDS can collapse in a heartbeat if too much traffic...

I've got a bit of experience at dealing with Airport and the various iterations... but having the layout would be helpful in making the right recommendations.

 

[StratmanX]

I found the network diagram, fairly simplistic to keep objects obvious.

Nice diagram. What did you use to make it?

 

[thorel]

OpenOffice's draw program, then output a PDF. Really easy to use.

Nice diagram. What did you use to make it?

 

[sfx2000]

I found the network diagram, fairly simplistic to keep objects obvious.

More details:
Objects #7 & #8 are upstairs (Channel 11), all other access points are on the first floor and on channel 1.

The clients are limited to 10 addresses total, normally only 2 iPhones, an AirBook, an iPad, and another laptop. Guests, up to 4, usually iPhone type loads.

It usually only has 2 wifi clients not in the chart when failures start occurring, an iPhone and an iPad.

Wow... ok - at first glance this looks like a WiFi collision domain, and extended airports (via WDS or the newer stuff) adds even more traffic.

Keeping the airports/timecapsules in bridge mode, try the following.

(APX == Airport Express)

Take the APX #6 offline, attach the printer to the iMac and use print sharing, or hook it up directly to the USB port on the TimeCapsule.

Alternately, if the printer supports wireless, have it attach to the TC #4, or configure APX #6 as a client only if you want to serve as a printer server.

Time Capsule #4 lights up the first floor

Take AP Extreme #11, move to upstairs, taking APX #9 offline, APX #7 offline, and APX #8 offline.

AP Extreme #11 lights up the second floor

Set all to common SSID's, same channel assignments for both in 2.4GHz, and 5GHz if the TC/AP Extreme are simultaneous dual band models. Try setting the multicast rate to medium from the default low if possible.

Both the AP Extreme and the TC are more than capable of lighting up 1600 sq ft or more - you have much more access points than you need, and likely you're self jamming as a result.

just my thoughts...

 

[thorel]

You would think it's self jamming. However, the shape of the house and wall materials prevents such an approach. The TC and AE units are higher powered, but literally no signal can be found from the locations used, one to the other. I used inSSIDer to look at signal levels of the single TC unit in various locations, from where it is now there is no visible signal in the kitchen.

I get more bleed from floor to floor, which is why the different channel selections.

Sorry, but reducing the WAPs creates large areas without any signal and about half the remainder the signal is almost too low to be usable.

Nice try though.

 

[sfx2000]

Every home network is unique

 

[tannebil]

Did you make any progress? I'm helping a friend with a similar problem created by lots of thick, steel-reinforced, concrete walls and I'm considering a solution using an Extreme/Express extended network. But she has no wiring so it would be all wireless. Regular power-cycling would be undesirable but the network failing intermittently would be worse.

Anybody have suggestions for wireless network gear for an extended network? The attraction of the Apple gear is that I have experience with it, configuration is simple, and it's widely available (she's in Buenos Aires).

 

[Mark Uhde]

Get rid of the WDS and hardwire everything. Start making use of Channel 6. Make sure all the AP's have the same SSID and security settings. Whatever you do DON'T make one floor all 1 and one all 11! Try to do 1, 6, and 11 in a "cellular" type pattern where overlap is minimized.

Now, that's ideal network design for best capacity and coverage, but other than the WDS repeaters which are horridly unstable, your hardwired AP's shouldn't just be dying... that's odd. I don't know what to tell you there.

 

[claykin]

When wireless drops out, can you still connect to the "dead" units using the Airport Utility?

How old are these units? There's been 5 generations of the Extreme for example.

I have found Apple Airport Extreme's to be some of the most robust routers out there. i.e., they almost never crash. I don't use Airport Express so can't comment there.