Troubleshooting a Network for us Beginners … Fix It Before It Breaks!

[Klueless]

I’m a total novice so take anything I say with a grain of salt … or an aspirin. I’m a part time entrance level employee at my brother-in-law’s car lots where I made a lot of mistakes in a short time. The purpose of this much … too … long post is to help other novices avoid the pain we went through!

Ben has a small car lot with a service center behind it. Five miles down the road is his larger car lot which he expanded a few years ago by buying the house and property next door. The sales office was getting crowded so he moved part of the staff to the house next door. Our Wi-Fi signal actually reaches but the computers were slow and kept losing connection.

One of my bosses tells me to go buy a new router. I puzzle, “Who bought and set up what we have?”

“I don’t know, a friend of a friend.”

So I go to the corner computer store and the clerk goes, “Here, you’ll like this router.”

I read the “quick install”, unplug the old one, plug in the new and poof … I’m now a “network engineer”! I briefly consider setting security but I’m not allowed to touch the computers unless they break so I have to leave the network “open”.

Everything works (no harm, no foul) but the PCs in the house next door are still slow and intermittent so it’s back to the corner computer shop. I learn all I had done was upgrade a B/G router to a B/G/N router over 2.4Ghz. $50 later I walk out a B/G range extender. I install it by the window in the house next door and presto, all the PCs in the house are working great.

Suddenly I’m the “network guy”. (But with a caveat, I’m not allowed to touch anything unless it’s broke.)

It wasn’t long before my luck ran out. Internet was running crappy and I had no clue. Nothing had changed so I called the cable company. “Nothing wrong here, how many computers do you have?”

“Uh, eight.” (We are up to twelve now plus printers, servers, iPADs and smart phones.)

“You need to buy more bandwidth.”

“How much?”

“$100 a month?”

“Total?”

“No, an additional $100 a month.” (Side note, when you’re a business around here you get to pay more than the home user and you get less bandwidth than the home user but … you get 24 hour access to a help desk … that tells you, “buy more”.)

“Uh, have you any statistics?”

“Yes, your peak day was 6 GigaBytes. You need more bandwidth.”

I do the math, “But that sounds like I’m only using, what, about 10% of my network?”

“You need more bandwidth, call us when you’re ready to place your order. Goodbye.”

More bandwidth? What I need is a real network engineer. An hour on Google and several phone calls later I’m beginning understand why they used a friend of a friend for the original installs.

I call the phone company, I luck out, they’ll double my bandwidth and total cost is less than we’re paying now. Install is set for a few weeks out.

While I’m waiting the problem just disappears and I’m thinking, “Damn, it was the cable company!”)

A year later I fight my way through a crisis at the other lot and a year later I’m in crisis mode at the main store again. One of my bosses is in the closet hollering and yanking on wires. I’m thinking to myself, “We're driving blind, we need data; this is no way to run a network!”

Then a break, a four hour area black out. Power comes back up but my router doesn’t. It is fried.

Back to the computer store but this time I’ve got a plan and … a wish list. I found a router (I chose an Asus RT-N66U) that met many of my requirements:

· Usable User Interface
· Traffic Monitor
· Displays of what’s on the network
· Quality of Service (QoS) settings with bandwidth limiters
· Two radios, 2.4Ghz and 5Ghz
· Lots of SSIDs
· Ethernet ports​

Other Sundries;

· A second monitor for my PC (yes, of course I found a free one : )
· WiFi Analyzer for Android

o Shows WiFi signal strength

§ Gives rough ideas for where to place devices and/or how well things might work​

o Shows what radio channels are already being used

§ Aids in selecting clearest channel​

· Internet Speed Test (Myself, I usually use my vendor's speed test as I assume it's optimized for their cloud of stuff.)

o Verify I’m getting what I’m paying for from my carrier
o Create stress for testing
o Confirm bad (or good) performance​

· NetStress

o Shows actual transfer rates between our devices
o Tests performance of just our network (rather than the Internet)
o I can actually walk around with a laptop verifying what WiFi Analyzer suggested with real data transfers
o Demonstrates that you made good (or bad) choices for placement of devices​

· Ping Plotter

o Automated Tracert and Ping package
o Graphs route from you to end point
o Graphs latencies and lost packets
o Very useful for suggesting where problems are and whether they’re local or somewhere in the Internet (it clears up the cloud)​

 

[Klueless]

Troubleshooting ... Fix your Network Before it Breaks - Part 2

This is how we started:

· Renamed devices to names that help identify who's who and what's what.
· Passwords to help reduce some of the unknowns

o Password protected our WLANs (except for "guest"). We went from 40 plus connections to 20 something - most of which I can identify now.
o Changed default passwords on all routers/modems. (ISP wasn't happy I changed theirs but they’ll get over it.)​

LANs and WLANs: This is how we restructured our network:
o Ethernet – put our servers and clients on Ethernet
o SSIDs

§ SSID "Faster" – put our production PCs on the 5Ghz radio

· Each radio is its own collision domain where devices compete and fight for shared bandwidth. Forcing important stuff to the less cluttered radio should give them more consistent performance.​

§ SSID "Further”

· Production PCs that only had a 2.4Ghz radio
· The Range Extender

o Thus … all the PCs in Bldg. 2​

§ SSID “BYOD” – for employees who bring their own devices for work, spans both radios
§ SSID "Guest" – for customers and visitors, spans both radios
§ Why so many SSIDs?

· I suspect most users will simply have two SSIDs; one for their stuff and one for guests’.
· I had extenuating circumstances I needed to mitigate

o Permanently located desktops were roaming between radios (and even picking out the worst fit)
o BYODs and Guests are unknowns to me; if things got rowdy I can now simply disable less needed devices/SSIDs and hope production returns to normal.​

Router Configurations:
· SSIDs – as above
· QoS – took the out-of-box configuration

o It prioritized web first.

§ For us web is work, everything else can wait a moment
§ Web is interactive, perception is everything​

o Bandwidth Limiters

§ Set up and down links to 80% of what the Internet Speed Tests showed (as per Toasted Tomato and others).
§ Intriguing sidebar; I notice the IEEE 802.11 protocol suite imposes most of its overhead when the network is congested while other protocols (e.g., Token Ring -IEEE 802.5) impose their overhead upfront to avoid problems later. Toastman et al (who published articles on QoS and bandwidth limiters) seem to follow the latter philosophy. I agree.​

Verification:

§ Everything worked
§ Ran NetStress – Results verified plan, everything was performing as well as one could hope
§ Users were happy​

Setup 2nd monitor on my PC.

§ I display both traffic monitor and ping plotter on monitor #2
§ Real time traffic monitor displays a ten minute moving window

o I set Ping Plotter for a ten minute display and adjusted the widow to overlay the traffic monitor’s display
o I can now see cause / effect, e.g., traffic goes up / latency goes up​

§ I conduct my work day on monitor #1 glancing at monitor # 2 to see if anything interesting is going on. If so I start digging.​

Results:

§ We had another crash after I had everything in place and we found our problem:

o Data that tended to point away from us

§ Traffic Monitor showed very little traffic
§ The intranet still worked​

o Data that tended to point towards the carrier

§ Ping Plotter showed no problems with hops 1 and 2 (us) but showed high packet loss on hop 3 (the ISP’s border router) that carried right through to the target system.
§ Users found the Internet unusable
§ Couldn’t even run an Internet Speed Test
§ Inverse cause / effect; traffic was down but latency was up (way up!)​

o I was finally able to have a fair fight with our carrier
​

§ Boss' daughter was in for the holidays. I don't know what kids do with iPADs but she had our uplink and downlink pegged for days. No one noticed and everything ran fine. Based on my experiences with the other lot I know that this behavior would have crashed us and that QoS was really working. (It also saved me a call to the help desk which would have sagely advised me to “buy more bandwidth” : )

§ Sales manager bought a new iPhone. It was saturating our 1.5 Mbps outbound link. No one noticed and we were running fine. Again, based on other experiences, I know QoS kept us from crashing. Even better I brought him in to look at the traffic monitor, had him shut off his phone and look at the traffic monitor again. Traffic stopped when he shut off his phone. He got mad at me. This went on for several more days, don't know root cause, but, eventually, his phone "fixed itself" and ... he's talking to me again.
​

Conclusion:

I’m really impressed with these home routers. Easy to setup and they really work … until something goes wrong. Then you’re screwed. I found the carrier will oft tell you, “It’s you!” Sometimes they’re right, sometimes they’re not. Either way the onus is on us.

I learned the hard way. Now when I look for a router I don’t look for the fastest or even the cheapest, I look for one that will help me help myself. They’re out there but they’re not always easy to find. Most reviews don’t cover those features. Mine’s not bad, there’s probably better, but mine’s getting me through the muck so I’m happy … very happy.

Prepare for failure now, it will make troubleshooting a lot easier later. Even better, it’s already prevented a couple outages!​

 

[Klueless]

This is what Ping Plotter looks like sitting on top of my Traffic Monitor. They are both showing the same ten minute view and are running in real time. (If the pic isn't full size you might have to login and/or click it to view.)​

Blue is outbound and the other is inbound traffic. Nothing special is going on. Ping Plotter shows just a little jitter over the traffic bumps but not really enough to notice. It's much more pronounced when there's actually traffic. It's currently showing my path to Google and it looks like all the routers in between are behaving very well. The high inbound "baseline" is my boss watching the security cameras mounted at the other lot five miles down the road.

 

[Klueless]

We had been experiencing numerous outages for over two months, several times a week, sometimes several brief (but long enough to notice) in a single day and sometimes a single outage that lasted a couple of hours. This is a Ping Plotter chart that we captured during a ten minute outage. (You might have to login to view the chart?)

It shows a 70% packet loss beginning at the ISP's border router and continuing through to the target system. The help desk still said it was my problem but I lucked out. Found a guy who heads up local operations and shared all my data with him. I haven't had a problem since ... going on over a month now.

 

[Klueless]

So I notice a new to me traffic pattern on my router's traffic monitor. Not crazy high peaks but not insignificant either. What catches my eye is they're consistent (height and spacing) and constant, running relentlessly throughout the day.

The next morning nothing and then they start again, about 8:30AM. I look at my 24 hour monitor and see they went from 8:30AM to 4:30PM the day before. Not a huge concern but it could be if a couple more PCs start doing whatever this PC is doing.

So I look at my wired, 2.4 and 5Ghz views (this is where it'd be nice to be able to view traffic by device) and see it's coming from one of our wired devices. Now we've only a few hard wired devices and, given the time block, I zoom in on our Title Clerk.

It turns out she's been listening to Christmas music.

"But I've got the volume turned down real low so there shouldn't be much network traffic." (I quietly file this away as one of the most adorable things I've heard in years : -)

But she's right, music shouldn't cause peeks that high. Turns out her music had accompanying videos (that she wasn't watching) so I set the resolution to low and that took care of that. Peaks then dropped to very reasonable levels.

 

[Klueless]

The dust has settled from our two month ordeal with our ISP and I stop at a new-to-me computer shop that advertises network setup / network repair. The owner is telling me about his services, pricing and clients (turns out he's not so little).

I tell him I'm considering engaging a professional to review our network, update and document as needed and be around for disasters. Anything to avoid the two months of fire and brimstone we had just been through.

Turns out he has the same ISP for his shop and had been having the same problems during the same two month window. (So much for the ISP telling me no one else is complaining so it must be me.)

Excitedly I ask, "So what did you do? How did you get the phone company to fix it?"

He told me the company next door uses a different provider so he connected to their network.

"So for $90 an hour you would have done what for me?"

"Well, nothing I can do if it's a vendor problem."

We went over our "tracert"s together, yep same paths, and I showed him my trace with 70% packet loss at the border router. We agree to contact each other if we should ever have that problem again and to join forces when talking to the phone company. Two small companies, 3 miles apart, having the same problems at the same time? Coincidence?

Nice to know that even professionals get the same level of respect that us Rodney Dangerfields get.

 

[L&LD]

Klueless, the best advice I got from a non-techie customer is that if 'x' technology doesn't work immediately and consistently, whatever issue I thought I fixed, wasn't.

This has saved me many hours of frustration. If I have done everything I can on my side; it is time to pursue other avenues (and not keep banging my head on the same wall).

Almost every single business (ISP or otherwise) will try to shift the blame to someone else (usually me!). But when I know (from experience) that I have done everything in my power, I don't let them get away so easily. They either have to prove it is beyond their control or else fix the issue itself.

I think that you are now at that same point where you won't take anything at face value again with regards to networking 'issues' from ISP's or otherwise.

 

[Klueless]

Almost every single business (ISP or otherwise) will try to shift the blame to someone else (usually me!).

Some tech vendors remind me of the school bullies of my youth except instead of going to the gym I have to go to SNB Forums to learn to defend myself : -)

 

[sfx2000]

Nice to know that even professionals get the same level of respect that us Rodney Dangerfields get.

You don't know the first of it, just kidding... fought that battle myself, and like a couple of other people, I'm more connecTed than the average bear...

My broadband provider updated their peering arrangements (moving from one to another) and messed up their SBC's and BGP tables - after properly diagnosing the problem, and going thru their retail support Tier's 1, 2, and 3 - I eventually opened a ticket using my professional/work hat with the new provider - and then I got a response within 30 minutes and a fix, which probably most of their other customers in the region appreciate to this day...

Funny what one's email address can do in a trouble ticket

 

[Klueless]

... and going thru their retail support Tier's 1, 2, and 3

That's awesome!

I tried as hard as I could but couldn't charm my way past level 1. It was like their jobs were on the line if they let anyone get by them. I really lucked out when I found the local op mgr and that he was as receptive as he was.