What's going on with Rogers?

[HWDan]

I have Rogers regular cable TV and internet. Cable TV was fine during this outage and one of the reasons I don't want to move over to their Ignite IPTV is stuff like this.

However, I've been working from home for over a decade and have always had a cheaper than Bell TPIA 50/10 DSL service as a backup.

 

[Tech9]

I have dual ISP Bell and Rogers lines in my business, but I deal with other businesses only and it was business as usual, mostly. A friend of mine runs a restaurant though and with working Bell line still couldn't process payments. Pearson airport was in a mess with ArriveCan and border services affected. Rogers has to be fined millions for disrupting an entire economy, including government services. This is second time happening in less than 2 years time period. This country badly needs more competition for better services and lower prices.

 

[shabbs]

INTERAC had a dependency on Rogers it seems, which is why a lot of payment issues were happening even for those not directly impacted. Seems crazy they'd only have one supplier. They are making a move to add redundancy. Surprised this was not something that was in place already. Next contract discussion should be interesting...

https://twitter.com/x/status/1545770735448330240

 

[Tech9]

Yes... why didn't they add a supplier after Apr 2021 outage?

 

[Tech9]

Email to customers:

Dear Valued Customer,

As you know, we experienced a service outage across the Rogers, Fido, Chatr and Cityfone wireless networks on Friday.

I am reaching out to share that our services have been restored, and our networks and systems are close to fully operational. Our technical teams are continuing to monitor for any remaining intermittent issues. I also want to outline an action plan we are putting in place to address what happened.

I want to share what we know about what happened on Friday. We now believe we’ve narrowed the cause to a network system failure following a maintenance update in our core network, which caused some of our routers to malfunction. We disconnected the specific equipment and redirected traffic, which allowed our network and services to come back online over time as we managed traffic volumes returning to normal levels.

We know how much you rely on our networks and I sincerely apologize. We’re particularly troubled that some customers could not reach emergency services and we are addressing the issue as an urgent priority.

We will proactively credit your account for Friday’s outage. This credit will be automatically applied and no action is required from you.

As CEO, I take full responsibility for ensuring we at Rogers earn back your full trust, and am focused on the following action plan to further strengthen the resiliency of our network:

[*]Fully restore all services: While this has been nearly done, we are continuing to monitor closely to ensure stability across our network as traffic returns to normal.
[*]Complete root cause analysis and testing: Our leading technical experts and global vendors are continuing to dig deep into the root cause and identify potential steps to increase redundancy in our networks and systems.
[*]Make any necessary changes: We will take every step necessary, and continue to make significant investments in our networks to strengthen our technology systems, increase network stability for our customers, and enhance our testing.
                                                           
We let you down on Friday. You have my personal commitment that we will do better.       

Tony Staffieri
President and CEO, Rogers Communications

 

[shabbs]

I got that too... and a Rogers Survey - not sure if that was just bad timing or what... but I filled up that feedback to the max.

 

[Tech9]

I am organizing a peaceful protest on the beach.

Tony, make sure the services are up when I come back, please.

 

[RMerlin]

TekSavvy uses Rogers network for high-speed Internet plans in most of Ontario and Fido is a Rogers company as well. This won't work here.

Agreed, you'd need the cell to be with a different provider. It works for me because of being on vCable.

 

[shabbs]

Rogers still handing out 1hr DHCP lease times since service came back... this is typically 24hrs.

 

[Tech9]

Good enough. It may not last that long anyway.

 

[dosborne]

EDIT: I lied - still having issues with a number of channels not coming in... one of them being TSN5 - need my F1!

Still having issues as of tonight despite Rogers closing my ticket saying it was resolved. I'd say it is now "mostly watchable" but video and /or audio stutters and pauses every 3-4 minutes. Will see how it is by tomorrow morning and get back into the support queue of it hasn't improved.

I feel for the businesses and people more severely impacted. In reality, I can live without cellular, internet, home phone or tv for an extended period if necessary but was listening about one elderly lady who depends on food delivery, which she couldn't order, nor would the door entry system in her building work (and she has mobility issues). I had kept TechSavvy as a backup in the past, until I realized it leased Roger's infrastructure. Maybe time to look at minimal Bell or Telus service or pay as you go options.

Being in the pandemic, and lived through many power outages due to storms, ice/snow, tornadoes, wind storms, gas shortages, etc we ensure we always have stock of food, water, cash and gas.

Losing electricity for an extended period would be (and has been) 50x worse IMO, but, we are fortunate to consider Rogers services as "non-essential" as we don't work from home, run a business, have any urgent need etc. The only real concern was potentially needing 911.

 

[shabbs]

Yeah Ignite TV here has that odd pause/black out every few minutes as well. Annoying and certainly not back to normal yet.

 

[cptnoblivious]

Interesting discussion, thanks for the link to the Cloudflare article @shabbs !

It's sad that we have to get good, straightforward analysis from an outside party and not directly from Rogers. I agree that they did an incredibly poor job managing communication. This stuff is basic, and any of us who've worked in high-demand type environments know that there is a real simple playbook for these events that must be created ahead of time and then followed, including how creates and distributes the comms and how, and who the fallback is.

None of this is rocket science.
1. Tell us what happened.
2. If you don't know, tell us what you're doing
3. Make sure you set expectations when the next communications even happens, even if it's just to say there is nothing new yet

If this were a tiny provider with minimal staff, OK, maybe then don't have a good Business Continuity Plan (which of contains the comms layout), but Rogers is a giant. They have no excuse for not having this in-place.

I've worked in highly regulated industries and this stuff is 'table-stakes'. I.E. it just has to be there.

Luckily my wife and I are on different providers, hers is Telus based, I'm on Rogers, so at least one of us had a working phone during this. But it's sure making me think about having a failback plan for data (I already have voip capabilities, but when away from home that's no good if I don't have any data ).

 

[shabbs]

$5.50 credit coming?

'We deserve more': Customers not happy with Rogers compensation for nationwide outage

Rogers users with an $85/month wireless plan can expect a $5.50 credit

nationalpost.com

 

[Tech9]

Sports bars and restaurants may get $10 compensation for destroyed Friday night business. I told my friend not to worry about it.

 

[shabbs]

My Ignite TV is back to totally useless. Can't even get a channel. Internet/Cell/Phone still fine so I can stream etc... but this is terrible.

 

[shabbs]

Well, today looks like Ignite TV is finally back to normal - none of the issues from yesterday have occurred so far... dare to dream.

 

[Tech9]

Ignite got reignited?

 

[dosborne]

Indeed. It was refreshing to be able to watch the news without it stuttering and freezing.

It will be interesting to see what the CRTC turns up, or Roger's response or whatever. I was confused by the news saying Roger's had not stated what the outage was caused by as the CEO email had indicated " We now believe we’ve narrowed the cause to a network system failure following a maintenance update in our core network, ". Maybe that was simply deflection?

 

[Tech9]

It was refreshing to be able to watch the news without it stuttering and freezing.

I don't watch news lately. They make me stuttering and freezing.