Human error caused 2022 Rogers outage, system 'deficiencies' made it worse: report | CBC News (2024)

Politics

The 2022 Rogers outage that left 12 million people without wireless and hard-wired services was caused by human error and made worse by management and system "deficiencies," according to an independent review conducted for Canada's telecommunications regulator.

The Rogers outage lasted 26 hours, cut 12M people off from 911 and other network services

Human error caused 2022 Rogers outage, system 'deficiencies' made it worse: report | CBC News (1)

Peter Zimonjic · CBC News

·

Human error caused 2022 Rogers outage, system 'deficiencies' made it worse: report | CBC News (2)

The 2022 Rogers outage that left 12 million people without wireless and hard-wired services was caused by human error and made worse by management and system "deficiencies," says an independent review conducted for Canada's telecommunications regulator.

The review report also sayssteps taken by Rogers since the outage are "satisfactory to improve the Rogers network resiliency and reliability, as well as to address the root cause of the July 2022 outage."

The 26-hour outage started early in the early morningof July 8 and left individualsand businesses without access to their mobile, home phone, internet and 911 services.

The Canadian Radio-television and Telecommunications Commission (CRTC) commissioned Xona Partners in September 2023 to undertake the review and determine what caused the outage.

The engineering consultancy was also tasked with looking at whether the measures taken by Rogers since the outage are sufficient to prevent another incident.

Full report to be released, with redactions

Xona Partners' findings were contained in the executive summary of the review report, released this month. The CRTC says the full report contains sensitive information and will be released in redacted form at a later, unspecified, date.

The report summary says that in the weeks leading up to the outage, Rogers was undergoing a seven-phase process to upgrade its network. The outage occurred during the sixth's phase of the upgrade.

  • Remember the Rogers outage? It's added 221,000 new phone customers since then

"The July 2022 outage is attributed to an error in configuring the distribution routers within the Rogers IP network," the report says.

Staff at Rogers caused the shutdown, the report says, by removing a control filter that directed information to its appropriate destination.

Without the filter in place, a flood of information was sent into Rogers' core network, overloading and crashing the system within minutes of the control filter being removed.

Algorithm designated network upgrade as 'low' risk

The report says Rogers' core network manages wireless and hard-wired data both internally, within the company, and externally, for outside customers and service providers.

"With both the wireless and wireline networks sharing a common IP core network, the scope of the outage was extreme in that it resulted in a catastrophic loss of all services," the report says.

  • Rogers CEO announces new protocols, investment in wake of major service outage

Having wireless and wireline services share the same network is a practice "common to many service providers," the report says, adding thatcompanies find it an efficient way to "balance cost with performance."

Rogers has since announced that it will developa new, separate, network for its wireless systems while keeping hard-wired services on the old core network. The report says that work is ongoing.

The review says that because the first five stages of the network update took place without incident, "the risk assessment algorithm downgraded the risk level for the sixth phase" of the upgrade.

How Rogers is addressing the issue

Designating risks in phase six as "low" meant Rogers' staff could avoid additional levels of scrutiny and approvals as the upgrade proceeded, even though doing so "contravenes industry norms," the report says.

Rogers says it has since installed a new risk assessment algorithm to address the issue.

WATCH | Report looks into 2022 Rogers outage:

Human error caused 2022 Rogers outage, system 'deficiencies' made it worse: report | CBC News (3)

Human error triggered massive 2022 Rogers service outage, report finds

4 days ago

Duration 2:08

The massive Rogers service outage that affected millions of Canadians two summers ago was triggered by a human error that overloaded key routers, a CRTC report says. Rogers says it has since upgraded its systems.

The executive summary of Xona Partners' review also says the "network failure could have been prevented" if Rogers had "overload protection mechanisms" limiting how much information is funnelled into the core network.

The review recommends that all telecom networks in Canada implement overload protection mechanisms for their core networks.

Challenges restoring the network

A central issue frustrating Rogers' efforts to get its systems back up once they went down was the corporation's inability to communicate properly.

The report saysthat when the core network went down, remote employees were unable to access Rogers' systems or use the internet and could not get online by using other service providers.

  • CRTC hires private company to investigate 2022 Rogers outage

"Rogers had to dispatch staff to remote sites to physically access the affected routers, which delayed network recovery efforts," the report says.

All incident response and crisis team members at Rogers have since been provided with backup, third-partyaccess to the internet to "maintain communication capabilities during outages."

The review also saysthat Rogers staff could not access critical error logs detailing the root cause of the outage until 14 hours after the outage began, which "adversely impacted outage recovery efforts."

More transparency needed, says critic

John Lawford, executive director of the Public Interest Advocacy Centre in Ottawa, has been pushing Rogers and the CRTC for more transparency onthe outage.

He criticizedthe CRTCfor taking two years to deliver a report on the outage, describing itas a "whitewash in the sense of both the CRTC and Rogers being very much let off the hook."

Human error caused 2022 Rogers outage, system 'deficiencies' made it worse: report | CBC News (4)

"The report makes a claim that Rogers has rectified the issue and there is insufficient evidence for me there to see that," Lawford said. "This is just one particular expert's viewpoint."

Rogers declined CBC News' requestfor an interview.

In a statement, a Rogersspokesperson said the company will "remain focused on delivering the most reliable network experience so Canadians can connect when and where they want."

Government claims network investments underway

The spokesperson said, citingan August 2023 report from analyticsfirm umault,Rogershasthe most reliable wireless network in Canada.

To come to that conclusion,umault ran a series of tests over a 47-day period last year and found Rogers was the most reliable wireless provider in Canadaduring the time surveyed.

The spokesperson also pointed to another recent study from Opensignal, anetwork testing company, that found Rogers hasthe most reliable internetservice in Canada.Opensignal collected its data on Rogers and other providersbetween February 1 and April 30 of this year.

"We completed a full review of our networks, strengthened our network resiliency, and implemented all the recommendations of this report. We will continue to invest to ensure Canadians enjoy the best networks in the world," theRogersspokesperson said.

  • Former industry minister Navdeep Bains gets Rogers job, sparking criticism

In a letter to Rogers, the CRTC said the company had "confirmed the implementation of all measures" recommended byXona Partners.

A spokesperson for Minister of Innovation, Science and IndustryFrançois-Philippe Champagnetold CBC News that Rogers has addressed all recommendations in the report and is continuing to invest in network resiliency.

Corrections

  • A previous version of this story said the Rogers outage started in the early evening of July 8 and lasted 15 hours. In fact, the Xona report found that the outage started in the early morning of that day and lasted more than a full day.

    Jul 05, 2024 6:06 PM ET

ABOUT THE AUTHOR

Human error caused 2022 Rogers outage, system 'deficiencies' made it worse: report | CBC News (5)

Peter Zimonjic

Senior writer

Peter Zimonjic is a senior writer for CBC News. He has worked as a reporter and columnist in London, England, for the Daily Mail, Sunday Times and Daily Telegraph and in Canada for Sun Media and the Ottawa Citizen. He is the author of Into The Darkness: An Account of 7/7, published by Random House.

    Corrections and clarifications|Submit a news tip|

    Related Stories

    • Rogers CEO announces new protocols, investment in wake of major service outage
    • Service restored to most Rogers customers hit by outage Wednesday
    • CRTC hires private company to investigate 2022 Rogers outage
    Human error caused 2022 Rogers outage, system 'deficiencies' made it worse: report | CBC News (2024)
    Top Articles
    Latest Posts
    Article information

    Author: Foster Heidenreich CPA

    Last Updated:

    Views: 5863

    Rating: 4.6 / 5 (76 voted)

    Reviews: 91% of readers found this page helpful

    Author information

    Name: Foster Heidenreich CPA

    Birthday: 1995-01-14

    Address: 55021 Usha Garden, North Larisa, DE 19209

    Phone: +6812240846623

    Job: Corporate Healthcare Strategist

    Hobby: Singing, Listening to music, Rafting, LARPing, Gardening, Quilting, Rappelling

    Introduction: My name is Foster Heidenreich CPA, I am a delightful, quaint, glorious, quaint, faithful, enchanting, fine person who loves writing and wants to share my knowledge and understanding with you.