Poor Change Management: Most Common Root Cause of Major Incidents?

Manish Rathi

Last updated 25/05/2023

---

       

As per recent Industry survey "80% of unplanned outages are due to ill-planned changes made by administrators ("operations staff") or developers in a Production environment."

Here are Key reasons - Major Incidents Are Caused By Changes

Here are the list of the companies which shows that ongoing failures and performance problems cost companies both in lost revenues and damaged reputations. The business effectively shuts its doors, puts out the 'We are closed' sign, and then is left wondering 'Will our customers be coming back?'

1. Bank of America Online Banking Down Across U.S.

• Duration: 6 days
• Impact: Affected 29 million online customers
• What Happened: The problem was noted as the result of a "multi-year project" to upgrade its online banking platform.

2. Amazon EC2 Goes Dark In Morning Cloud Outage

• Duration: 4 Days
• What Happened: The trigger for this event was a network configuration change.

3. Google Suffers First Gmail Outage of 2011

• Duration: 2 days
• Impact: 120,000 users affected
• What Happened: After analyzing the issue, Google Engineering determined that the root cause was a bug inadvertently introduced in a Gmail storage software update.

4. BlackBerry Outages Spread Throughout the World

• Duration: 24 hours (some more)
• Impact: Unavailable worldwide affecting millions of users around the world
• What Happened: Service outages due to "core switch failure within RIM's infrastructure.

5. Intuit Service Outages Leave Frustrated Customers

• Duration: 2 days (some users up to 5 daysz
• Impact: Thousands Affected
• What Happened: The problem was caused by a change to their network configuration.

6. Yahoo Mail Suffers Outage

• Duration: 24-hours plus
• Impact: Affecting people around the globe
• What Happened: Change configuration issues.

A recent Gartner study projected that "Through 2015, 80% of outages impacting mission-critical services will be caused by people and process issues, and more than 50% of those outages will be caused by change/configuration/release integration and hand-off issues.

Cost of Service Outage?

Unplanned outages are the responsibility of IT to resolve. However, at the end of the day, they are, essentially, business issues. Part of a thorough evaluation process is calculating how much money you will lose for each hour (or minute, or another time increment of your choice) of downtime.

Here is the average cost of unplanned Service outage as per the recent study.

Conclusion:

poor change management practices can have far-reaching consequences for organizations, including financial, operational, and reputational losses. To mitigate these risks, organizations must prioritize a structured change management framework, robust control mechanisms, effective communication, stakeholder involvement, risk assessment, and continuous improvement. By investing in these areas, organizations can navigate change more successfully and safeguard their long-term success.

ITIL Vs DevOps: Is That They Fit Together For Innovation? Let's Check

How your Personal certification helps your companys ITSM

How ITIL and ITSM Are Protagonist In Process Improvement?