IT outages can have a massive impact on your company's balance sheet and reputation. According to multiple surveys, 33% of all outages are caused by vendor operational bugs. According to Gartner, when outages occur, the reported cost to the Enterprise is more than $300k per hour, and $8.7 million per year. Those averages pale in comparison to what we saw from the recent global outage, caused by the cybersecurity vendor, CrowdStrike. Parametrix, the leading provider of cloud monitoring, modeling, and insurance services, estimates that the total direct financial loss facing the US Fortune 500 companies (excluding Microsoft) from the CrowdStrike outage on July 19 is $5.4 billion. Delta Airlines, CrowdStrike’s customer, recently shared that they incurred $500M in financial impact over a 5 day period.  

In this article, we want to shed light on how companies can proactively detect and prioritize vendor operational bugs, to round out their IT risk management strategy. Specifically, we’ll cover:  

1. What are vendor operational bugs? 

2. How can vendor operational bugs be proactively addressed before they cause an outage? 

3. What needs to change to make proactive risk mitigation a reality? 

Vendor operational bugs are flaws that exist in the code used to run our systems and applications. These bugs are a byproduct of releasing new code into production to update features, address known defects, patch security flaws, and more, a process that always carries some risk. However, when the most catastrophic bugs aren’t found and mitigated, the result is what we saw happen with CrowdStrike. Systems crash, consumers are impacted, employees work exhausting hours, some people lose their job, millions of dollars leave the balance sheet, and brand reputations are ruined.    

While $10B is spent by IT organizations every year proactively addressing vendor reported security vulnerabilities, there is no reporting on how much enterprises spend addressing operational vendor bugs. Thich begs the question, why aren’t we doing more to prevent these outages that cost companies millions annually?   

In today’s cloud economy, SaaS applications run on multiple third-party platforms which provide essential functionality, uptime, and streamline operations for their customers. A 2019 report estimated that enterprises use an average of 1,295 cloud services, with this number increasing annually. Considering the numerous daily and weekly code releases from these vendors, each containing dozens to hundreds of updates, enterprises are exposed to millions of vendor-related bugs.   

Consolidating all information related to vendor bugs is the first challenge. Every vendor shares different information in different places and in different formats. Relevant information for each bug is also published by other stakeholders in other forums across the internet. IT teams need a solution to automatically ingest vendor bug updates and normalize the data to manage in one central place. 

The second challenge is accurately correlating the impact of these updates on active infrastructure. Vendor bug announcements offer general information about impact versions, so IT teams need a solution that can map updates to their CMBD to reduce millions of bugs down to the few hundred priority bugs. 

The third challenge is to identify truly catastrophic bugs that pose the most significant risk. Vendor bug data is often incomplete, and no standard exists to evaluate the risk of an operational bug. IT teams need to enhance bug data and apply risk scoring to identify only the truly catastrophic bugs to be addressed in their ITSM.    The analogy of a needle in a haystack feels quite appropriate, and BugZero was built to be your magnet.  

We built BugZero to solve the pain of IT outages. BugZero is the only software solution that automatically identifies and consolidates vendor bugs, correlates them to your environment, identifies the highest risks, and integrates with ServiceNow. BugZero enables your team to take proactive action and prevent costly operational downtime. With BugZero, ITOps teams can manage operational defects comprehensively, just as IT Security teams handle security vulnerabilities. 

ITOps leaders need to shift their mindset away from accepting vendor operational risk as a mere cost of doing business. They need to move on from reliance on manual research from bandwidth-constrained teams or overpaying vendors to alert them to impending problems. Instead, they should prioritize proactive detection and prioritization of operational bugs to keep their organization resilient.