BugZero: Protecting Financial Services Firms against $15M outages

Operational disruptions are harmful – both in terms of cost and reputation. In the financial world, the stakes are even higher. People who invest in a fiduciary to manage and maintain their finances put an enormous amount of trust in those firms. As a financial services provider, do you really want your reputation to be one of unreliability? It’s safe to say the answer is no.

In early 2022, a financial services firm with over $200B in assets was reeling from a series of IT outages caused by software bugs (more formally known as operational defects). In the first five weeks of the year, the firm dealt with four different outages – and the losses due to those outages tallied to a staggering $15 million dollars.

In need of a solution, the firm came to BugZero in July of 2022 to learn more about this first-of-its-kind operational defect risk management platform. The firm knew they needed a more targeted approach to reducing risk in their third-party software supply chain, with a strategy that looks beyond Common Vulnerabilities and Exposures (CVEs).

In this case study, we explore how BugZero encouraged this firm to re-assess their previously accepted IT risks with an actionable solution – preventing future outages while increasing their operational resilience.

For the financial services firm, 2022 started with a barrage of IT outages that totaled over $15 million in losses. The final straw and catalyst for the firm to change was a Cisco bug that wreaked havoc on their data center. 

Clearly, change was needed. Repeated outages are not only costly, but also cause significant reputational damage. That can spell disaster for any organization – but especially a financial services firm. Implementing a proactive and automated solution to mitigate these operational defects was paramount.

When BugZero entered the picture, our team quickly assessed the situation. The first order of business was onboarding the firm.

Like many financial institutions, the firm must adhere to strict compliance requirements. BugZero is prepared for these situations. Our SOC 2 compliance ensured an accelerated approval process. Our SOC 2 audit helped the financial services firm expedite the evaluation and approval of BugZero as a trusted partner.

Further, BugZero’s presence in the ServiceNow app store proved invaluable.  The certified app made for an easy integration between BugZero’s defect risk management solution and the financial services firm’s existing IT service management solution.

This ensured that BugZero was painlessly incorporated into the firm’s daily IT operations, improving their processes and maturity.  The BugZero team worked with the financial services firm’s IT team to configure specific filtering options.  This ensures maximal noise reduction by creating remediation tasks for only the most severe and catastrophic defects.

Vendors publish far too many bugs for any human or team to properly evaluate. The financial services firm leveraged BugZero’s automation and AI capabilities to whittle hundreds of thousands of bugs down to just the ones that are truly important to the firm.

1. Assessment: BugZero begins by assessing current defects using a proprietary algorithm that calculates a Risk Score. This score provides immediate insight into the severity of the issue. Cisco alone has over 991,000 documented software defects, and BugZero filters them down to the most severe ones. In this specific customer environment, our software found 25 catastrophic bugs, any one of which could have caused a dreadful outage.  

2. Risk Scores: BugZero taps into an array of data sources, including community replies, likes, comments, severity indicators, status updates, and available workarounds in the Operational Defect Database (ODD). This data collection ensures that the firm has a 360-degree view of their defects, empowering them to make well-informed decisions.  

3. Automation at Work: Here’s where BugZero really shines. We take on the cumbersome and repetitive tasks that humans simply don’t have the time for. Our solution correlates data, cross-references it, and compiles reports all so the firm doesn’t have to. 

4. Remediation and Change Management: BugZero doesn’t stop at identification. The platform also plays a pivotal role in the remediation process. Through an integrated change management system, BugZero tracks the progress of defect resolution, ensuring that every step is well-documented. Our recommendations align with the ISO 31000 risk management process.

5. Closure and Reporting: Once a defect is resolved, BugZero assists in the closure process, providing an organized record of the issue’s lifecycle. Moreover, BugZero reporting visualizes the firm’s IT risk as well as the value added once those risks are mitigated.

For this financial services firm, the introduction of BugZero helped them see immediate benefits. From a reporting perspective, BugZero generates reports that offer insights into operational defect trends, resolution times, and overall system health.  

Since the implementation of BugZero, the financial services firm has not experienced any bug-related outage.

Noise reduction and operational resiliency is not a one and done process. Over the ensuing months, BugZero continued to work with the financial services firm to further fine-tune the defect filtering process. The goal is to reduce the number of alerts even more, ensuring that the firm can focus on the most critical issues. BugZero continues to work with the firm to implement and refine their risk management for Microsoft, NetApp, VMware and Redhat among others.   

Technology is meant to make lives easier, not harder. BugZero’s integration and automated defect management has helped mature the financial services firm’s IT operations while reducing their IT risk. The overall cost/benefit is significantly better than one would imagine.  We typically see a 20-40x ROI, if not more, depending on the size and complexity of the organization’s IT environment.   

It's time for a solution to help ITOps be more proactive and reduce downtime. 

Learn more about how BugZero can lower your IT risk and mature your organization – read through our FAQ!