What You Need to Know About the Recent CrowdStrike Incident

CrowdStrike is a cloud-based cybersecurity organization primarily offering agent-based endpoint detection and response (EDR) software. Several major corporations rely on software from this leader in the EDR space.  

Overview of the CrowdStrike Update Incident  

All technology requires regular updates to protect against possible viruses. On July 19, 2024, CrowdStrike pushed an update to its Rapid Response Content that caused a Windows system to crash.  A problematic file, Channel File 291, caused an out-of-bounds memory to read, forcing the operating system to crash and reboot. CrowdStrike marked the drivers as bootable, so the system “boot looped,” meaning the machine continuously rebooted until it finally booted into a Windows recovery environment.  

This issue was found and reverted in under 90 minutes, but due to CrowdStrike’s vast reach, the damage was already done.   

How the CrowdStrike Update May Affect Your Organization 

The solution was simple: delete the corrupt channel file and reboot the system without the corrupt file. However, the crashed systems were effectively offline, so repair required a tedious, manual process, which was further complicated by encrypted hard drives that required recovery keys. Since most machines do not have remote access capabilities, a person must manually remediate each machine, making the recovery time much slower.  

Who Was Affected?  

Microsoft estimates that the breach affected at least 8.5 million machines worldwide. Some claims estimate that 25% of Fortune 500 companies were affected by this CrowdStrike outage, with many visible to the public through service disruptions and travel delays.   

While back-end server infrastructure was recovered quickly if it was affected at all, virtually every endpoint for certain airlines was down and in need of remediation.  This included potentially every terminal at every gate for the airline agents and every monitor showing flight status; all would require someone to physically inspect the device and correct the issue.  Any outage of this scale takes time to resolve.  

Addressing the Impact 

Preventing an issue such as this is not easy. There is an inherent level of trust placed in software vendors, particularly EDR software, given their level of access to the Windows operating system. Customers were given no control over the deployment of CrowdStrike's channel file updates, which meant customers had to trust that the organization was performing proper testing before deploying the updates. This issue is not unique to CrowdStrike for antivirus or EDR vendors. In fact, it has happened to several organizations like McAfee, Kaspersky, and to Microsoft.   

What Are CrowdStrike's Support Resources? 

CrowdStrike released guidelines after the incident to help streamline the recovery process. These guidelines included a script to build recovery media for local and network boot. This allows for a mostly automated recovery process but still essentially requires a person to initiate the process physically. This resource ultimately provided customers with tips and tools while also helping to cut down recovery time.  

Best Practices for Preventing Future IT Outages 

When something of this magnitude happens, you question what you can do for your organization to help. Unfortunately, IT outages are not entirely preventable from a customer standpoint, but there are two things you can do to improve your recovery process: 

• Review your business continuity plan  
• Review your disaster recovery plans 

Even if your business was not directly affected by this outage, take the lessons learned and apply them to your policies. This is an ideal time to review similar scenarios, identify any gaps in current plans, or build entirely new business continuity and disaster recovery plans.  

If your organization is still navigating issues related to the CrowdStrike outage, or if you need help creating a proactive plan for business continuity and recovery, contact us below to learn how we can help.