CrowdStrike Apologizes on the Hill, Overhauls Rollout Procedures

CrowdStrike gave testimony before a U.S. House Homeland Security subcommittee on Tuesday, with senior vice president of Counter Adversary Operations at CrowdStrike Adam Meyers stating that “more oversight” is now in place.

Reflecting on cause of outage has led to new processes, according to Meyers

On July 19, 2023, CrowdStrike found itself in the middle of a historic IT outage caused by a defective update to its Falcon platform. The update caused mass system crashes worldwide that impacted payment services, airlines, hospitals, and others. Meyers said that CrowdStrike generally sends out about 10 to 12 configuration updates per day, but this one update unfortunately had an error in its configuration.

“If you think about a chessboard trying to move a chess piece to someplace where there’s no square, that’s effectively what happened inside the sensor,” said Meyers.

In an effort to prevent another catastrophe, CrowdStrike has outlined a new set of protocols, including more carefully controlled rollouts of software updates, better validation of code inputs, and new testing procedures to cover a broader array of problematic scenarios.

“We are deeply sorry this happened and we are determined to prevent this from happening again. We have undertaken a full review of our systems and begun implementing plans to bolster our content update procedures so that we emerge from this experience as a stronger company,” Meyers said.

CrowdStrike’s threat detection configuration information, Rapid Response Content, is released gradually across increasing rings of deployment now, according to Meyers. This will allow the company to monitor for issues in a controlled environment and proactively roll back changes if problems are detected before they affect a wider population.

Further, CrowdStrike has made changes to provide customers with additional controls over the deployment of configuration updates to their systems.

Hearing took a turn to unrelated AI questions

The hearing also involved questions about whether AI was somehow a contributing factor to the defective update, but Meyers denied that AI was related in any way. While AI and generative AI have  the ability to lower the barrier of entry for low-skilled threat actors, it was not what caused the crash.

Meyers said that AI “gets better” every day, but is not quite there yet.

Testimony is over, but legal concerns are just beginning

In the coming months, CrowdStrike could still see several lawsuits against the company as businesses threaten to file claims on grounds of financial loss or personal disruption. It is possible that customers could be waiting to see if new details emerged from this week’s hearing before moving forward with any legal action.

“As of this morning, to the best of my knowledge, we actually haven’t seen a lawsuit against us by a customer for the incident,” CrowdStrike CFO Burt Podbere said two weeks ago on a Q2 earnings call. “So we don’t know how it’s all going to shake out.”

Delta Airlines threatened to file a lawsuit against the company and Microsoft for negligence, citing that $500 million dollars of profit vanished due to canceled flights after thousands of passengers were stranded. 

Additionally, CrowdStrike is facing a putative class action lawsuit from investors who argue they were misled by the company and told its technology was “validated, tested, and certified” before the faulty update triggered the global outage.

Discover more about how companies like Microsoft are working to ensure another major outage doesn’t happen again.