Review: CrowdStrike Under Scrutiny: Examining the July 19th IT Outage
Summaries and fullscript.
Summary:
CrowdStrike takes responsibility for the July 19th global IT outage.
The outage was caused by a faulty content update, not a cyberattack.
CrowdStrike has implemented changes to prevent similar incidents.
Concerns remain about kernel access and AI-driven cyber threats.
The importance of public-private partnerships in cybersecurity is stressed.
Stats:
8.5 million: Devices impacted by the CrowdStrike outage.
$5.4 billion: Estimated losses caused by the outage.
10-12: Number of content updates CrowdStrike releases daily.
99%: CrowdStrike sensors back online by July 29th.
250+: Threat actors tracked by CrowdStrike.
Fullscript
House Committee on Homeland Security
Subcommittee on Cybersecurity, Infrastructure Protection, and Innovation
Hearing: Examining the July 19, 2024 Global IT Outage
Date: Sep 24, 2024
Witnesses:
Mr. Adam Myers, Senior Vice President for Counter Adversary Operations, CrowdStrike
Opening Statements:
Chairman Andrew Garbarino (R-NY):
Good afternoon. The purpose of this hearing is to examine the global IT outage that occurred on July 19th, 2024, as a result of a faulty software update released by CrowdStrike. We seek to gain detailed insights into how the update was developed and deployed, and what errors led to the widespread disruption. We will discuss the extent of the outage and its impact on key sectors of the economy. We will also examine how malicious cyber actors leveraged the outage for malicious activity.
Just over two months ago, essential functions came to a halt. Hospitals saw disruptions, flights were grounded, banks experienced downtime, and U.S. federal government agencies were temporarily unable to access certain data. We learned that this global IT outage, regarded as the largest in history, was not due to a malicious cyberattack, but a faulty software update pushed out by CrowdStrike.
According to a company statement, a sensor configuration update triggered a logic error, leading to system crashes and the blue screen of death appearing on impacted systems worldwide. CrowdStrike software updates are essential for addressing vulnerabilities, enhancing threat detection, and ensuring the cybersecurity infrastructure of its customers remains robust.
We are here today to get answers: what went wrong, what was required in response, and what we have learned for the future of our nation's cybersecurity posture. The sheer scale of this error was alarming. A routine update caused this level of disruption. Imagine what a skilled, determined nation-state actor could do.
Our adversaries have assessed our response, recovery, and true level of resilience. Our enemies are not just nation-states with advanced cyber capabilities; they include malicious cyber actors who thrive in the uncertainty and confusion that arise during large-scale IT outages. For example, CISA noted that it observed threat actors taking advantage of this incident for phishing and other malicious activity.
Mr. Myers, I look forward to hearing your testimony about how the faulty software update was pushed out globally, what CrowdStrike has learned from this event to prevent future outages, and how CrowdStrike is working to rebuild trust. I would also like to discuss the impact this global outage has had on our nation's critical infrastructure sectors, what support CrowdStrike has provided to those who were disrupted, and how the company has addressed malicious cyber actors who attempted to take advantage of the global outage.
Ranking Member Eric Swalwell (D-CA):
Thank you, Chairman. We need CrowdStrike to be effective and successful because its effectiveness is the success of the companies it protects. I appreciate CrowdStrike being part of this hearing.
We are here to get to the bottom of the circumstances and failures that enabled one content update to crash the operating system of 8.5 million devices worldwide. The impacts were diverse: flights were grounded, surgeries were canceled, 911 systems were disrupted, and stores had to close.
CrowdStrike must ensure its product adequately balances the need for access in an operating system against the risks that access poses. With the exceptional level of access that CrowdStrike has within a customer's operating system, CrowdStrike has an obligation to employ rigorous quality assurance processes for any updates it releases.
We are here to determine if any of these things happened before the July 19th outage. I appreciate CrowdStrike's commitment to ensuring its customers are protected against the most novel threats, but speed cannot come at the cost of operability.
This is not the first time this has happened to a company. In 2007, a different security firm released a faulty update that resulted in the blue screen of death. In the aftermath, the company undertook a thorough review and implemented changes to its product architecture and processes. Notably, it developed a mechanism to automatically roll back an operating system to a working state when an error is detected, began releasing updates incrementally, and removed code from the operating system kernel.
I'll be interested in whether CrowdStrike considered the 2007 incident as it defined its own processes for testing and releasing updates or defining the level of kernel access it needs to operate.
For the record, this is not the first time this Congress that we have had to ask a technology company why it failed to integrate lessons learned from an incident at a competitor company into its own security practices.
We get better when companies cooperate. Because Microsoft came to a hearing earlier this year and worked with us and CISA, and because CrowdStrike is here today and is working with us and CISA, we will get better. One of our goals is to ensure that we stop relearning yesterday's lessons so we can more proactively defend against the threats we'll face in the future.
I was pleased that earlier this month Microsoft unveiled and convened the Windows Endpoint Security Ecosystem Summit, which brought together security firms to discuss issues ranging from safe deployment practices to providing additional security capabilities outside of the kernel mode.
Today, I hope to get a better understanding of the trade-offs between kernel access and risks to the operating system and learn how we can better manage risks. I'm also pleased that last week I had the opportunity to speak with CrowdStrike CEO George Kurtz. He assured me of the company's commitment to making sure nothing like the July 19th incident happens again and shared updates on the actions CrowdStrike has already taken to address some of the key deficiencies that contributed to it.
Chairman Mark Green (R-TN) - Full Committee:
Thank you, Chairman Garbarino and Ranking Member Swalwell. On July 19th, Americans woke up to a shock: flights grounded, medical procedures canceled, 911 calls wouldn't go through. A global IT outage that impacts every sector of the economy is a catastrophe we would expect to see in a movie, carefully executed by malicious and sophisticated nation-state actors.
To add insult to injury, the largest IT outage in history was due to a mistake. In this case, CrowdStrike's content validator used for its Falcon sensor did not catch a bug in a channel file. It also appears that the update may not have been appropriately tested before being pushed out to the most sensitive part of the computer's operating system. This caused about 8.5 million devices to crash.
Mistakes happen, however, we cannot allow a mistake of this magnitude to happen again. As the July 19th outage has demonstrated, our networks are increasingly interconnected. While we know that nation-state actors and criminals try to exploit our networks, we don't expect companies to defend themselves from these targeted attacks. However, we do expect companies to implement the strongest cybersecurity practices.
Our nation's security depends on a strong public-private partnership for protecting our networks. Ensuring our partnership is strong is important because our adversaries always watch how we respond to incidents like the July 19th outage.
The good news is that since this was not due to a cyberattack, we can learn from the incident. Today's hearing is both timely and overdue. Timely because we now have two months of information to understand exactly what happened. It's overdue because we hoped to give Americans the answers they deserve much sooner.
Although I'd hoped to hear from CrowdStrike's CEO directly, I'm grateful for Mr. Myers's presence. I'm confident he will deliver the answers we need.
Mr. Myers, thank you for taking the time to walk us through the course of events leading up to July 19th and the steps CrowdStrike has taken. CISA Director Jen Easterly described this incident as "a useful exercise, a dress rehearsal for what China may want to do to us." We look forward to working with you to make sure we never make it to opening night.
(Witness is sworn in)
Witness Testimony - Mr. Adam Myers:
Chairman Green, Chairman Garbarino, Ranking Member Thompson, Ranking Member Swalwell, members of the subcommittee, good afternoon, and thank you for having me today. I am Adam Myers, Senior Vice President for Counter Adversary Operations at CrowdStrike.
At CrowdStrike, our vision is to protect good people from bad things, and we've been very successful at doing that for more than a decade. I'm proud to lead the threat intelligence side of our business. I direct a team of cyber threat experts tracking criminal, state-sponsored, and cyber adversary groups across the globe. Our goal is to produce actionable intelligence to protect our customers.
Despite our strong track record, I'm here today because just over two months ago, on July 19th, we let our customers down. CrowdStrike was in the process of updating our customers on a new threat but released a content configuration update for the Windows sensor that did not work as expected. This resulted in Microsoft system crashes for a number of our users.
On behalf of everyone at CrowdStrike, I want to apologize. We're deeply sorry, and we are determined to prevent this from ever happening again. We appreciate the incredible round-the-clock efforts that our customers and partners, working alongside our teams, mobilized immediately to restore systems. We were able to bring many customers back online within hours. I can assure you that we continue to approach this with a great sense of urgency.
I want to underscore that this was not a cyberattack. The incident was caused by a CrowdStrike rapid response content update focused on addressing new threats. CrowdStrike began working with customers and partners to bring systems online as quickly as possible, initially through manual remediation. CrowdStrike then introduced automated techniques to accelerate remediation. To further help customers, CrowdStrike also put boots on the ground to assist with recovery efforts. We also provided regular updates to customers throughout our response, available on our website and shared with policymakers and our customers.
We've also taken numerous steps to make sure this can't happen again, and we're pleased to report that as of July 29th, approximately 99% of Windows sensors were back online. We've endeavored to be transparent about what happened and are committed to learning from what took place. We have undertaken a full review of our systems and are implementing plans to bolster our content update procedures so that we emerge from this experience as a stronger company.
Finally, as we have enhanced our own resiliency, we remain laser-focused on protecting our customers against disruptive cyberattacks, as we have for a decade. While we have fixed the issue that led to this incident, there are many other threats that remain on the horizon. The threat environment is particularly challenging given global unrest and the upcoming elections here in the United States. We are focused on threats from nation-state adversaries, issue-motivated activists, and sophisticated e-crime adversaries motivated by profit.
At CrowdStrike, we are particularly focused on threats from North Korea, Iran, China, and Russia. Recent events have also highlighted the often underappreciated supply chain security considerations. Additionally, in the e-crime sphere, ransomware remains a chronic problem targeting victim organizations across the globe.
We appreciate your leadership on these issues, and I hope we can discuss some of these horizon threats with you today. Like you, we recognize the importance of remaining vigilant. I'm especially grateful to you and your staff for being accessible over these past several weeks to receive briefings and updates from our team and myself, and I look forward to our continued discussion here today. Thank you, and I welcome your questions.
(Questioning Begins)
Chairman Mark Green (R-TN):
Thank you for your statement. There was a degree of humility there that is impressive, and I appreciate the transparency.
In terms of the update, who made the decision to launch the update? Did AI do that, or did an individual do that? Can you tell me how that decision was made?
Mr. Myers:
Thank you for your question and your comments. AI was not responsible for making any decision in that process. It is part of a standard process. We released 10 to 12 of these content updates every single day, and so that was part of our standard operating procedure. These updates are automatic globally; they go global all at once.
Chairman Green:
When you send an update out, the updates were distributed to all customers in one session?
Mr. Myers:
We've since revised that. In the full testimony, I've included a graphic that depicts what that now looks like, and that is no longer the case.
Chairman Green:
So CrowdStrike is no longer fulfilling your updates like that, simultaneously, universally?
Mr. Myers:
That is correct.
Chairman Green:
Good. Honestly, that was probably my biggest question. That single fix was in. That's huge and I think would have prevented what happened from happening.
I'm glad to hear that you guys have decided to stair-step that update implementation. I don't really have any other questions; that was my biggest concern.
Rep. Troy Carter (D-LA):
Thank you, Mr. Chairman and Ranking Member. Thank you to our witness for joining us today. Cybersecurity protection means securing a network overall for better security and resilience. Despite our annual investment of over $60 billion dollars in cybersecurity, we continue to face significant shortages of trained personnel. This reality underscores a critical point: none of our protective measures, be they standards, technologies, or regulations, can succeed without a well-trained workforce. Addressing ongoing and emerging cyber challenges is critical to our nation's security.
One contributing factor to the widespread impact of the incident that brings us here was the inability of customers to control when they received this kind of update. If customers had the ability to schedule the receipt of updates, is it possible that fewer devices would have been impacted?
I'm pleased that CrowdStrike has addressed this issue and is giving customers greater control over when they receive content updates. Can you elaborate on what kind of options customers now have with regards to receiving these updates, and how are you ensuring their customers understand the ability to control content updates?
Mr. Myers:
Thank you for your question, Congressman. Moving forward, what we've implemented is a system of concentric rings. Think of it as the initial internal release process being the first step in releasing new content updates. From there, customers can select to be part of the early adopter program, where they can choose to receive content updates as quickly as we can make them available. From there, there's another factor they can select, which would be general availability, which would come after early adopter. And then from there, they can select to wait some period of time before those updates get pushed out or choose not to receive them as well.
Rep. Carter:
Is there a preferred process, which, if any, is safer, more efficient, and would grant the greatest amount of protection to the consumer?
Mr. Myers:
Thank you, Congressman. I think the early adopter is appropriate for systems for testing purposes. In other words, if an organization would like to receive those content updates in a timely manner and make sure that there is no outcome or unexpected behavior, and then from there, the general availability for mission-critical systems or things that they would prefer to wait even longer for, they can choose to do that. But that comes, of course, with the risk that they're not getting the most up-to-date threat intelligence information provided to their system.
Rep. Carter:
Is there a capacity to overrule those options and automatically push forward an update because of the sensitivity of the risk?
Mr. Myers:
No, sir.
Rep. Carter:
Given the revelations that we've had, is there a mechanism for there to be a greater amount of cooperation between CrowdStrike, Microsoft, and the other good actors, as opposed to the bad actors who seem to have a leg up in being nefarious? Are there mechanisms that you guys have in place that you're coordinating and working more closely in sync to prevent these types of issues in the future?
Mr. Myers:
That's a great question, Congressman. Yes. In fact, on the weekend of July 19th, we began working very closely with Microsoft to ensure that our mutual customers had the benefit of us working together. And then subsequent to that, just last week, there was a meeting at Microsoft that was, I think, already mentioned, where CrowdStrike and others participated in a sit-down with Microsoft to plan for future improvements and to ensure continued resiliency by CrowdStrike and Microsoft and other companies as well.
Rep. Carter:
Lastly, before my time runs out, is there a mechanism for the community to be made more aware, in a kind of layman's sense, of what citizens can do to better protect themselves and to be aware? We know there's all sorts of phishing scams and all kinds of stuff out there. What can the general consumer, what might we be doing as this committee to help educate the general public?
Mr. Myers:
Awareness is a key factor, as you point out. And during this incident, we were able to push out threat intelligence on our blogs to advise general consumers of threat actors that were trying to take advantage of these situations. In general, I think better awareness of the threats is absolutely critical and something that I would be happy to continue to work with you and your staff on to come up with strategies to help educate the public about some of these threats.
Rep. Steven Palazzo (R-MS):
Thank you, Mr. Chairman. The global outage we experienced on July 19th, 2024, impacted millions of people and is estimated to have cost billions of dollars. This disruption impacted a wide range of services, including airlines, hospitals, emergency service call centers, and more, impacting our nation and the global economy.
While I appreciate how quickly CrowdStrike identified and deployed a fix to the problem, it turned out that the solution was a manual reboot process. For workers in rural areas, like many communities in Mississippi, this means it could have taken them days to find the right IT person to get to the scene and get the computers up and running.
Mr. Myers, why was that the only solution, a manual fix?
Mr. Myers:
Thank you for your question, Congressman. Initially, there was a manual, hands-on process that was required, which we mobilized our entire team to support. We activated our partners; we worked around the clock. I offered myself to drive 10 hours to visit a customer to help them get some of their systems back online. Within the next day or so, we were able to identify some automated systems that would enable us to facilitate the recovery at a much faster pace, and that was where the bulk of that recovery occurred. We saw a massive uptick in systems coming back online once we deployed the automated process.
Rep. Palazzo:
What steps are you taking to protect your systems from a difficult recovery process if something like this happens again?
Mr. Myers:
Thank you, Congressman. We, as in the testimony I referred to this earlier, provided a new system to ensure that the content updates have an opt-in and kind of the ability to choose when you receive those updates. Prior to this, our sensor packages, all of our source code, had those established best practices already in place, and now we are applying this as well to the content updates.
Rep. Palazzo:
I represent South Mississippi. Can you please describe the support CrowdStrike offered or provided to rural communities such as in my district?
Mr. Myers:
I'll need to get back to you with some details on that, sir.
Rep. Palazzo:
A recent article stated that, and I quote, "Engineers and threat hunters were given just two months for work that would have normally taken a year." Additionally, the article noted that CrowdStrike confirmed its use of, and again I quote, "existing engineers instead of hiring a new team of cloud threat hunters."
Pearl River Community College and many others in my district offer an excellent cybersecurity technology program for our next generation of students to help fill this unsettling skills gap. Do you make these staffing decisions because of the lack of an adequate workforce in the industry?
Mr. Myers:
Thank you, sir. We have a robust internship program. We bring some of the most, a lot of our talent from these internal and external internship programs and recruit from all over the country and all over the world to fill positions.
Rep. Palazzo:
What steps do you take to better support your staff and ensure that they have the right tools and skills to succeed?
Mr. Myers:
That's a great question, sir. We have extensive internal training programs. We also send our team to various trainings across the globe, different industry trainings at conferences and other programs where they can learn new skills and continue to develop their existing skills. And then we also have some of our own researchers and analysts conduct trainings at those same events to help train individuals that are not yet in the workforce or working at other companies to learn some of the critical skills that are needed to identify and track advanced threat actors.
Rep. Palazzo:
Mr. Myers, we know that there are many single points of failure in our cyber ecosystem. These can be exploited either through a mistake or an attack and cause an impact similar to what we saw from the CrowdStrike incident. Additionally, when products are modified or updated, another single point of failure is created.
What do you suggest this committee focus on to mitigate single points of failure and improve the stability of these systems?
Mr. Myers:
That's a tough question. I think there's a lot that needs to be done to identify and mitigate single points of failure. We have been acting at CrowdStrike in terms of identifying vulnerable systems across the globe, doing research to try to determine where vulnerabilities may exist, and coming up with mitigating strategies for that. And it's a continuing effort. It's something that takes everybody. It's a team sport, and we all need to work together for that.
Rep. Palazzo:
Thank you, Mr. Chairman. I yield back.
Ranking Member Eric Swalwell (D-CA):
Mr. Myers, CrowdStrike released its content configuration update into the kernel, which is the core part of the operating system, where an error will crash the entire system rather than just one single application. Some competitors of yours have claimed that this kind of kernel access is dangerous and that a better practice is to deploy such updates directly to the user mode, where the impacts would only affect an application.
Can you explain, just for folks who may not understand the kernel and this dialogue, why CrowdStrike issues updates to the kernel, how it balances the benefits of kernel access to the risk that it creates, and is CrowdStrike planning to make any changes to how it uses kernel access to reduce the risk of crashing entire systems?
Mr. Myers:
Thank you, Congressman. The Windows kernel, and all operating systems have a kernel, is the central, kind of most important part of the operating system. In many cases, you'll hear it referred to as ring zero. The kernel is responsible for interfacing with all the hardware associated with that operating system or that computer.
CrowdStrike is one of the many vendors out there that uses the Windows kernel architecture, which is an open kernel architecture. This is a decision that was made by Microsoft to enable the Microsoft operating system to support a vast array of different types of hardware and different systems.
The kernel is responsible for, or the key area where you can ensure that you have performance, where you can have visibility into everything happening on that operating system, where you can provide enforcement, in other words, threat prevention, and as well to ensure anti-tampering, which is a key concern from a cybersecurity perspective.
Anti-tampering is very concerning because when a threat actor gains access to a system, they would seek to disable security tools. And in order to identify that that's happening, kernel visibility is required to see when that's occurring.
The kernel driver is a key component of every security product that I could think of, whether they would say that they do most of their work in the kernel or not varies from vendor to vendor. But to try to secure the operating system without kernel access would be very difficult.
Ranking Member Swalwell:
The Cyber Safety Review Board has demonstrated that it can conduct a meaningful review of cyber incidents, including the most recent Microsoft review, which has proven helpful to this committee's oversight of recent incidents involving Microsoft. Some cybersecurity experts have called for the CSRB to conduct a review of this incident. Do you believe they should? And second, if they do, will you fully cooperate with that review?
Mr. Myers:
Thank you, sir. We would fully cooperate with everything. We've been working with CISA, the various ISAC communities, and many of your staffs directly since the July 19th incident to ensure that we have provided transparency and visibility into everything that's occurring.
Ranking Member Swalwell:
Shifting toward your mission of protecting your customers, we have seen over the years that our adversaries, particularly around even years in the fall, like to weaponize their own capabilities to attack democracy. Can you just give us a picture of the threat environment right now as far as what you're seeing from our adversaries, any new, unique lines of attack, and any particular countries that you see escalating their attacks as we head to November 5th?
Mr. Myers:
Thank you, Congressman. Our adversaries watch these elections very closely. We've already seen in this election process that Iran has played a role in targeting campaigns. We continue to see China and others as well.
With regard to what activity we've seen thus far, espionage continues to be the primary motivator for countries like China and Russia. We have seen, certainly in the past, that these adversaries have stolen sensitive information and leaked them. We also see a rich array of disinformation and misinformation occurring as a result of foreign adversaries using social networks and things of that nature to drive narratives that are supportive of their agendas.
Rep. Sheila Cherfilus-McCormick (D-FL):
Thank you, Mr. Chairman. Mr. Myers, your root cause analysis attributed this global IT outage to a failure in your validator tool. However, it seems like another important contributing factor here was closely related to how you released the software update. So I'd like to go back to the topic of a phased rollout approach. I believe you were describing this change when you used the phrase "concentric circles." But starting here, was this update pushed out in a phased way?
Mr. Myers:
Thank you for the question, Congresswoman. To be clear, the configuration update that occurred, the content update, was not code. This was threat information that was being provided to the sensor. The code is pushed out, and that is, the sensor itself had been pushed out using a phased deployment methodology where the code would go through extensive quality assurance and quality checks. It would then be deployed internally, in what we would call dogfooding. And then from there, it would be rolled out to customers who could select N+1 or N+2, meaning that they could wait some set period of time before they would actually roll out the new sensor. This sensor was rolled out in February of 2024.
The content updates are not code. That had not previously been treated as code because they were strictly configuration information. What we have undertaken since the July 19th incident, what I referred to earlier in the testimony, is that we are now treating the content updates as code, which is something that I don't believe to be an industry standard at this time. By us treating that as code, it is now going through that process of, again, the internal testing or dogfooding before it goes to early adopters, general availability, and then the N-X strategy.
Rep. Cherfilus-McCormick:
Thank you for that distinction. Would you agree that in this case, while it may have been a content update, clearly the failure to have it implemented or take effect in a phased approach ended up being catastrophic?
Mr. Myers:
We've moved to a phased approach as a result of the incidents of July 19th, and we've put a lot of time and effort into making sure that that phased approach will ensure customers have the ability to choose when and how they receive those updates.
Rep. Cherfilus-McCormick:
Going back to the Ranking Member, in his comments, you two were discussing the fact that CrowdStrike has extraordinary access into the kernel of the operating system, and you all were talking a bit about the risk versus efficiency of having this kind of access and making updates within the kernel. Share with me your thoughts on whether this incident could have been averted, or future incidents could be averted, by using the user space for this kind of update.
Mr. Myers:
Thank you for the question. The kernel, as I said, provides the visibility, the enforcement mechanism, the telemetry and visibility, as well as the anti-tamper. So I would suggest that while things can be conducted in user mode from a security perspective, kernel visibility is certainly critical to ensuring that a threat actor does not insert themselves into the kernel themselves and disable or remove the security products and features.
Rep. Cherfilus-McCormick:
So is it your assessment then that it's not possible, really in realistic terms, to do it outside of the kernel with the current kernel architecture?
Mr. Myers:
This is the most effective way to get the visibility and to prevent an adversary from tampering with security tools.
Rep. Cherfilus-McCormick:
So it's the most effective way, but it's not the only way possible.
Mr. Myers:
It is certainly the industry standard to use the kernel for visibility, enforcement, and anti-tamper, and to ensure that you can stop a threat.
Rep. Cherfilus-McCormick:
You've testified thus far that you've made modifications to the phased rollout approach and also the pre-deployment testing. What other modifications has CrowdStrike made, or changes to your internal practices, to avert future similar incidents?
Mr. Myers:
The primary changes that we've made, we've come up with an entire new mechanism by which we distribute the content updates. Again, making sure that customers have in their control the ability to select when they receive those updates is what will prevent that from happening.
Rep. Cherfilus-McCormick:
Mr. Chairman, I yield back.
Rep. Tony Gonzales (R-TX):
Thank you, Mr. Chairman. Good afternoon, Mr. Myers. Can you explain to me, at a more granular level, for internal testing, you said that there's, there was a human element involved in this. I'm curious how, because you guys use the OODA loop method inside your company, correct? Big fan of it, by the way. So great job.
I'm just curious if we're doing internal testing in a way that's trying that most certainly would prevent this from happening, and the human element is involved, and it's not artificial intelligence pushing out this, this information. Can you walk me through that, please?
Mr. Myers:
Absolutely. Thank you, Congressman, for the question. The process for testing the content updates was reliant on validators.
Rep. Gonzales:
How many?
Mr. Myers:
I have to get back to you on the exact number, but we tested each of the channels. So each of the, the different rules that were inside that content file were tested individually, and those validators ensured that the rules conformed and were compliant with the very structure that CrowdStrike had built for those content updates.
Rep. Gonzales:
Meaning they test individually. Is there something to be said about not testing them collectively? It seems like a very large miss. You guys touch a lot of things. I mean, you touch a lot of infrastructure in the United States, something that we count on you for, which you've been doing a great, great job, right, with this one.
You mentioned North Korea, China, and Iran. Our outside actors are trying to get us every day. We got, we shot ourselves in the foot on the inside of the house. So I'm curious if we're testing these things individually, should we be, is there a point at which we test them collectively before we push it out?
Mr. Myers:
That's, thank you, sir. That's where we are now. So the new methodology is to test all of the content updates internally before they're released to the early adopters.
Rep. Gonzales:
So that, where that's where the fault was internal. We were testing the coding individually instead of collectively, and one of them was off.
Mr. Myers:
The testing process looked at each configuration and made sure that it conformed with the standard. It is now being tested internally before it's rolled out to customers, and then the customers have control over when, what systems get those updates.
Rep. Gonzales:
Yes, sir. Which I'm still trying to figure out exactly how this thing got launched with it not being absolute...
Mr. Myers:
The rules, the validator itself was in place for over a decade, and we've released 10 to 12 of these updates every single day since we started using the configuration updates. That was tested against the standard to make sure that the configuration conformed with the standard that CrowdStrike had written for those configuration updates. This happened to test positive, and we sent it out.
Rep. Gonzales:
I'm trying, I think I'm trying to find out, did it test, did it test positive and we launched it, or did, was there, did it, did it fail and we launched it accidentally?
Mr. Myers:
I see. Thank you. Yes, it tested as clean, or good, and that's why it was allowed to roll out.
Rep. Gonzales:
So as we unpack this, where exactly did it fail? This may be a complicated question to answer in a minute and 18 seconds.
Mr. Myers:
I'll give it a try. All right. So the content file triggered an issue within the kernel sensor. So that when the sensor processed the configuration, it is almost like, if you think about a chessboard, trying to move a chess piece to someplace where there's no square. That's effectively what happened inside the sensor. So when it tried to process the rule, it was not able to do what the rule was asking it to do, which triggered the issue within the sensor.
Rep. Gonzales:
Should we have known that? Should CrowdStrike have known that about the kernel, correct? Or is this something that we weren't aware of?
Mr. Myers:
This was a kind of a perfect storm of issues that resulted in the sensor failure.
Rep. Gonzales:
Okay. I'm going to need about 20 more seconds, Mr. Chairman.
Chairman Garbarino:
You go with that.
Rep. Gonzales:
So knowing what we know now, what is the response mechanism in place, worst-case scenario, this happens again, for all the users?
Mr. Myers:
Thank you. So this would trigger, this would be detected within CrowdStrike before it ever made it to the early adopters now.
Rep. Gonzales:
We're doing it as code?
Mr. Myers:
Yes.
Rep. Gonzales:
Okay. Thank you, Mr. Chairman. I yield back.
Rep. Robert Menendez (D-NJ):
Mr. Chairman, Ranking Member, it's, I don't often get to say what a privilege it is to serve not just with the folks on this committee, but with two incredible leaders like the Chair and Ranking Member. I'm just so thankful for their friendship and stewardship of this subcommittee.
As we've discussed today, the global IT outage triggered by a faulty CrowdStrike sensor update disrupted critical services across various sectors. In my district, passengers at Newark Liberty International Airport experienced delays and cancellations. Some New Jersey hospitals had to delay or cancel procedures, and some 911 dispatch centers were even rendered inoperable, jeopardizing public safety.
This incident wasn't only a significant disruption, but also a preventable event that could and should have been avoided with basic quality assurance practices. Our government services rely, and our constituents deserve, the highest level of security and reliability. CrowdStrike must implement robust measures to prevent future incidents and ensure that their technology truly protects and serves our communities effectively.
The root cause analysis also mentions increased customer control over rapid response content deployments. How does providing customers with control over rapid response content deployments improve overall security?
Mr. Myers:
Thank you for the question, Congressman. And let me first start by saying, again, we apologize. We are deeply sorry for the impact for the folks at the airport and in the hospitals. We have a long legacy of stopping threats, and that's our primary objective as a company.
In terms of how these controls will enable customers or constituents to have more control over what happens on their systems, it effectively gives them the ability to select which systems they can themselves test on. And let's say that's the early adopters. They can, they can select any number of systems that they would like to enroll in that early adopter program to receive those content updates before any of the other systems in their environment. And then from there, they can have the rest of their systems on general availability. And if there are systems that are particularly sensitive and they want to withhold, even for a few more iterations, they can do that.
Rep. Menendez:
So this gives them complete control over where updates go and when they get them?
Mr. Myers:
Yes.
Rep. Menendez:
And thank you for that. I appreciate, we just want to make sure this all works right, and there's the benefit of your customers, our constituents. And
Rep. Menendez:
And thank you for that. I appreciate, we just want to make sure this all works right, and there's the benefit of your customers, our constituents. And so we're just trying to, to make sure that we all get on the same page. What type of support does CrowdStrike provide as the customers are making these individualized decisions to make sure that in them having a bespoke approach to adopting the sort of new technologies, etc., that, that we believe that they are getting the full suite of options that they need for their particular industry? Right? Because, you know, cybersecurity is obviously a quickly evolving field. And so we want to make sure that there, there are no gaps for any particular customer or industry now that they have a little bit more control over what they bring online on their systems.
Mr. Myers:
Thank you, Congressman. CrowdStrike, I would say in my experience, has been a customer-focused organization from the very beginning, 13 years ago when we launched the company in the wake of what was called Operation Aurora, where security tools of the day failed to detect Chinese adversaries who were conducting espionage operations against Western businesses. From the moment we launched the company, we've been very focused on ensuring that we hear our customers and that we're here to support our customers and that we are part of their mission, whatever that may be.
We continuously hear from our customers through customer advisory boards. We are constantly engaging with customers. On the 19th of July, we started round-the-clock phone calls, talking to every customer we could get a hold of to hear what was going on and ask them how we can help. We've also been briefing ISACs and government agencies, whether it be CISA or others, and working with congressional staff to ensure that everybody's questions are answered. And we do this throughout the year, not just in the wake of that incident. So it's really, as I said earlier, a team sport. We need to work together with our customers, with the government, and with everybody involved to ensure that we're all marching in the right direction.
Rep. Menendez:
Yeah, I appreciate that. And hopefully we'll do a second round, but I appreciate that. I yield back.
Rep. Carlos Gimenez (R-FL):
Thank you, Mr. Chairman. I was gone for a little while, so maybe you've answered this, but did you say that CrowdStrike issues hundreds of updates daily?
Mr. Myers:
Congressman, I had said that 10 to 12 times per day, we have issued content updates, which contain the latest threat intelligence information to instrument our sensor, our tool, to understand what new threats are evolving. The threat landscape changes sometimes minute by minute. And so, in order to keep ahead of those threats, to allow the CrowdStrike platform to detect and prevent those threats, it needs routine updates. So 10 to 12 times a day, CrowdStrike will update its systems to react to a new threat that you've seen. Is that, is that accurate?
Rep. Gimenez:
That's accurate. I would say that it updates the configuration information, not the system itself.
Rep. Gimenez:
Did, did the, the events of, of July 19th, was that the system upgrade, or what, what was that? It was something different than this 10 to 12 per day thing, or what, what was it? Just, yeah, tell me what that was.
Mr. Myers:
Thank you, Congressman. It is, that was a configuration update.
Rep. Gimenez:
And how often do you do that?
Mr. Myers:
As I said, 10 to 12 times per day, sir.
Rep. Gimenez:
Okay. So 10 to 12 times a day, you've got these, these updates. You've done them every day, so you've done thousands of these updates. What made this one different? And do you run, do you run system tests on each one, each 10 to 12 times a day? You run a system test to make sure that this thing is not going to do more harm than good, I assume, right?
Mr. Myers:
Thank you, sir. We, what caused, answer that question first. What caused this update was that the configuration update had a mismatch in fields that resulted in one of the fields not being linked to a rule. And so I, you may have stepped out earlier, but I had said that it was kind of like on a chessboard, if the chess piece moved to a square that wasn't present, then that would be an example of effectively what happened there. That...
Rep. Gimenez:
So before, okay. So since you've gone through it, I don't want to go through it again. So you do this 10 to 12 times a day. You've done this thousands of times. This one moved the chess piece outside the board. Therefore, the computers didn't know how to figure, "Wait a minute, that's outside the game. Therefore, I'm going to crash." All right. And that's never happened before. You've never moved the piece outside the board. This is the first time that this issue has manifested, to my knowledge. Have you tried to move the system, have you tried to move the piece outside the board, and was it caught sometime before?
Mr. Myers:
The validators that the configuration information went through were meant to ensure that it didn't move outside of the board. So was the, was the process, was your, were your internal processes followed, and then you just saw this as a problem with your internal processes that you have to fix, or was it somebody went outside your processes?
Rep. Gimenez:
Thank you. That's a great question. It was not a lack of following the process. This was an issue with the content validator. We've subsequently ensured that there's now additional steps in place so that this cannot happen again.
Rep. Gimenez:
So it's your process?
Mr. Myers:
In our process, yes.
Rep. Gimenez:
All right. Let's, I'm going to switch, switch gears, go to AI because I don't, I only have about a minute. AI, do you consider AI a threat to cybersecurity?
Mr. Myers:
That's a great question. AI, I think, can be...
Rep. Gimenez:
I just asked great questions, you know. I appreciate that.
Mr. Myers:
Yeah, it can be a threat or it could be a benefit. It can be used to facilitate and to ensure that cyber defenders have more tools at their disposal, that they can leverage AI to more quickly process information and analyze it.
Rep. Gimenez:
So before I, before I only got 30 seconds, I have to make a statement. So, so the nation that leads in AI would be better, better protected against, against a nation that's somewhat behind. The better your AI is, the better you are going to be at protecting yourself. And the better you are at AI, the, I guess, the better you are going to be at attacking your adversaries. Is that correct?
Mr. Myers:
I agree with your statement, sir.
Rep. Gimenez:
Okay, thank you. I have eight seconds, and hopefully you'll have a second round. Thank you. I yield back.
Chairman Andrew Garbarino (R-NY):
I recognize myself for five minutes of questions. Because I want to get into it, you called it the perfect storm happened. You know, failsafes failed. But can you talk about what was the perfect storm and why it will never happen again? Because, you know, a lot of perfect storms or 100-year floods are all happening now every other year. So I want to make sure that you all know what happened, can explain it, and then how you're making sure it's not going to happen again. We, we're dancing around it. I just, you know, let's get into the, let's get technical.
Mr. Myers:
Thank you, Chairman. The content validator was looking at the content channel file, which had 21 fields in it. The content validator allowed those 21 fields to go out to the sensor fleet. The sensor was looking for a configuration rule that was not present. When it attempted to use that rule, that's where the sensor failed. And so there was a, that's what caused the blue screen. And so that was detailed in the RCA. And I don't want to, I don't know if I could explain it in the three minutes that are left here, but effectively, because the con, the perfect storm was the content validator allowed the content configuration to go out to the sensor, and the sensor was not able to find the rule that it was looking for, causing the issue.
Chairman Garbarino:
So you fixed it. This can't happen again.
Mr. Myers:
It's a combination of process and the methodology at which we're now deploying those configurations. As I said, the configuration now is being treated as code, whereas before it was treated purely as configuration information. So we're providing a lot more oversight and visibility into what that is and how it goes out to the system.
Chairman Garbarino:
And how it goes out. You all changed, I was, but the Chairman asked before, it's not all going out at once to everyone. So even if this does happen again, you've fixed it where it won't, it won't affect everyone all at once. And it's able...
Mr. Myers:
Is now another problem with fixing what happened. There was a, people had to be on site, correct? You had to go to the different computers and reboot them individually. That, that's how you, that's how we got everything back up and running, correct?
Mr. Myers:
Initially, the systems would need to be rebooted, the file deleted, and then the system allowed to boot. From there, subsequent to that, we came up with a USB boot disk that could be plugged in, and the system could be rebooted, and that would automate the removal of the file. And then finally, we were able to deploy an automated solution, which allowed us to do this without manual intervention.
Chairman Garbarino:
Okay. So I, there was reporting about CrowdStrike's faulty software update has largely focused on commercial operations like emergency services, flights. But there was also a big impact on federal agencies such as FCC, Social Security, CBP, and even CIS, although networks are becoming increasingly interconnected. Government networks should be isolated from commercial ones. Why were federal agencies impacted by this outage? Does your process for pushing out updates include, are there different updates to test for commercial versus government business with, when you're dealing with your clients, or is that, is it all the same? Is it one computer, just one computer here?
Mr. Myers:
The updates went to Microsoft Windows operating system sensors that CrowdStrike had deployed. So that would have impacted any system that was running a Microsoft operating system with that particular version of CrowdStrike Falcon that was online during the time period that the channel file was distributed.
Chairman Garbarino:
So as long as Microsoft was on that, that computer using that system, it, whether it was government or commercial, didn't matter. It was affected.
Mr. Myers:
As long as the CrowdStrike sensor was running on the Microsoft operating system on those systems at that time, yes.
Chairman Garbarino:
Okay, wonderful. What... Actually, you know what? I'm going to, I'm going to come back for my second round because this is a much longer question than I, and I have my time left for. So I'm going to yield back and then I'm going to recognize the gentleman from Texas, Mr. Gonzalez, for five minutes of questions.
Rep. Tony Gonzales (R-TX):
Thank you, Chairman. And thank you, Mr. Myers, for testifying before our committee. I'll tell you what, I was very, not surprised, but disappointed to see how everything went down. But that's kind of the way it works in this space. And I was, I was grateful on how quickly CrowdStrike responded when they did find an error. I, I tell you what, I mean, I'm in government. I don't hear people, I don't see people that fess up and said, "Hey, we made a mistake, work to fix it." And then as you're doing that, send a report out so other people don't find that same mistake. Usually, it's try to cover it up and move on to the next thing. So I was grateful for the fact on how hard y'all worked to get things back up.
I'm, I'm interested, and I was grateful for the call that we got on literally days after it occurred. Clearly, you know your stuff inside out. I'm interested in how do we make sure, you know, if this happens again, that we're in a spot where we can fix it, right? And I'd argue that CrowdStrike is probably one of the better organizations that are out there. So what if it's a different vendor that maybe doesn't have the same resources, the same integrity, and whatnot? And so I'm looking at it through the lens, you know, speed is the name of the game in this industry. And a lot of times you want to get ahead of the problem before, as it's evolving.
Your technical report, how, let's dive into that a little bit. I mean, how has that been received in the industry? How has that been received in government? Have you had any conversations with CISA or others on the technical piece to what went wrong and how do we fix it?
Mr. Myers:
Thank you for the question, sir. The, let me start by saying that we were immediately in contact with CISA and many of your staffs to talk about this issue when it happened. Once we had gotten most of the recovery underway, we issued a preliminary incident report, which was available on the website, which was effectively put out to ensure that everybody understood what we knew at that point in time. We then gathered as many of our engineers as we needed to to start work on the comprehensive root cause analysis. We brought in external parties as well. And then we produced that root cause analysis as soon as it was available, which was, I believe, the first week in August or so. The dates are a little bit fuzzy at this point. But we had the RCA out.
The response that I've heard from most of the folks that I've spoken to have been that they appreciated the level of depth that that report went into. And I think, more importantly, that the plan that we put in place to prevent this from happening in the future is something that everybody acknowledged to me was going to enable our customers to have more control.
Rep. Gonzales:
Yeah, one, one of the concerns I have is every, every company does it a little bit different. Like there is no standardized process to it. And that's just a little troublesome because, you know, when everyone's doing it differently, you're relying on, "Well, we've never had this problem before." We're about to have a lot of problems that we've never had before. And I want us to get ahead of it. I just don't want this to be a flash in the pan where like, "Hey, you're in, you're in the flash today, someone else is in the flash tomorrow." Like we're all in this thing together. And so I'm, I'm really focused on solutions, getting ahead of it. And like I said, you know, 8 million, 8 million people may have gotten impacted, but how fast you turned that around, I thought was, was, I was grateful for that. But what happens when next time it isn't that fast?
And so my question is on, I'm, I'm trying to figure out what role government plays in this. And so my question is, I've, I've introduced this piece of legislation. It's called the National Digital Reserve Corps, which would recruit cybersecurity professionals to help during major incidents that occur. Based on your experience with large-scale, large-scale cyber incidents, do you think having a reserve of cybersecurity experts on standby would improve our response and recovery efforts?
Mr. Myers:
Thank you for the question, Congressman. And I think that, first, I would say transparency is the answer to your initial question. It is important for CrowdStrike and for others to be transparent when these things occur because every system is different, every product is different. They all have different components. So they can't all be uniformly thought of. So fixing one problem on one product isn't necessarily directly applicable to other products. That said, transparency is absolutely critical and why we endeavored to be so transparent when this occurred.
As far as having additional reserve forces on standby, I think in a situation where there's a cyber threat, and again, this was not an attack, a cyberattack, I think that that is certainly beneficial. There's, you know, never a situation where less skilled operators is going to be better.
Rep. Gonzales:
I'm out of time. I appreciate you coming to testify before the committee and once again being transparent throughout this process, the good, the bad, and the ugly. With that, I yield back.
Chairman Andrew Garbarino (R-NY):
I recognize the gentleman from South Carolina, Mr. Timmons, for five minutes of questions.
Rep. William Timmons (R-SC):
Thank you, Mr. Chairman. I appreciate you letting me weigh in on this committee. I appreciate your testimony regarding the changes that have been made to make sure this cannot happen again. For better or worse, the number one reason it probably won't happen again is damages. I mean, insurers have estimated in excess of $5 billion in damages to your customers. I'm sure that there's going to be lawsuits and settlements for days.
Can we talk about making the victims whole? I mean, you know, whether it's airlines, other critical infrastructure, what steps is CrowdStrike taking to, I guess, make it right other than making sure it doesn't happen again, which is fantastic, and your response has been very admirable. But how do we make sure that any other, any future incident is held accountable as it relates to making the victims of the, the breach, it wasn't a breach, making victims of the incident whole? How does that work?
Mr. Myers:
Thank you for the question, Congressman. We've been working with our customers to ensure that they are up and running. We've identified that as of July 29th, 99% of the sensors were back up and running. And we're working with our customers to ensure that we are able to help them through any issues that they are dealing with and continuing to support them in any way that they need.
Rep. Timmons:
I mean, tens of thousands, hundreds of thousands of people missed flights, businesses were inoperable for days or weeks. I mean, again, this isn't necessarily about CrowdStrike. This is about future cybersecurity incidents and creating a system through which people can be made whole. So in addition to getting people back up and running when their systems were down, I mean, you all have insurance policies. There's a wide variety of legal mechanisms that will create accountability. Are you able to speak to any of that, or is that something that your lawyers would probably tell you not to talk about?
Mr. Myers:
Congressman, I know people who were impacted by this as well. And as I said earlier, we're deeply sorry for what happened. We are working with customers to ensure that they have everything that they need to get back online. Most of them are back online and ensure that they have what they need to feel comfortable that they're working with CrowdStrike. We're continuing to rebuild that trust. Trust takes years to make and seconds to break, and we understand that we broke that trust and that we need to work to earn it back.
Rep. Timmons:
Do you think your customers care that this was an innocuous fat finger as opposed to an actual breach? I mean, they're still damages associated with both. So I mean, again, going forward, the global economy needs to have consequences for all types of cybersecurity shortcomings. So I mean, do you distinguish between a breach and this faulty update?
Mr. Myers:
Yes, I would say that there is a difference between a breach and when one...
Rep. Timmons:
That I can tell my constituents that missed flights and were stuck in airports for weeks that they'll care about? Probably not. I mean, so again, you don't want to talk about the damages and making victims whole. But I think that's an important part of this. You're taking additional steps, which are very important, to make sure it doesn't happen again. But at the end of the day, part of this is making it right with the people that missed flights, that weren't able to engage in commerce. And that's part of the conversation that we need to be having because that is the deterrent threat to the future. And future incidents will occur. And the reason that businesses that have major cybersecurity breaches end up settling for hundreds of millions, billions of dollars in certain circumstances, is because that is how they are made whole.
And again, this was a fat finger. So it's not the same as a breach. I get what you're saying, but the damages are still the same in many, in many respects. So we're going to, I'm sure, hear more about the manner in which your customers and their, their customers were made whole as a result of this incident. And I think that that's an important part of the story because ultimately, that's the accountability that our system provides to make sure it doesn't happen again, the deterrent threat. And with that, Mr. Chairman, I yield back.
(Second Round of Questioning Begins)
Rep. Carlos Gimenez (R-FL):
Thank you, Mr. Chairman. I want to kind of continue on what I ended up with, which was AI. And, and the problems that AI, of, does AI pose a threat right now? And what do you see as the threat from AI in the future, if you can, if you can elaborate on that?
Mr. Myers:
Absolutely. Thank you, Congressman. The threats from AI that I see today are primarily, and what we've seen adversaries using various AI for, whether it be large language models or stable diffusion or different algorithms that can be used to generate new content, has been primarily around disinformation, misinformation, and enabling faster research. In other words, threat actors have used artificial intelligence, LLMs, to automate writing scripts while they can use during an intrusion or during a ransomware operation.
Rep. Gimenez:
If I can, if I can interrupt, will that, will that mature into AI writing code that will be malicious in nature?
Mr. Myers:
We've, I've personally done some research in that area, and I think right now it's not there. You still need to be very familiar with the tool chain, and you need to be able to actually compile the code and debug it and understand where, where there's issues. But every day, this technology gets better. And it's something that we need to keep a close eye on to ensure that we understand how threat actors may use it, as well as good actors.
Rep. Gimenez:
Yeah, I mean, AI scares me. And that's why we have to be on top because the only, the only defense against AI in the future, I mean, if it gets totally mature and it just starts writing code, and it's, you have to have AI on your side that writes the counter code just as fast as it's getting the, this code. So it's, you're going to have this, you're going to have millions of attacks per day, and you have to defend against millions of attacks per day because these things will just generate attack after attack after attack. Now then you put that with quantum computing, now you really got a problem. So, where is your company on that? I mean, you're, you're a leader in cybersecurity. I'm concerned about the fact that something, a program that was supposed to protect systems against cyberattacks actually kind of destroyed the system they were trying to protect. Didn't destroy it, but certainly disrupted it. All right. So in the future, I can see this as being a tremendous problem and a tremendous risk. And it may be that the future says that we're going to have to, we can't be so connected anymore because of the vulnerabilities involved. Can you foresee such a future?
Mr. Myers:
I don't foresee a future where things get, need to be disconnected necessarily. But I think that we need to be very careful and thoughtful as we roll out artificial intelligence solutions. Happy to work with your staff and yourself to spend more time on that issue if you'd like.
Rep. Gimenez:
Well, I don't think it'd be in your best interest for us to disconnect. All right. But I'm not sure if it wouldn't be in our best interest sometime in the future, for security reasons, to disconnect somewhat, that somehow, no matter what comes in, that the system won't be disrupted. That...
Mr. Myers:
Oh, that's a hiccup. Oh, I got, I just got punched in the face, but I'm not going to get knocked out. All right. And that's the problem that we have. My, my fear is that an adversary, before they, they launch something, will try to knock us out. And I think we're vulnerable to that right now, that we can get knocked out. Our electrical systems can get knocked out, transportation systems can get knocked out, all at the same time, causing massive disruptions. And I, and I don't think anybody can tell me that, "No, that's not possible." It is possible, right?
Mr. Myers:
What I would suggest, sir, is that I think in the future, we can see that organizations will have their own AI workloads. They'll be deploying artificial intelligence to solve customer challenges, business challenges. And it won't be a handful of AIs that are being used across the globe. I think we'll see very localized artificial intelligence workloads. And this is something that we need to be thinking about, how do we secure those AI workloads into the future? Because adversaries can leverage that AI workload. They can poison the data that goes into AI training. So there's a whole new wave of horizon threats that pertain to AI and something that I think is a very critical thing for us to be talking about.
Rep. Gimenez:
Thank you. My, my time is up. I yield back.
Rep. Tony Gonzales (R-TX):
Thank you, Chairman. And thank you again for allowing me to speak today. Mr. Myers, I just would highlight to you, there are very few people in Congress that really understand this issue. And this is one of the committees, you mean, you have a chairman, you have members on this committee that are that want to find solutions, that want to get ahead. As, as Mr. Menendez just mentioned, you know, the future is already here. And I worry what role government is going to play. And I would just once again highlight to you, like this isn't just a one-off. The more that your team and your, your, you all can be working with our staff as we build out meaningful responses, either through the appropriations process or through legislation, I think is very critical because I do worry that they, that we'll get it wrong, right? Or we will be delayed in it. Or it will have meaningful intentions but have second and third-order impacts that may make it more difficult for you, whether it's, whether it's a self-inflicted kind of incident or whether it's an intrusion. So I just would highlight, you know, once again, thank you for coming and testifying. But you know, let's, let's work towards fixing this long-term on other issues that that happen.
The other, so the, the question I have for you is, is, from an industry standpoint, is it, is it more impactful that government get out of your way? I'm, I'm trying to frame it the right way, you know, without putting you on the spot because I want to get as real an answer as possible. How, maybe, maybe I'll frame it this way. In your dealing with CISA, as you went through this intrusion, as you went through this incident, I'm sure you probably dealt with them far more than you had in the past. What, what was, what was a takeaway that you know that you think we could improve on from a government response, government interacting with industry on dealing with a real-world situation to get things back on track? What was one of your takeaways that maybe this committee could work on?
Mr. Myers:
Thank you, Congressman. First, let me say it's an honor to be here. Thank you for having me. We work with CISA on a daily basis. We've been a, we were a plank holder at JCDC and have been working hand-in-hand across the US government, as well as other friendly governments, every single day. And the way that we succeed, in my mind, is through public and private partnership. We need to all be able to share and work off the same sheet of music. We track over 250 threat actors today that are coming from places like Iran, Russia, North Korea, but other countries that haven't made it into the news, so to speak, and a whole bevy of threat actors that are engaged in ransomware and data extortion operations, as well as activists who are looking to conduct hack and leak operations.
So I guess the one takeaway I would say is that in this situation, our job was to inform the government and inform your staffs about what was going on within CrowdStrike, being transparent so that they could know that this one wasn't a cyberattack, and two, what the impact was and what we were doing to remediate that. I think in the situation where there is a cyber incident, then the responsibilities change, and it becomes us supporting the government, helping to understand who these threat actors are, what they're after, and how to stop them. So it depends on the situation. But I would reiterate that private-public partnership is absolutely essential because this is a team sport, and we all are on the same team.
Rep. Gonzales:
I appreciate the response. And I, I would just close with this. This is the committee you want to work with because, guess what, this is where legislation is going to come out of that's either going to make your job easier or harder, right? And we, the intentions are going to be there, best intentions for our country and our allies to defend against these adversaries, some of them what you mentioned that are trying to kill us every single day. So that's what we're up against, and we need partners. And I want to make sure, from a legislative standpoint, we're getting it right. We're committed on our side, but we need partners, right? We, not everybody up here has the same level of expertise you try. I mean, I read that, I read the outage report right before I went to bed for a reason, right? I mean, it's very technical, right? And so, once again, thank you for coming and testifying. You're a brave man. And please continue to work with the committee as we find solutions. Thank you. I yield back.
Chairman Andrew Garbarino (R-NY):
I recognize myself for five minutes of questions. Mr. Myers, other cybersecurity providers, competitors of yours, have said that the access to the kernel, so much, how many updates you're pushing out daily, goes against industry standards and is, it's not safe. What would you say in response to that?
Mr. Myers:
I'm not aware of any industry standards that govern how or what to do with regards to any one operating system or best practice with that regard. I could tell you that when we launched CrowdStrike 13 years ago, we did so with the mission to stop bad things from happening to good people. And we have worked tirelessly in the last 13 years to ensure that our products, our services, and our intelligence information is the best possible product that our customers can consume. And I would say that we got it wrong in this case, and we are learning from what happened, and we've implemented changes to ensure that that doesn't happen again.
Chairman Garbarino:
I understand that. And would, would you think now, because I believe you testified before you do maybe a dozen a day updates because that's what you do. You find a threat, and you update the system to protect against that threat. You think it's, you will still continue to do 12 a day, or as, as many as are needed, or will this, will you be more, more, I don't even know if conservative is the right word, but more conservative in your, in your approach on how many updates you will have daily?
Mr. Myers:
We will continue to update our product with threat information as frequently as we need to to stay ahead of the threats that we're facing.
Chairman Garbarino:
And your, and in your belief, that that is access to the kernel as much as you have it and updates as much as you have it is, is needed for cyber, for your client's cybersecurity?
Mr. Myers:
I think, as we said earlier, speed does matter in this, in this domain to stay ahead of these threat actors. And the visibility that we get through the kernel, the performance that you gain through using the kernel, the ability to stop bad things, the enforcement mechanism that's provided through that kernel, and the anti-tamper to stop a threat actor... One of the threat actors that we track very closely is a group called Scattered Spider, who has been using techniques to elevate their privilege into the kernel to disable security tools on a regular basis. In order to stop that from happening, we will continue to leverage the architecture of the operating systems that we're on in the, in the most effective way that we can to stop those threats.
Chairman Garbarino:
Is it just unlucky that this update was for a Microsoft operating system? Could this have happened to any, pretty much any other operating system, right? Depend on the update. It was just an unlucky coincidence that it was Microsoft.
Mr. Myers:
The, I would say that a lot of businesses rely on Microsoft for their operating systems. And I think that's where the number of impacted systems from this update came from.
Chairman Garbarino:
But what I mean is, it was, the problem was with the update. It wasn't with this, with the Microsoft system. It was, it was with the update. It was faulty.
Mr. Myers:
This was a CrowdStrike issue.
Chairman Garbarino:
And I did want to, are you developing any, I want to make sure I got this in, are you developing any agentless technologies to scan for infrastructure remotely, which could avoid, avoid this type of outage in the future?
Mr. Myers:
Sorry, could you repeat that?
Chairman Garbarino:
Agent, are you developing any agentless technologies to scan infrastructure remotely, which could avoid this type of outage in the future?
Mr. Myers:
Thank you. We have a number of platforms that we provide to our customers, including attack surface management tools, which can scan without an agent or a sensor in place. But in order to detect and to prevent threats, you need to have that enforcement mechanism in place on the operating system to stop that from occurring.
Chairman Garbarino:
Okay. And just lastly, because I'm, I'm going to yield back in a second and let Mr. Swalwell go, is there now something in place because what happened, someone didn't catch it. There was an additional parameter in the channel B file before it was, it went through content interpreter or the validator. Is there now a process, is there something new in place to prevent that from happening again?
Mr. Myers:
Yes, sir. Thank you.
Ranking Member Eric Swalwell (D-CA):
Thank you. And I wanted to follow up on the line of questioning that Ms. Lee and I were kind of going back and forth with with regard to the kernel. Part of the discussion at the Microsoft Endpoint Security Summit involved reducing reliance on the kernel. My understanding is that Microsoft agreed at the summit to make additional capabilities available at the user level. Do you know the timeline for that process? And do you have a sense of how Microsoft and security vendors will engage to reduce the kernel and more additional
Ranking Member Swalwell:
Thank you. And I wanted to follow up on the line of questioning that Ms. Lee and I were kind of going back and forth with with regard to the kernel. Part of the discussion at the Microsoft Endpoint Security Summit involved reducing reliance on the kernel. My understanding is that Microsoft agreed at the summit to make additional capabilities available at the user level. Do you know the timeline for that process? And do you have a sense of how Microsoft and security vendors will engage to reduce the kernel and more additional activity, you know, to the user space? How would you reduce risk to the kernel and move it to the user space, I guess, is the question.
Mr. Myers:
Thank you. The, I don't have that timeline available. Happy to follow up with you on that. What I will say is that things can crash in user space too. And so this is not unique to the kernel space.
Ranking Member Swalwell:
What, trade one set of risks for another, right? It's, there, there's, there's some, there's definitely things that can break in user space as well as in the kernel space.
Mr. Myers:
Yes, sir.
Ranking Member Swalwell:
The relevant update in this incident only affected, as you've pointed out, Windows systems. But my understanding is that Apple's restrictions on kernel access might have prevented a similar incident from taking place on Mac systems. Do you view Apple's restrictions on kernel access to be beneficial, or do they negatively impact the effectiveness of security software like yours?
Mr. Myers:
We have security products that work on Windows systems, Apple systems, and Linux systems as well. We leverage all of the features of those various operating systems. They have pros and cons for each, and we leverage everything to have the most effective security solution for those platforms. The Windows architectures and open architectures that I mentioned earlier, Apple has a tighter supply chain. And with Linux, you have to pre-compile the kernel for every possible configuration of hardware that you would want to support. So there's, there's different features of each kernel.
Ranking Member Swalwell:
Would you agree, though, that if something crashes in the app space, it's limited in its effect to the app, whereas if something crashes with the kernel and Microsoft, it could crash the whole system?
Mr. Myers:
Yes.
Ranking Member Swalwell:
We've got about two and a half minutes left. Is there anything that we didn't cover that you think would be helpful for us to understand, anything that you wanted to just revisit and further articulate?
Mr. Myers:
I think it's important to note that this is not a cyberattack. This is something that happened within the system during an update process, which we've spent a considerable amount of thought and effort to ensure that this doesn't happen again. My concern is, if I may, these cyber threat actors that we're seeing across the globe, this is something that we need to be paying close attention to. We are seeing the constant evolution of those threat actors looking to subvert systems. They've moved into the identity space, stealing usernames and passwords, and are leveraging identity access to move into new environments and to conduct additional ransomware and data extortion attacks. So this is an area that continued, would like to continue working with the committee on. And anything that we can do to help...
Ranking Member Swalwell:
Great. Mr. Chairman, do you have anything you wanted to ask?
Chairman Garbarino:
No, I'm just saying he, speed with cybersecurity, we don't work with speed here in Congress, unfortunately. Speed, what's that? Yeah, you don't recognize that. All right, I yield back.
Chairman Garbarino:
I want to thank the witness for his valuable testimony and the members for their questions. I think this is a very good hearing. The members of the subcommittee may have some additional questions for you, and we would ask that you respond to these in writing pursuant to committee rule 7D. The hearing record will be held open for 10 days. Without objection, the subcommittee stands adjourned.