Null Pointer Exception Through Interrupts: Crowdstrike Technical Analysis

The world was shocked when cybersecurity firm “Crowdstrike”, a provider of endpoint protection software, released an update on Friday, July 19, 2024, that devastated millions of Windows computers worldwide.

In this article, I will examine what exactly happens, how it should never happen, and why 3rd party endpoint protection software itself is a dangerous security threat.

Windows public blue screens dominate the world on July 19

Before we dive deep into CrowdStrike and the fiasco that disrupted the lives of millions of people on July 19, 2024, we need to take a look at what CrowdStrike does and why Microsoft can’t be blamed for it.

“Endpoint protection” is the most vulnerable door in the cybersecurity world and describes software running on local machines to ensure that malware or unwanted code cannot be run.

Sound familiar?

Yes, it is. To put it bluntly, endpoint protection is just a fancy name for your good old antivirus and firewall software.

ZoneAlarm, Norton AntiVirus and similar scenarios were just a few of the scenarios that occurred in the late 90s and early 2000s. So with this case, I can say that we are back to the 90s 🙂

So how does endpoint protection work?

Endpoint protection systems consist of two main components: a back-end control center and an agent installed on the endpoints. Endpoints here mean mobile devices, computers, servers, and network devices.

The fundamental problem with such endpoint protection middleware has been the same for decades: it needs to embed itself deep into the operating system (Windows), bypassing the protection and security measures of the operating system.

This agent runs persistently on endpoints. It comes with a core application, including a user interface (GUI) and “sensors” that connect to the operating system to block processes and executions. If you try to run a program on Windows that the agent should block, a “sensor” is notified by the operating system about this execution, checks the executable, and of course blocks its execution. It then reports the blocked execution to the main application, which then reports it to the control center over the Internet.

This is a decade-old “challenge”: deeply embedded system software is crashing operating systems. With this type of “endpoint protection”, IT administrators can monitor the use of their equipment and prevent it from performing malicious or unwanted actions. This is essentially a surveillance system. The problem with such systems is that they need to be deeply embedded in the operating system. The operating system (in CrowdStrike’s case: Microsoft Windows) already provides its own protection mechanism.

In order for such an agent to interfere with system processes and “control” the entire operating system, the security software itself would need to bypass and disable a wide range of internal security mechanisms. This has led many to argue that Antivirus and Endpoint Protection are themselves a security threat. This was outlined in a Blackhat Europe 2008 whitepaper by Feng Xue titled “Attacking Antivirus”.

How CrowdStrike destroyed millions of systems

On July 19, 2024 at 04:09 UTC (around 09:09 UTC on July 18, 2024), CrowdStrike performed a regular release of a “sniffer” of its endpoint protection agent software.

This “sensor” comes in the form of a Windows device driver, as it needs to be hooked and nested deep within Windows. Something that only device drivers with very specific permissions can do. These device drivers are written in C and C++, just like Windows’ own kernel and kernel libraries (DLLs etc.).

“On July 19, 2024 at 04:09 UTC, as part of ongoing network operations, CrowdStrike released a sensor configuration update for Windows systems. Sensor configuration updates are an ongoing part of the Falcon platform’s protection mechanisms.

“This configuration update triggered a logic error that resulted in a system crash and blue screen of death (BSOD) on affected systems.”

Technical Analysis of CrowdStrike Network Outage

The device driver was installed by the CrowdStrike agent software as a uhs block like C-00000291-*.sys in the C:\Windows\System32\drivers\CrowdStrike\ folder where Windows drivers are located.

When this updated driver was installed, it attempted to access a memory address that did not exist. This resulted in a NullPointer Exception (NPE), which is a system exception that occurs when an application attempts to access memory that does not exist.

The Hex dump of the section where the Null Pointer Exception occurs (in the blue screen image above) highlights the section where the driver is trying to access invalid memory. Usually, Windows protects itself against such problems by crashing the application in question and giving the user an error message that it has crashed. However, because the CrowdStrike driver (they call it a “sensor”) is so deeply embedded in Windows, and bypasses Windows’ security measures, it has disabled the entire operating system as if we are back in 1994. This “sensor” is not a regular device driver like the ones that come with printers, webcams, keyboards, etc.

Why is such dangerous software written in C or C++ rather than languages ​​like Rust or Go that might better prevent these types of memory errors?

The Windows APIs required for this type of middleware are generally not designed for use by 3rd parties.

Antivirus, system tuning or customization, and endpoint protection software have something in common: they often use undocumented and unsupported Windows APIs that are only available by interacting with them directly at runtime – something you can currently only do reliably in C and C++ (of course, assembly language too).

Windows Defender doesn’t need “device drivers”; it’s built into Windows. From the early days of Windows all the way back to Windows XP, it wasn’t uncommon for device drivers in Windows 9x to fry the entire operating system. Antivirus software, graphics card drivers, and the like were notorious for causing blue screens and full system crashes with illegal memory access.

Such scenarios should be very rare, thanks to the protections Microsoft introduced in Windows XP and significantly increased in Windows 10.

Microsoft also introduced Windows Defender and Windows Firewall years ago. By integrating security software into the operating system, Microsoft ensures that the security mechanism itself does not become a security threat. Windows Defender, Microsoft’s own alternative to CrowdStrike and SentinelOne, is a core component of Windows, so it does not have to rely on dangerous device drivers and other “hacky” applications. All was well in the security world – until that day.

This mistake should never have happened…

This is deja vu for cybersecurity and puts 3rd party vendors of endpoint protection software in a difficult position.

Their business model is based on being able to “nest” or “connect” deeply into Windows and other operating systems. This is something only the operating system vendor can do safely. Even running such software securely on Linux is equally dangerous and risky.

The outages were no surprise – they were the beginning of a disaster

CrowdStrike has provided its customers with software for Windows that already has extreme threat potential in itself because it interacts with the Windows operating system in a dangerous way.

They probably didn’t take the necessary security measures to create and release such a dangerous software. Moreover, the fact that they released the software at midnight on a Friday was the step that triggered the surprise

This global outage was not a surprise.

As I mentioned, this crisis was unfortunately the beginning of the disaster.

How can you best protect yourself?

Use security tools and measures provided by your operating system vendor: Players are certain Microsoft, Apple or Linux. A software that is deeply embedded in the operating system, especially Windows, is itself a security threat. Therefore, security systems must be part of the operating system.

So where and how were you affected by the CrowdStrike incident?