# The Day 8.5 Million Devices Couldn't Boot -- and How Microsoft Rebuilt Recovery as a Security Surface
> The Windows Recovery Environment worked perfectly on July 19, 2024. That was the problem. How WinRE, Quick Machine Recovery, and the Windows Resiliency Initiative re-priced fleet-scale recovery.
*Published: 2026-05-12*
*Canonical: https://paragmali.com/blog/the-day-85-million-devices-couldnt-boot----and-how-microsoft*
*License: CC BY 4.0 - https://creativecommons.org/licenses/by/4.0/*
---
**On July 19, 2024, the Windows Recovery Environment worked exactly as designed -- and that was the problem.** WinRE assumed a human operator per machine, and CrowdStrike's Channel File 291 priced that assumption at 8.5 million endpoints. The Windows Resiliency Initiative -- Quick Machine Recovery, MVI 3.0, the user-mode endpoint security platform, Intune-surfaced WinRE state, Point-in-Time Restore, and Cloud Rebuild -- is Microsoft's first systemic admission that the recovery path is part of the security architecture. This article maps the architecture, the program, and the trade-off it cannot remove.
## 1. A Fleet That Cannot Boot Itself
At 04:09 UTC on July 19, 2024, CrowdStrike pushed a new Channel File 291 to its Falcon sensor on Windows. Forty-eight minutes later -- 04:57 UTC, give or take an hour depending on which time zone the failing devices happened to wake into -- the calls began. By the time CrowdStrike reverted the file at 05:27 UTC, roughly 8.5 million Windows endpoints were stuck in a bug-check loop on `csagent+0xe14ed`: a read-out-of-bounds page fault inside a kernel-mode driver registered as `SERVICE_SYSTEM_START` (`Start=1`), so it reloaded on every reboot [@crowdstrike-tech-details, @ms-security-jul27, @ms-crowdstrike-jul20, @wiki-cs-outage].
The fix was published almost immediately. "Boot to Safe Mode," it said. "Delete `C-00000291*.sys`. Reboot." If the volume was [BitLocker](/blog/bitlocker-on-windows-architecture-attacks-and-the-limits-of-/)-encrypted, find the recovery key first [@ms-kb5042421]. The instruction was technically correct. It was also a procedure for one machine. The Windows Recovery Environment that the procedure depended on -- WinRE -- worked exactly as it was designed to work, on every one of those 8.5 million devices [@ms-crowdstrike-jul20]. That was the problem.
Think about the engineering. The recovery partition was where it should be. The Boot Configuration Data store pointed at the right `winre.wim`. The two-failed-boots trigger fired. The blue Safe Mode tile rendered. The keyboard input handler took keystrokes. The NTFS read-write driver inside WinRE deleted the bad channel file. The reboot succeeded. Every line of code in the recovery path behaved exactly as the engineers in Redmond had specified. The architecture did not break.
What broke was the architecture's central assumption: that a person would be sitting in front of the screen.
This article makes the case that the assumption was a security choice as much as it was a usability choice, and that the cost of that choice was a denial-of-service event measured not in seconds of downtime but in person-days of triage. It walks the WinRE architecture as it actually exists on every Windows 11 device today; it walks the lineage that produced that architecture; it walks the failure mode that priced the architecture's blind spot; and it walks the program -- the Windows Resiliency Initiative -- that Microsoft began assembling in the months after the incident.
A second thesis follows from the first. *Recoverability is a security property.* A platform that cannot recover at scale cannot guarantee availability; a platform that cannot guarantee availability cannot keep its confidentiality and integrity promises either, because operations teams in the middle of a fleet-down event will eventually pull every encryption layer and every signing check that gets in their way. The two halves of the CIA triad we usually study -- confidentiality and integrity -- have spent decades crowding out the third. CrowdStrike forced the third one back onto the page.
If WinRE worked perfectly on July 19, 2024, what does it actually do? And how did a recovery primitive end up being the architecture's single point of human dependence? Those questions are next.
## 2. The Architecture: WinRE, `winre.wim`, `boot.sdi`, ReAgentC
Before we explain how WinRE failed at scale, we have to be precise about what WinRE *is*. Most engineers know it as the screen that appears after two bad boots. That description is correct and unhelpful. WinRE is a Windows Preinstallation Environment image -- `winre.wim` -- backed by a system deployment image ramdisk and managed by `ReAgentC.exe`, registered with the Windows Boot Manager via an entry in the Boot Configuration Data store [@ms-winre-tech-ref, @ms-reagentc, @ms-bcd]. Each of those four moving pieces does one job; together they make the recovery surface possible.
A small, self-contained Windows operating system used to install, deploy, and repair Windows desktop editions and Windows Server [@ms-winpe-intro]. WinPE is the substrate of Windows Setup, the install media's `boot.wim`, and `winre.wim`. The base image requires 512 MB of RAM and automatically reboots after 240 hours of continuous use on Windows 10 1803 and later [@ms-winpe-intro]. Originally released to manufacturing in 2002 by a Microsoft team that included Vijay Jayaseelan, Ryan Burkhardt, and Richard Bond [@wiki-winpe].
A small image-format file that the Windows Boot Manager uses to allocate a RAM disk into which a WIM image can be mounted at boot time. The WinRE BCD entry references `boot.sdi` through a `ramdiskoptions` element; the `osdevice` element then names `winre.wim` as the image to mount inside that RAM disk [@ms-bcd, @ms-winre-tech-ref].
The binary database that replaced `boot.ini` in Windows Vista. The BCD lives on the EFI System Partition on UEFI machines and is the data structure the boot manager reads to decide what to boot. Each entry is a typed collection of *elements* -- `device`, `osdevice`, `path`, `winpe`, `ramdiskoptions`, `recoverysequence`, and others -- manipulated with `bcdedit.exe` [@ms-bcd].
A dedicated GPT partition holding `winre.wim`, identified by partition Type ID `DE94BBA4-06D1-4D40-A16A-BFD50179D6AC` and recommended for placement immediately after the Windows partition. The minimum size is 300 MB, with 250 MB of free space recommended to accommodate future updates [@ms-uefi-gpt]. On Image Configuration Designer media, this partition is the default layout; clean Setup may instead use a `\Recovery\WindowsRE` folder inside the Windows partition [@ms-winre-tech-ref].
Restated in the order a practitioner encounters them on disk, the four pieces are:
1. **The recovery partition.** The default UEFI/GPT layout from the Image Configuration Designer places a Windows RE Tools partition after the Windows partition, sized to hold `winre.wim` with headroom for cumulative-update growth [@ms-uefi-gpt]. The GPT Type ID `DE94BBA4-06D1-4D40-A16A-BFD50179D6AC` lets `bootmgr` find the partition without depending on the Windows volume's drive letter. A `\Recovery\WindowsRE` folder inside the OS volume is an equally valid alternative; some OEMs use one, some the other.The variability is invisible at runtime: `bootmgr` follows the BCD, not the disk layout. But it matters at provisioning time. Always check `reagentc /info` after deployment to know which arrangement you have, because the *Microsoft-recommended fix for "winre.wim is too small after a cumulative update"* (KB5028997) depends on which partition the image lives in.
2. **`winre.wim`.** A customised WinPE image. The lineage goes back to Windows PE 1.0, RTMed in 2002 from Windows XP RTM [@wiki-winpe]. Today's `winre.wim` is built from Windows 10 / 11's WinPE 10 line and includes the recovery shell, Startup Repair, System Restore (when enabled on the host), command prompt, and a curated list of optional drivers. The base image still inherits the WinPE rules: 512 MB minimum RAM, 240-hour reboot cap on Windows 10 1803+ [@ms-winpe-intro].
3. **`boot.sdi`.** Sits on the recovery partition (or in `\Recovery\WindowsRE\`) and acts as a fixed-size container into which the boot manager creates a RAM disk at boot time [@ms-bcd].The `.sdi` extension stands for *System Deployment Image*, the same file format used by older Windows Deployment Services workflows in which a thin ramdisk holds a `boot.wim` for PXE installs. The RAM disk is where `winre.wim` is mounted. `boot.sdi` is small (a few megabytes), unmodifiable in normal operation, and one of the parsers later abused by the BitUnlocker chain [@ms-bitunlocker-blog]; we return to that in Section 9.
4. **`ReAgentC.exe`.** The in-box management tool. Microsoft Learn documents the supported switches: `/info`, `/enable`, `/disable`, `/setreimage /Path `, `/boottore`, `/setbootshelllink`, and the now-deprecated `/setosimage` (no longer used on Windows 10 or later) [@ms-reagentc]. The same page notes that for *offline* operations on WinPE 2.x/3.x/4.x images, administrators must instead use `Winrecfg.exe` from the Windows Assessment and Deployment Kit -- a clue that the *online* mode of `ReAgentC.exe` predated the offline mode. The tool has shipped since at least Windows 7; the precise RTM month is not surfaced on Microsoft Learn today.The web is full of confident claims that `ReAgentC.exe` first shipped in Vista, Windows 7, or Windows 8. The safe attribution is "Windows 7 onwards" because that is the era when the recovery-partition + ReAgentC model became the supported default. Microsoft Learn does not name an exact ship version, and the AI summaries that do are inferring from circumstantial evidence [@ms-reagentc].
All four pieces have to cooperate at the worst possible moment: when the Windows partition refuses to boot. The question for the next section is the literal handoff. How does the firmware end up running `winre.wim`?
## 3. The Mechanism: How a WinRE Boot Actually Happens
There is a sentence that appears in dozens of TechNet-era guides and AI summaries: *Windows boots WinRE by running `winload.exe /recovery`.* That sentence is wrong. There is no `/recovery` switch on `winload.efi` or `winload.exe`. The BCD Boot Options Reference enumerates every legal element on a boot entry, and `recoverysequence` is one of them; a command-line switch with that name is not [@ms-bcd]. WinRE is selected through the BCD, not through a flag passed to the loader.
> **Note:** The BCD Boot Options Reference defines every element on a boot entry: `device`, `osdevice`, `path`, `description`, `recoverysequence`, `winpe`, `ramdisksdidevice`, `ramdisksdipath`, and a few dozen others [@ms-bcd]. None of them is exposed as a `winload.exe /recovery` command-line flag. The recovery handoff happens entirely inside the boot manager, before `winload.efi` ever runs.
Walk the literal boot sequence on a UEFI machine [@ms-winre-tech-ref, @ms-bcd]:
1. Firmware passes control to `bootmgfw.efi` on the EFI System Partition. (On legacy BIOS, it would be `bootmgr` from the active partition.)
2. The boot manager reads the BCD store. There is one entry of type *Windows Boot Manager* and one or more entries of type *Windows Boot Loader*.
3. The OS loader entry carries an element called `recoverysequence`, set to the GUID of a *separate* BCD entry. That separate entry is the WinRE configuration.
4. On a normal boot, the boot manager loads the OS entry's `path` (`\Windows\System32\winload.efi`) against the OS volume named in `device`/`osdevice`, and `winload.efi` brings up the kernel.
5. On a recovery trigger -- two failed boots, a corrupted system file, an explicit `reagentc /boottore`, or the user choosing *Restart* from the Advanced Startup menu -- the boot manager instead follows `recoverysequence` to the WinRE entry.
6. The WinRE entry's elements look like this: `winpe Yes`, `osdevice ramdisk=[recovery]\Recovery\WindowsRE\Winre.wim,{ramdiskoptionsguid}`, `device ramdisk=[recovery]\Recovery\WindowsRE\Winre.wim,{ramdiskoptionsguid}`, and `path \Windows\System32\Boot\winload.efi`. The `ramdiskoptions` element it points to in turn carries `ramdisksdidevice` and `ramdisksdipath` (`\Recovery\WindowsRE\boot.sdi`).
7. The boot manager creates a RAM disk backed by `boot.sdi`, mounts `winre.wim` inside it, and starts `winload.efi` against that ramdisk. From `winload.efi`'s point of view, the OS being booted is the one inside `winre.wim`. The kernel comes up in the RAM disk and presents the Windows RE entry-point UI.
flowchart TD
F[UEFI firmware] --> BM[bootmgfw.efi on ESP]
BM --> BCD[Read BCD store]
BCD --> CHK{Trigger fired?}
CHK -- No --> OS[OS loader entry, winload.efi, Windows partition]
CHK -- Yes --> RS[Follow recoverysequence GUID]
RS --> WRE[WinRE BCD entry: winpe Yes, osdevice ramdisk=...winre.wim]
WRE --> RD[Allocate RAM disk from boot.sdi]
RD --> MNT[Mount winre.wim into RAM disk]
MNT --> WL[winload.efi loads WinPE kernel]
WL --> UX[WinRE entry-point UI]
The five auto-trigger conditions are enumerated verbatim in the Windows RE Technical Reference [@ms-winre-tech-ref]:
1. Two consecutive failed attempts to start Windows.
2. Two consecutive unexpected shutdowns within two minutes of boot completion.
3. Two consecutive system reboots within two minutes of boot completion.
4. A [Secure Boot](/blog/secure-boot-in-windows-the-chain-from-sector-zero-to-userini/) error (except for issues related to `Bootmgr.efi`).
5. A BitLocker error on touch-only devices.
flowchart LR
A[Two failed boots] --> ENT[Enter WinRE]
B[Two unexpected shutdowns within 2 min of boot] --> ENT
C[Two reboots within 2 min of boot] --> ENT
D[Secure Boot error -- not Bootmgr.efi] --> ENT
E[BitLocker error on touch-only device] --> ENT
Walking the BCD elements themselves makes the absence of any `/recovery` switch visible. Here is a minimal model of what the boot manager actually consumes.
{`
// Paraphrased from the BCD Boot Options Reference. Real bcdedit output is text,
// but the boot manager reads it as a typed key/value store.
const bcd = {
bootmgr: {
type: 'Windows Boot Manager',
default: '{current}',
displayorder: ['{current}'],
},
'{current}': {
type: 'Windows Boot Loader',
device: 'partition=C:',
osdevice: 'partition=C:',
path: '\\\\Windows\\\\system32\\\\winload.efi',
description: 'Windows 11',
recoverysequence: '{a1b2-...-winre-guid}',
recoveryenabled: 'Yes',
},
'{a1b2-...-winre-guid}': {
type: 'Windows Boot Loader',
device: 'ramdisk=[\\\\Device\\\\HarddiskVolume4]\\\\Recovery\\\\WindowsRE\\\\Winre.wim,{ramdiskopts}',
osdevice: 'ramdisk=[\\\\Device\\\\HarddiskVolume4]\\\\Recovery\\\\WindowsRE\\\\Winre.wim,{ramdiskopts}',
path: '\\\\Windows\\\\system32\\\\Boot\\\\winload.efi',
description: 'Windows Recovery Environment',
winpe: 'Yes',
nx: 'OptIn',
},
'{ramdiskopts}': {
type: 'Device Options',
description: 'Ramdisk Options',
ramdisksdidevice: 'partition=\\\\Device\\\\HarddiskVolume4',
ramdisksdipath: '\\\\Recovery\\\\WindowsRE\\\\boot.sdi',
},
};
// The boot manager picks one of these entries, depending on whether
// recoverysequence has been activated. No command-line flag is involved.
function bootDecision(failureCount, secureBootError, bitlockerError) {
if (failureCount >= 2 || secureBootError || bitlockerError) {
const winreGuid = bcd['{current}'].recoverysequence;
return bcd[winreGuid];
}
return bcd['{current}'];
}
const chosen = bootDecision(2, false, false);
console.log('Loader path the boot manager invokes:');
console.log(' ' + chosen.path);
console.log('Backing device:');
console.log(' ' + chosen.osdevice);
console.log('winpe flag (Yes means "boot a WIM into a ramdisk"):');
console.log(' ' + (chosen.winpe || '(unset, normal OS boot)'));
`}
That is the entire mechanism. Two failed boots flip an in-BCD counter; the boot manager follows `recoverysequence` instead of the default loader path; the WinRE entry mounts `winre.wim` in a RAM disk; the kernel inside `winre.wim` comes up. No flags, no shells, no scripts.
Now we know what WinRE is and how it boots. The remaining historical question is how this architecture *came to be*, and what about it did not change between 2007 and July 19, 2024.
## 4. Historical Origins: From the Recovery Console to the Recovery Partition (2000-2012)
Every architectural choice in WinRE was a response to something that did not work the year before. Walk the four pre-WRI generations of Windows recovery and the story is one long relaxation of the assumption that recovery requires physical media.
### Generation 1: Emergency Repair Disk (NT 3.x and 4.0, 1993-2000)
A floppy disk plus a `%SystemRoot%\repair` directory contained snapshotted SYSTEM, SOFTWARE, SAM, and SECURITY registry hives [@wiki-recovery-console]. The administrator booted from the three Windows NT Setup floppies, pressed `R` for Repair, fed the floppy when prompted, and Setup wrote the snapshotted hives back over the damaged on-disk copies. ERD repaired the registry, nothing more. If `NTOSKRNL.EXE` itself was missing, the operator was reduced to a DOS floppy plus `EXPAND` from the install CD. The architecture's failure mode was the obvious one for a floppy-based snapshot system: the floppy got lost; the snapshot was stale; the scope was too narrow.
The Windows NT 3.x and 4.0 recovery mechanism: a snapshot of the registry hives written to a floppy by `RDISK.EXE` plus a small `%SystemRoot%\repair` folder. Restored only the registry; required the NT Setup floppies to boot. Wikipedia's *Recovery Console* article identifies the Recovery Console as ERD's successor [@wiki-recovery-console].
### Generation 2: Recovery Console (Windows 2000, February 17, 2000)
The Recovery Console replaced the binary "restore the snapshot" decision with a programmable shell. Boot from the Windows 2000 or XP install CD; choose Repair; the operator landed in a `cmd.exe`-shaped environment with around three dozen internal commands: `copy`, `del`, `attrib`, `chkdsk`, `fixboot`, `fixmbr`, `bootcfg`, and the rest [@wiki-recovery-console]. Authentication required the local Administrator password; filesystem access was sharply constrained (read-only by default; on the boot volume only the root and `%SystemRoot%` were writable, unless Group Policy relaxed those limits).
The Windows 2000/XP/Server 2003 command-line repair shell. Initial release February 17, 2000; superseded by the Windows Recovery Environment in Windows Vista. Loadable from the install CD or installable as a startup option via `winnt32 /cmdcons`. Wikipedia lists Windows Recovery Environment as its named successor [@wiki-recovery-console].
The Recovery Console did not fail technically. It failed *culturally*. By 2005 the Windows administrator population had shifted decisively to GUI tools. A 2005 user with a corrupt `WINLOAD.EXE` and no install CD had no path to repair the box without buying replacement media. There was no automatic-repair logic and no on-disk presence; the install CD was always required, and every fix demanded muscle memory the typical administrator no longer had.
### Generation 3: WinRE on Installation Media (Windows Vista, January 2007)
Vista shipped a full GUI recovery environment built on the brand-new Windows PE 2.0 [@wiki-winpe]. `winre.wim` carried Startup Repair (a probe-and-fix playbook for boot failures), System Restore (now backed by the Volume Shadow Copy Service), Complete PC Restore, Windows Memory Diagnostic, and a command prompt for the cases nothing else fit. Vista was also the version that introduced the Boot Configuration Data store and `bootmgr`, replacing `NTLDR` and the plain-text `boot.ini` [@ms-bcd]. The same BCD that today still routes the recovery handoff was written for Vista.The Microsoft Learn "Vista WinRE Overview" page in the previous-versions archive (`cc766056`) is now misdirected and renders an unrelated USMT migration topic instead of the original article. The load-bearing claim that WinRE was introduced in Vista is independently supported by the Windows PE Wikipedia article's version table (WinPE 2.0 built from Vista RTM) and by Microsoft Learn's *Push-button reset overview*, which dates Push-Button Reset to Windows 8 and frames it as built on the existing WinRE architecture [@wiki-winpe, @ms-pbr-overview].
Vista WinRE had two architectural problems that the next generation fixed. OEMs were free to put `winre.wim` wherever they wanted on disk; there was no standard partition. And the install DVD remained the fallback for any user whose OEM had not pre-installed WinRE -- which, by 2010, was most users, none of whom still owned the DVD.
System Restore is itself a sub-thread worth noting. It first shipped in Windows ME (year 2000), was re-implemented atop VSS in Vista, and remained off by default on Windows 10 and 11 [@wiki-system-restore]. The Vista move made it callable from WinRE even when the host Windows would not boot -- a property that, twenty-five years later, Point-in-Time Restore is re-engineering for the cloud.
### Generation 4: Recovery Partition + ReAgentC + BCD `recoverysequence` (Windows 7, 2009; standardised in Windows 8 and beyond)
This is the architecture every Windows 11 device still runs.
Windows 7 dropped `winre.wim` onto a dedicated recovery partition with a GPT Type ID that lets `bootmgr` find it without depending on the Windows volume's drive letter [@ms-uefi-gpt]. `ReAgentC.exe` became the in-box management tool [@ms-reagentc]. The BCD `recoverysequence` element became the mechanism by which the OS loader entry points at the WinRE entry. The two-failed-boots trigger entered the Windows RE Technical Reference's enumeration of automatic conditions [@ms-winre-tech-ref].
Generation 4 *did not fail*. The five auto-trigger conditions still fire on Windows 11 24H2. ReAgentC's switches are still the supported management surface. The recovery-partition GPT Type ID is still `DE94BBA4-06D1-4D40-A16A-BFD50179D6AC`. It is the architectural floor every later generation extends, including Quick Machine Recovery.
What Generation 4 *did not solve* was the cost of recovery at fleet scale. WinRE-on-disk handled one machine perfectly; it had nothing to say about ten thousand machines, each still bounded by the time it took to walk to a desk.
gantt
dateFormat YYYY
axisFormat %Y
section Pre-WinRE
Emergency Repair Disk (NT 3.x / 4.0) :1993, 2000
Recovery Console (Windows 2000 onwards) :2000, 2008
section WinRE
WinRE on installation media (Vista) :2007, 2009
Recovery partition + ReAgentC (still current) :2009, 2026
section Recovery flavours
Push-Button Reset (Windows 8 onwards) :2012, 2026
Autopilot Reset (Win 10 1709) :2017, 2026
Quick Machine Recovery (24H2) :2025, 2026
Intune Remote Recovery / Cloud Rebuild :2025, 2026
A few parallel paths deserve naming. Push-Button Reset, introduced in Windows 8 in 2012, gave consumers an in-WinRE "Refresh" or "Reset"; image-less reset in Windows 10 and Cloud Download in Windows 10 version 2004 (May 2020) made the reset progressively less dependent on locally-staged install images [@ms-pbr-overview]. Autopilot Reset, shipped in Windows 10 1709 (October 2017), let Intune issue an MDM-initiated wipe-and-rebuild that preserved the device's Entra ID join. Microsoft Diagnostics and Recovery Toolset (DaRT) -- the descendant of Winternals ERD Commander acquired in 2006 and shipped under MDOP starting July 2007 (MDOP 2007), with subsequent releases through MDOP 2008 (April 2008) -- gave Software Assurance customers a richer enterprise tool on top of WinPE [@wiki-mdop-dart]. Older recovery mechanisms quietly aged out: Last Known Good Configuration was no longer the default boot-failure response on Windows 8 onward, and the deprecated-features lifecycle framework is the canonical place to track such retirements today [@ms-deprecated].
By the early 2010s, the architecture that still runs on every Windows 11 device today was largely in place [@ms-winre-tech-ref, @ms-reagentc]. None of these tools gave WinRE permission to call Windows Update from inside the recovery environment. That gap is the next chapter.
## 5. The Forcing Function: July 19, 2024
We know what WinRE is. We know how it boots. We can now see the CrowdStrike incident as the architecture's stress test. The headline numbers are well-rehearsed at this point; what matters here is the technical cause, the kernel-resident dependency it expressed, and the procedure Microsoft published.
### The fault
CrowdStrike's Falcon sensor for Windows version 7.11, released in February 2024, introduced a new IPC Template Type used by behavioural detection logic [@crowdstrike-rca-pdf]. The Template Type *declared* twenty-one input parameter fields. The integration code that invoked the in-driver Content Interpreter to evaluate Template Instances against host activity *supplied only twenty inputs* [@crowdstrike-rca-pdf]. For more than four months, Channel File 291 contained no Template Instance whose criterion read the twenty-first field. That made the mismatch latent.
At 04:09 UTC on July 19, 2024, CrowdStrike pushed a new Channel File 291 containing a Template Instance that referenced the twenty-first field with a non-wildcard matching criterion [@crowdstrike-rca-pdf, @crowdstrike-tech-details]. The Content Interpreter loaded the instance, looked up the twenty-first input pointer in its input-pointer array, and read past the end of that array. Sensors running 7.11 or later that received the update between 04:09 and 05:27 UTC tripped the latent out-of-bounds read [@crowdstrike-tech-details].
### The crash
Microsoft's Windows Error Reporting analysis, published in the security blog on July 27, 2024, recorded the global crash signature as `nt!KeBugCheckEx` followed by `nt!KiPageFault` and then `csagent+0xe14ed`, with `r8=ffff840500000074` as the invalid pointer that the read tried to dereference [@ms-security-jul27]. Microsoft confirmed that the analysis matched CrowdStrike's own conclusion: a read-out-of-bounds memory safety error in the `csagent.sys` driver.
flowchart TD
A[Falcon 7.11 ships in Feb 2024 with IPC Template Type declaring 21 fields] --> B[Integration code supplies only 20 inputs]
B --> C[Latent OOB potential -- no instance references field 21]
C --> D[July 19 04:09 UTC: new Channel File 291 adds non-wildcard 21st-field criterion]
D --> E[Content Interpreter reads input-pointer index 20]
E --> F[Page fault at csagent+0xe14ed]
F --> G[nt!KiPageFault -> nt!KeBugCheckEx]
G --> H[Bug check; system reboots]
H --> I[csagent.sys reloads -- registered SERVICE_SYSTEM_START Start=1 -- bug check again]
I --> J[Boot loop on 8.5 million endpoints]
### The kernel-resident dependency
`csagent.sys` loaded early in boot. Microsoft's WER post-mortem shows the driver registered with `REG_DWORD Start 1` -- the `SERVICE_SYSTEM_START` class, loaded by the kernel before user-mode comes up [@ms-security-jul27]. That placement is the entire point of a kernel-mode security agent: it has to instrument the kernel boundary at the moment user-mode would otherwise be invisible to it. The cost of that placement is that when an early-boot driver page-faults, the bug check happens *before* the operating system is interactive. The remediation -- *delete `C-00000291*.sys`* -- could not be issued from a running Windows, because there was no running Windows.
### The recovery procedure
Microsoft published KB5042421 within hours [@ms-kb5042421]. The text reduced to three steps: boot to Safe Mode (which on Windows 11 means letting WinRE select Safe Mode from the *Advanced startup options* tree); delete `C:\Windows\System32\drivers\CrowdStrike\C-00000291*.sys`; reboot. For BitLocker-encrypted volumes the procedure had a fourth, preliminary step: surface the recovery key. KB5042421 walks the user through the Entra ID self-service flow at `aka.ms/aadrecoverykey`: log on from a phone, choose Manage Devices, View BitLocker Keys, Show recovery key [@ms-kb5042421, @aka-ms-aadrecoverykey].
The instruction was correct. It was also unambiguously per-machine.
"We currently estimate that CrowdStrike's update affected 8.5 million Windows devices, or less than one percent of all Windows machines." -- Microsoft, *Helping our customers through the CrowdStrike outage*, July 20, 2024 [@ms-crowdstrike-jul20].
### The bottleneck
Each device's recovery was a function of *time-to-physical-access*, plus *time-to-BitLocker-key*, plus *time-to-keyboard*. None of those terms scaled. A laptop on a desk that the owner happened to be near recovered in five minutes. A laptop on a desk where the owner was on holiday recovered when someone arrived to swipe their badge. A server in a remote data centre recovered when a hand reached the iLO or KVM. A point-of-sale device in a checked-bag-only baggage hall recovered when someone wheeled a USB keyboard out to it. Multiply by 8.5 million.
The architecture that delivered Safe Mode to every one of those devices did exactly what its 2009 specification said it would do. The architecture that delivered Safe Mode to every one of those devices left enterprises stranded for days. Both sentences are true. The contradiction is the whole point.
> **Note:** WinRE booted correctly. The Safe Mode tile rendered. The two-failed-boots trigger fired. The recovery partition was where it should be. The BCD `recoverysequence` led to the right `winre.wim`. The keyboard handler took keystrokes. Every line of code did what it was specified to do. The single unwritten line of the specification -- *one operator, please* -- was the line that did not scale.
The instruction was correct, the procedure was published within hours, and the floor was on fire for days. The next question -- the one Microsoft was already being asked at WESES, the closed-door September 10, 2024 endpoint-security partner summit [@ms-weses] -- was whether the floor could not be on fire next time.
## 6. The Breakthrough: Quick Machine Recovery
Quick Machine Recovery, announced at Microsoft Ignite on November 19, 2024 [@ms-wri-ignite-2024] and generally available on Windows 11 24H2 build 26100.4700+ in August 2025 per the November 18, 2025 update [@ms-wri-ignite-2025], did not add any new *technology* to WinRE that had not been in WinPE since 2002. Networking drivers, DHCP clients, HTTPS stacks: all of these were already in `winre.wim`'s base image, inherited from the WinPE Optional Components that have shipped with the OS for two decades [@ms-winpe-intro]. What QMR added was an *answer to a question WinRE had never been asked*: when you are inside the recovery environment with no operator at the keyboard, who do you call?
The Windows 11 24H2 feature, available on build 26100.4700 or later, that lets WinRE establish network connectivity from inside the recovery environment, query Windows Update for a remediation matching the current failure signature, download and apply that remediation, and reboot -- all without requiring an operator at the keyboard [@ms-qmr]. Announced at Microsoft Ignite on November 19, 2024 [@ms-wri-ignite-2024]; first shipped in Windows 11 Insider Preview build 26120.3653 on March 28, 2025 [@ms-qmr-insider-mar2025]; generally available in August 2025 [@ms-wri-ignite-2025].
### The five-phase loop
Microsoft Learn documents QMR as five phases [@ms-qmr]:
1. **Crash detection.** The same two-failed-boots trigger already in the Windows RE Technical Reference [@ms-winre-tech-ref] fires the recovery path.
2. **Boot to recovery.** The existing BCD `recoverysequence` mechanism from Section 3 routes the system into WinRE.
3. **Network connection.** WinRE establishes wired Ethernet, or WPA/WPA2 password-based Wi-Fi using a credential pre-staged via `reagentc.exe /SetRecoverySettings`. As of the Microsoft Learn page's current wording, *only* wired and WPA/WPA2 password-based wireless are supported [@ms-qmr]; enterprise certificates and WPA3-Enterprise are on the November 18, 2025 roadmap but not yet shipped [@ms-wri-ignite-2025].
4. **Remediation.** The recovery environment scans Windows Update for a published remediation matching the device's failure signature, downloads it, and applies it.
5. **Reboot.** On success, the device boots normally. On no-match, the device can either present the manual recovery menu (the *one-time scan* mode, the default for unmanaged systems) or loop with a configurable interval (the *looped* mode) until either a remediation arrives or the operator-set total wait time expires [@ms-qmr].
sequenceDiagram
participant D as Device (OS)
participant W as WinRE
participant N as Network
participant WU as Windows Update
participant O as OS partition
D->>W: Two failed boots -> follow recoverysequence
W->>N: Acquire Ethernet or WPA2 Wi-Fi
W->>WU: Query for remediation matching failure signature
WU-->>W: Remediation package (or "none found")
alt Remediation available
W->>O: Apply remediation to OS partition
W->>D: Reboot
D-->>D: Normal boot succeeds
else None found, one-time mode
W->>D: Present manual recovery menu
else None found, looped mode
W-->>W: Sleep wait_interval, retry until total_wait_time
end
### The default-on/off matrix
The Microsoft Learn QMR page is explicit on defaults [@ms-qmr]. Cloud remediation is enabled by default, with one-time scan auto-remediation, on systems that are not under enterprise management -- Windows Home and unmanaged Pro. It is disabled by default on enterprise-managed systems -- Windows Enterprise, Education, and managed Pro. The rationale follows from how those populations think: enterprise administrators want to gate cloud remediation behind their own deployment-ring process, and consumers benefit from the default-on behaviour because they do not have a ring process at all. The same Microsoft Learn page documents an Intune Settings Catalog policy under *Remote Remediation > Enable Cloud Remediation* for administrators who want to switch the policy on at the tenant level [@ms-qmr].
### The test-mode flow
QMR ships with a dry-run mechanism. `reagentc.exe /SetRecoveryTestmode` configures the WinRE entry for a simulated recovery cycle; `reagentc.exe /BootToRe` triggers the cycle on the next reboot; the simulated remediation appears in Settings > Windows Update > Update history rather than mutating the production OS [@ms-qmr]. Microsoft suggests using the test mode to validate the per-device QMR configuration before relying on it in production.
### The pseudocode
The five phases collapse into a short loop. The version below is paraphrased from the Microsoft Learn QMR page [@ms-qmr] and shows how the two settings interact.
{`
// Paraphrased from the Microsoft Learn QMR specification.
const config = {
cloud_remediation_enabled: true, // default on Home/unmanaged Pro
auto_remediation_mode: 'looped', // 'one_time' | 'looped'
total_wait_time_minutes: 60,
wait_interval_minutes: 10,
wifi: { ssid: 'corp-recovery', psk: '***', encryption: 'WPA2' },
};
function detectFailureSignature() {
return { driver: 'csagent.sys', offset: '0xe14ed', signature: 'oob-read' };
}
function scanWindowsUpdate(signature) {
if (signature.driver === 'csagent.sys' && signature.signature === 'oob-read') {
return { id: 'qmr-csagent-291', action: 'delete', path:
'C\\\\Windows\\\\System32\\\\drivers\\\\CrowdStrike\\\\C-00000291*.sys' };
}
return null;
}
function qmrEnterRecovery() {
console.log('Phase 1: crash detected (two failed boots)');
console.log('Phase 2: booted into WinRE via BCD recoverysequence');
if (!config.cloud_remediation_enabled) {
console.log('Cloud remediation disabled; falling back to Startup Repair');
return;
}
console.log('Phase 3: acquiring network (' + config.wifi.encryption + ' Wi-Fi)');
const sig = detectFailureSignature();
let elapsed = 0;
while (true) {
console.log('Phase 4: scanning Windows Update for remediation matching ' + sig.driver);
const remediation = scanWindowsUpdate(sig);
if (remediation) {
console.log(' -> Applying ' + remediation.id + ' (delete ' + remediation.path + ')');
console.log('Phase 5: reboot into repaired Windows');
return;
}
if (config.auto_remediation_mode === 'one_time') {
console.log('No remediation found; presenting manual recovery menu');
return;
}
elapsed += config.wait_interval_minutes;
if (elapsed >= config.total_wait_time_minutes) {
console.log('Looped mode exhausted; falling back to manual recovery menu');
return;
}
console.log(' -> No match; sleeping ' + config.wait_interval_minutes + ' min');
}
}
qmrEnterRecovery();
`}
### The counterfactual
Had QMR existed on July 19, 2024, the per-device labour would have been zero. Microsoft and CrowdStrike would have published a Windows Update remediation that deletes `C-00000291*.sys`; every affected device would have entered WinRE on its second failed boot, picked up the remediation, applied it, and rebooted. The 8.5-million-device fleet cost would have collapsed from operator-days to network-minutes. The CrowdStrike RCA published August 6, 2024 documents that the fault-to-rollback time was 78 minutes [@crowdstrike-tech-details, @crowdstrike-rca-pdf]; QMR would have made *time-to-rollback* and *time-to-fleet-recovery* the same number, plus the per-device Windows Update transit. That is the empirical case Microsoft is making.
> **Key idea:** Quick Machine Recovery did not add new technology to WinRE. It added a question. WinRE has always had networking drivers; it had never been told it had permission to phone home. The technical innovation is policy, not code -- the *Windows Update endpoint* framing is a commitment that the recovery environment may, in well-defined circumstances, act on behalf of the operator who is not there.
QMR re-priced the per-device cost of recovery from O(N) to roughly O(1). But QMR alone does not explain why Microsoft is calling this the *Windows Resiliency Initiative* rather than the *Quick Machine Recovery Release*. The next section unpacks the five layers WRI puts around QMR.
## 7. The Program: The Windows Resiliency Initiative as Five Layers
WRI is not one feature. It is a layered program. Each layer is a Microsoft-named deliverable with a Microsoft-cited source. The temptation, on reading any single WRI blog post, is to confuse the layer with the program. The layers are concentric. They are also dated.
Walk the five layers. Each has a Microsoft term, a primary anchor, and a published status as of November 18, 2025.
| Layer | Microsoft term | Anchor | Status as of Nov 18, 2025 |
|---|---|---|---|
| Prevent: stop bad updates leaving the partner | Safe Deployment Practices (SDP), part of **MVI 3.0** | [@ms-wri-ignite-2024], [@ms-mvi], [@ms-wri-jun-2025] | Effective April 1, 2025 [@ms-wri-ignite-2025] |
| Prevent: stop bad code being kernel-resident | **Windows endpoint security platform** (user-mode antivirus) | [@ms-wri-ignite-2024], [@ms-wri-jun-2025], [@ms-wri-ignite-2025] | Private preview July 2025; named partners in [@ms-wri-jun-2025] |
| Manage: see the incident at scale | **Intune surfaces WinRE state**; Mission Critical Services for Windows | [@ms-wri-ignite-2025] | Coming soon |
| Recover: heal the unbootable machine | **Quick Machine Recovery** | [@ms-wri-ignite-2024], [@ms-qmr], [@ms-wri-ignite-2025] | GA August 2025 |
| Recover: rebuild without shipping hardware | **Point-in-Time Restore**, **Cloud Rebuild**, **Windows 365 Reserve** | [@ms-wri-ignite-2025] | PITR Insider preview Nov 2025; W365R GA; Cloud Rebuild coming |
flowchart LR
subgraph L1[1. Prevent: stop bad updates at the partner -- MVI 3.0 SDP]
subgraph L2[2. Prevent: stop bad code being kernel-resident -- user-mode AV platform]
subgraph L3[3. Manage: see the incident at scale -- Intune surfaces WinRE state]
subgraph L4[4. Recover the unbootable: Quick Machine Recovery]
subgraph L5[5. Rebuild without shipping hardware: PITR / Cloud Rebuild / W365 Reserve]
CORE[Windows endpoint -- recoverable at fleet scale]
end
end
end
end
end
### Layer 1: Safe Deployment Practices and MVI 3.0
Microsoft Virus Initiative 3.0 became effective on April 1, 2025 [@ms-wri-ignite-2025]. Membership now requires partners to commit to four named obligations [@ms-mvi]: a signed nondisclosure agreement; use of Microsoft Trusted Signing (the hosted descendant of [Authenticode](/blog/authenticode-and-catalog-files-the-crypto-foundation-under-w/)) for AV/EDR driver code-signing; documented Safe Deployment Practices for content updates (gradual rollouts with deployment rings and monitoring); and certification within the last 12 months by at least one of AV-Comparatives, AVLab Cybersecurity Foundation, AV-Test, MRG Effitas, SE Labs, SKD Labs, VB 100, or West Coast Labs [@ms-mvi]. The June 26, 2025 WRI update lists eight named partner endorsements -- Bitdefender (Florin Virlan), CrowdStrike (Alex Ionescu), ESET (Juraj Malcho), SentinelOne (Stefan Krantz), Sophos (John Peterson), Trellix (Jim Treinen), Trend Micro (Rachel Jin), and WithSecure (Johannes Rave) -- and the November 18, 2025 update confirms the effective date verbatim: "Effective April 1, 2025, Version 3.0 of the Microsoft Virus Initiative added new requirements for all Windows antivirus (AV) partners to maintain signing rights for Windows AV drivers" [@ms-wri-jun-2025, @ms-wri-ignite-2025].
Microsoft's program for third-party antivirus and endpoint detection vendors that ship products on Windows. MVI 3.0, effective April 1, 2025, adds Safe Deployment Practices, mandatory Trusted Signing, NDA, and 12-month independent test-lab certification as preconditions to maintain Windows AV driver signing rights [@ms-mvi, @ms-wri-ignite-2025].
The model is structurally identical to the canary / progressive-rollout pattern formalised in the Google SRE Book chapter on Release Engineering: hermetic builds, multiple deployment rings, gated promotion between rings, "Push on Green", and the option to cherry-pick at the same revision when a critical change is needed mid-cycle [@sre-release-eng]. MVI 3.0 is not a Microsoft invention; it is a Microsoft *mandate* of a model that has been industry practice for two decades. The mandate is what is new.
### Layer 2: The Windows endpoint security platform
The same November 19, 2024 keynote committed to a *Windows endpoint security platform* that lets partners ship their detection logic outside kernel mode, with a private preview promised to security-partner programs by July 2025 [@ms-wri-ignite-2024]. The June 26, 2025 update confirmed the date with named partner endorsements [@ms-wri-jun-2025]. The architectural premise is the one BSOD survivors recognise immediately: a faulty user-mode component can be killed by Task Manager; a faulty kernel-mode driver bug-checks the system.
"Graphics drivers, for example, will continue to run in kernel mode for performance reasons." -- Microsoft, *Preparing for what's next*, November 18, 2025 [@ms-wri-ignite-2025].
Microsoft is careful to frame WRI as a floor-raiser, not a kernel ban. The November 18, 2025 update enumerates the driver-resiliency playbook for the surfaces that *will* remain in kernel mode: mandatory compiler safeguards (control-flow integrity, [CFG](/blog/process-mitigation-policies-cfg-acg-cig-and-the-layer-betwee/), stack canaries), driver isolation, DMA-remapping, a higher signing bar, and expanded in-box Microsoft drivers and APIs that third parties can call rather than reimplementing [@ms-wri-ignite-2025]. The argument is that the kernel surface that *must* exist (graphics, storage, some networking) should be smaller, better isolated, and equipped with mitigations that contain a single fault.
The June 2025 partner roster is the most pointed piece of evidence that the user-mode direction predates and outlasts the July 2024 incident. CrowdStrike itself is named [@ms-wri-jun-2025]. The vendor that started the chain reaction is publicly endorsing the architectural concession the chain reaction priced into existence.
### Layer 3: Intune-surfaced WinRE state
The November 18, 2025 update names a new Intune signal: "Intune will surface when a Windows device has booted into the Windows Recovery Environment (WinRE)" [@ms-wri-ignite-2025]. The same signal will appear in the Azure Portal for Windows Server VMs that switched into WinRE. The same update introduces a WinRE plug-in model: IT administrators can push custom recovery scripts through Intune, with the model documented as third-party-MDM-adoptable. Both are "coming soon" as of that announcement [@ms-wri-ignite-2025].
The architectural insight here is that *Microsoft-pushed remediations* (QMR) and *administrator-pushed remediations* (Intune scripts) must be expressible against the same WinRE surface, with Intune providing the visibility and audit layer.
### Layer 4: Quick Machine Recovery
Already covered in Section 6. Status: GA August 2025 on Windows 11 24H2 build 26100.4700+ [@ms-qmr, @ms-wri-ignite-2025]. Autopatch QMR management is in preview at the November 2025 announcement [@ms-wri-ignite-2025].
### Layer 5: Rebuild without shipping hardware
The November 18, 2025 update introduces three Microsoft-cloud-side recovery actions [@ms-wri-ignite-2025]:
- **Point-in-Time Restore (PITR).** Cloud-orchestrated rollback to an earlier point-in-time snapshot of the device's full state. Status: available in the Windows Insider preview build the week of the announcement.
- **Cloud Rebuild.** Intune-portal-triggered clean OS reimage using Autopilot for zero-touch provisioning, with user data and settings restored from OneDrive and Windows Backup for Organizations. Status: coming.
- **Windows 365 Reserve.** A temporary Cloud PC for users whose endpoint is unusable. Status: generally available.
Each of these targets a scenario QMR cannot fix. PITR addresses regressions that the user-mode WU pipeline cannot patch back -- driver downgrades that need to roll back state, not push a new patch. Cloud Rebuild addresses devices whose local Windows is genuinely beyond surgical repair. Windows 365 Reserve addresses the productivity gap while the local device is being recovered.
All five layers are anchored on Microsoft blogs and Microsoft Learn pages. None of them is unique to Microsoft. Apple, ChromeOS, and the Linux atomic distributions have each chosen a different layered architecture for the same problem. What does the field actually look like?
## 8. Competing Models: Apple, ChromeOS, and the Linux Atomic Distributions
Microsoft is not the first vendor to treat recovery as part of its security architecture. It is, at consumer scale, among the last. Apple, Google, and the Linux atomic-distribution community each picked a different layer to anchor on.
### Apple macOS: Signed System Volume + paired/fallback recoveryOS + 1TR
macOS 10.15 (Catalina, 2019) introduced the read-only system volume. macOS 11 (Big Sur, 2020) added the *Signed System Volume* on top of it: a SHA-256 Merkle tree over every block of the system volume, sealed by Apple at install or update time [@apple-ssv]. On Apple Silicon, the bootloader verifies the seal before transferring control to the kernel; on Intel-based Macs with the T2 Security Chip, the bootloader forwards the measurement and signature to the kernel, which verifies the seal directly before mounting the root file system [@apple-ssv]. On verification failure, the Mac drops into recoveryOS automatically and prompts the user to reinstall.
The recovery side has three flavours [@apple-boot]: a *paired recoveryOS* that exactly matches the installed system version; on Apple Silicon, a *fallback recoveryOS* (the previous OS version); and a hardware-anchored *1TR* ("one true recovery") environment that survives even when the paired recoveryOS is broken. The 1TR environment is anchored in the Secure Enclave, which is the macOS analogue of Windows's signed `bootmgfw.efi` on the EFI System Partition.
What Apple excels at is *tampered* system files and *failed* updates: the first block read fails Merkle verification; the snapshot pointer flips to the prior good snapshot; the user reboots into a working system. What Apple does *not* have is an analogue of QMR's targeted remediation pipeline. The macOS answer to a faulty signed third-party security agent is "reinstall macOS". That is wipe-and-reload, not surgical repair.
### ChromeOS: Verified Boot + A/B root partitions + auto-rollback
ChromeOS's verified-boot design has been the same since 2010 [@chromium-verified-boot]. A read-only boot stub, anchored in write-protected EEPROM, computes a cryptographic hash of the read-write firmware (SHA-1 in the original 2010 specification; SHA-256 in current production firmware) and verifies an RSA signature (at least 2048 bits) against a permanently stored public key [@chromium-verified-boot]. The verified read-write firmware then hashes the kernel and verifies its signed hashes. A transparent block device in the kernel verifies each block against a stored hash tree on every read, with the tree's root signed by the firmware.
The recovery story is the brilliant part. ChromeOS devices have two root partitions, *ROOT-A* and *ROOT-B*, plus a separate stateful partition for user data [@chromium-autoupdate]. Each root partition carries a `remaining_attempts` counter (default 6) stored in unused GPT bits next to the bootable flag. On N consecutive failed boots, the boot loader falls back to the *other* partition. Auto-updates always write to the partition not currently in use, never the booted one. The result is that ChromeOS recovers from a faulty signed system update in *one reboot* per device, automatically, without an operator action. This is the empirical upper bound on automation: no fielded platform recovers a signed-but-faulty boot path faster than one reboot.
### Linux atomic distributions: OSTree, rpm-ostree, bootc
OSTree, the upstream of Fedora's atomic desktops and CoreOS, is "Git for operating system binaries" [@fedora-silverblue]. It stores content-addressed objects under `/ostree/repo`, builds atomic *deployments* as hardlink farms under `/boot/loader/entries/ostree-$stateroot-$checksum.$serial.conf`, performs a three-way merge of `/etc` between the booted deployment and the new one, and atomically swaps the boot directory by flipping a symlink between `/ostree/boot.0` and `/ostree/boot.1` [@ostree-atomic]. The crash-safe guarantee is verbatim: "if the system crashes or you pull the power, you will have either the old system, or the new one" [@ostree-atomic].
Fedora Silverblue, Fedora CoreOS, Endless OS, and (since 2024) Fedora's bootc container-based desktops all ship OSTree by default [@fedora-silverblue]. Where OSTree excels is server fleets and developer workstations; where it struggles is layered third-party packages crossing deployments (the rebase/deploy friction) and the absence of a network-reachable in-recovery remediation analogue to QMR.
### Traditional Linux: dracut + GRUB rescue + initramfs
The "manual safe-mode + delete-the-file" model. A skilled operator with shell access plus iLO / iDRAC / IPMI serial-over-LAN can repair a Linux box; everyone else is in trouble. The CrowdStrike-style incident response on traditional Linux would look exactly the same as it did on Windows: per-device, skilled operator, no automation. The Linux distributions that *did* avoid this fate are the OSTree-based atomic ones; the conventional ones are at the same operator-bound floor Windows just climbed off.
flowchart TB
subgraph WIN[Windows: WinRE + QMR]
WIN_WIM[winre.wim on recovery partition or in OS-volume folder] --> WIN_WU[Windows Update endpoint]
end
subgraph APL[Apple: macOS]
APL_PR[Paired recoveryOS] --> APL_SNAP[APFS snapshot revert]
APL_FB[Fallback recoveryOS / 1TR in Secure Enclave] --> APL_SNAP
end
subgraph CHR[ChromeOS]
CHR_BOOTA[ROOT-A] --> CHR_FALLBACK[Boot loader falls back to other root]
CHR_BOOTB[ROOT-B] --> CHR_FALLBACK
end
subgraph OS[Linux atomic / OSTree]
OS_DEPNEW[New deployment] --> OS_PRIOR[Prior deployment retained for rollback]
end
### A head-to-head comparison
The dimensions that matter are: year shipped, in-recovery network capability, auto-remediation, signed-but-faulty-driver protection, per-device operator cost during a fleet event, trust floor, and encrypted-volume recovery story.
| Dimension | Windows WinRE + QMR | Apple SSV + recoveryOS | ChromeOS A/B + verified boot | Linux atomic (OSTree) | Conventional Linux |
|---|---|---|---|---|---|
| Year shipped | WinRE 2007 [@wiki-winre]; QMR 2025 [@ms-qmr] | SSV 2020; recoveryOS / 1TR 2020 [@apple-ssv, @apple-boot] | Verified Boot 2010 [@chromium-verified-boot] | OSTree 2012 (dev started 2011); rpm-ostree later [@ostree-atomic, @fedora-silverblue] | dracut 2009; GRUB 2 2009 |
| In-recovery network capability | Yes (WPA/WPA2 Wi-Fi or wired) [@ms-qmr] | Yes for reinstall; no targeted remediation | Yes for recovery image fetch | No standard pipeline | No |
| Auto-remediation without operator | Yes (one-time or looped) [@ms-qmr] | No (user confirms reinstall) | Yes (boot loader fallback) [@chromium-autoupdate] | No (user selects rollback in GRUB) | No |
| Protection against signed-but-faulty drivers | Behavioural via MVI 3.0 SDP + user-mode AV [@ms-mvi, @ms-wri-jun-2025] | DriverKit / System Extensions push third parties out of kernel | A/B rollback auto-recovers in one boot cycle | Layered package rolls back with deployment | None |
| Per-device operator cost in a fleet event | O(1) -- publish remediation once | O(N) -- each user reinstalls | O(0) -- automatic per device | O(N) -- each user selects rollback | O(N) -- skilled operator per device |
| Trust floor (unrecoverable without external media) | Corrupted `bootmgfw.efi`, missing WinRE, lost BitLocker key | Failed 1TR (very rare) | Both root partitions plus EEPROM corrupted | GRUB unreachable | GRUB unreachable |
| Encrypted-volume recovery story | BitLocker recovery key required [@ms-qmr] | FileVault key required if at-rest read needed | Stateful partition holds user data only | LUKS passphrase required | LUKS passphrase required |
The notable row is the *per-device operator cost during a fleet event*. QMR moves Windows from O(N) (pre-WRI) to O(1) (post-WRI). ChromeOS was already at O(0) thanks to the A/B rollback. Apple, conventional Linux, and OSTree-based Linux remain at O(N).
> **Key idea:** The per-device operator cost row is the one Microsoft engineered WRI to change. QMR moves Windows from O(N) to O(1). ChromeOS was already at O(0) by virtue of A/B rollback. Apple, conventional Linux, and OSTree-based Linux remain at O(N). This is the empirical justification for the thesis that resilience is a security property: pre-WRI Windows, despite shipping BitLocker, [HVCI](/blog/wdac--hvci-code-integrity-at-every-layer-in-windows/), and Secure Boot, had a *recoverability complexity class* worse than ChromeOS. A faulty signed driver could exploit that gap to deny service at fleet scale.
Three vendors got to fleet-scale recovery earlier. Microsoft's catch-up move is constrained by what Microsoft does not control: OEM partition layouts, BIOS/UEFI variance, BitLocker key escrow.Apple ships hardware-plus-OS and Google ships ChromeOS against an OEM-certified hardware spec, both of which let those vendors specify partition layout end to end. Microsoft ships the OS and asks OEMs to follow the Image Configuration Designer defaults; some do, some do not. The KB5028997 workaround for "recovery partition too small for new winre.wim" is precisely the artefact of Microsoft *not* being able to mandate the layout [@ms-winre-tech-ref, @ms-kb5028997]. Those constraints set hard limits on what WRI can fix, and they are the reason the trust-floor row in the table is longer for Windows than for ChromeOS.
## 9. Theoretical Limits and the BitUnlocker Counter-Current
Two well-known results from the systems and security literature say that no fielded recovery primitive can be perfect, and Microsoft's own offensive-research team demonstrated, at Black Hat USA 2025 in August 2025, exactly which limit WRI runs into [@alon-leviev].
### The trust-floor lower bound
No system can recover from corruption of *all* of its boot-path code without external media, because the verification step that detects corruption is itself part of the boot-path code. ChromeOS encodes this with a write-protected EEPROM that an attacker cannot rewrite without a hardware write-protect override [@chromium-verified-boot]; Apple encodes it with the 1TR environment anchored in the Secure Enclave [@apple-boot]; Windows encodes it by requiring the EFI System Partition plus a signed `bootmgfw.efi`. Below that floor, QMR, OSTree, and APFS snapshots are all helpless. The recovery surface bounded by what fits in write-protected non-volatile storage is the lower bound on automated recovery.
### The end-to-end argument applied to recovery
Saltzer, Reed, and Clark's 1984 *End-to-End Arguments in System Design* [@saltzer-reed-clark-1984] argued that correctness checks belong at the endpoints of a communication system, not in intermediate nodes. Applied to update pipelines, the argument predicts that *bug-free updates cannot be guaranteed by intermediate nodes* (the vendor's QA fleet, the CDN, the Windows Update service). Correctness can only be observed at the endpoint. The corollary is that the probability of a faulty update reaching production cannot be driven to zero by any amount of pre-release testing; the platform's design must instead bound *blast radius* and *time-to-recovery* of the faulty updates that will inevitably ship. MVI 3.0's SDP bounds the first (deployment rings); QMR bounds the second (network-reachable remediation). The argument is identical to the canary / progressive-rollout pattern in Google's SRE Book Release Engineering chapter [@sre-release-eng].
### The attack-surface trade-off
An auto-unlocking, network-reachable recovery environment expands the Trusted Computing Base. Every additional capability added to the recovery path is a new code path; a new code path is a new attack vector. The BitUnlocker research, by Netanel Ben Simon and Alon Leviev at Microsoft's Security Testing and Offensive Research (STORM) team [@alon-leviev, @ms-bitunlocker-blog], is the most pointed evidence we have that the trade-off is real.
The set of hardware, firmware, and software components on which a system's security policy ultimately depends. A bug in a TCB component can undermine the entire security policy; everything outside the TCB is, by definition, untrusted relative to it. Recovery environments expand the TCB because they need privileged access to encrypted user state.
The four BitUnlocker CVEs are all rated CVSS 6.8 [@nvd-cve-48800, @nvd-cve-48003, @nvd-cve-48804, @nvd-cve-48818]:
- **CVE-2025-48804** [@nvd-cve-48804, @ms-bitunlocker-blog] -- BitLocker Security Feature Bypass via `boot.sdi` parsing.
- **CVE-2025-48003** [@nvd-cve-48003, @ms-bitunlocker-blog] -- BitLocker Security Feature Bypass via `SetupPlatform.exe` / Shift+F10 abuse during the WinRE Apps Scheduled Operation.
- **CVE-2025-48800** [@nvd-cve-48800, @ms-msrc-cve-48800, @ms-bitunlocker-blog] -- BitLocker Security Feature Bypass via `tttracer.exe` abuse during Offline Scanning.
- **CVE-2025-48818** [@nvd-cve-48818, @ms-msrc-cve-48818, @ms-bitunlocker-blog] -- BitLocker Security Feature Bypass via BCD parsing in the Online PBR exploit chain; the fourth pillar of the chain.
The published Microsoft Security blog post on BitUnlocker enumerates the architectural attack surfaces verbatim under three section headings: *Attacking Boot.sdi Parsing*, *Attacking ReAgent.xml Parsing*, and *Attacking Boot Configuration Data (BCD) Parsing* [@ms-bitunlocker-blog]. The premise is the same in every case. WinRE must read the OS volume's BitLocker recovery material to perform repairs. Therefore WinRE has code paths that, given the right inputs, can obtain the decrypted Full Volume Encryption Key. The four CVEs each find a parser or debugger inside WinRE whose input handling can be steered by an attacker with brief physical access to flip the recovery flow into a state where the decrypted FVEK becomes reachable.
flowchart TD
PA[Physical access foothold] --> SDI[Attacking boot.sdi parsing -- CVE-2025-48804]
PA --> RA[Attacking ReAgent.xml / SetupPlatform.exe -- CVE-2025-48003]
PA --> BCD[Attacking BCD parsing / Online PBR -- CVE-2025-48818]
PA --> TT[Abusing tttracer.exe Offline Scanning -- CVE-2025-48800]
SDI --> FVEK[Reach decrypted FVEK on OS volume]
RA --> FVEK
BCD --> FVEK
TT --> FVEK
FVEK --> EX[BitLocker bypass; data exfiltration]
### The encrypted-volume impossibility
Unattended recovery of an encrypted volume *without the key* is impossible. It is a security correctness requirement, not a limitation that engineering can fix. QMR explicitly does not bypass BitLocker [@ms-qmr]. Apple's FileVault, ChromeOS's TPM-bound user partition, and Linux LUKS all share this property; none of them gets to be exempt from the requirement that the key be present somewhere before the encrypted volume can be modified offline.
> **Note:** Every additional capability added to the recovery path is an additional attack vector against the encrypted user state that the recovery path is privileged to access. QMR's network reachability is a feature for the operator and a feature for the attacker. The article's thesis is not *WRI makes Windows safer in absolute terms*; it is *WRI moves the trade-off to a different curve*. The same vendor making the recovery surface reachable from Windows Update is the vendor that has to harden it against itself.
### The upper bound
ChromeOS A/B auto-rollback recovers a single device in one reboot cycle without operator action [@chromium-autoupdate]. This is the empirical upper bound on automation. No fielded platform recovers a signed-but-faulty boot path faster than one reboot per device. QMR matches the ChromeOS upper bound in the steady state once a remediation is published; the only thing QMR cannot do that ChromeOS does is recover from the *first* signed-but-faulty update before Microsoft has authored the remediation. The lower bound on time-to-fleet-recovery is set by the production lead time of Microsoft's own QA pipeline plus the time to author and publish the targeted patch.
Microsoft's own offensive-research team published the BitUnlocker chain one Patch Tuesday before QMR went generally available. That is not a coincidence; it is the price of moving WinRE up the trust ladder. The next question -- what has not been priced yet? -- belongs in the open-problems list.
## 10. Open Problems: Where Microsoft Has Not Committed
WRI is a current commitment with a published roadmap. The roadmap has explicit holes. Each of the six below is documented from a primary Microsoft source -- either by what the source *says* or, in the most honest cases, by what it *does not say*.
**Network protocol surface in WinRE.** The Microsoft Learn QMR page is explicit: only wired Ethernet and WPA/WPA2 password-based Wi-Fi are supported as of November 2025 [@ms-qmr]. Enterprise 802.1X and WPA3-Enterprise with device certificates are committed in the November 18, 2025 update as *coming soon* under the *Wi-Fi 7 for Enterprise* and WinRE-reads-from-Windows lines, but no shipping date is published [@ms-wri-ignite-2025]. For an enterprise on 802.1X, this is the most visible gap: a managed-fleet device on a corporate SSID cannot reach Windows Update from inside WinRE today.
**Safe-mode hardening as a discrete deliverable.** The phrase "safe mode hardening" has no first-party Microsoft anchor as a discrete WRI deliverable. The closest documented item is [*Administrator Protection*](/blog/adminless-how-windows-finally-made-elevation-a-security-boun/), announced in the November 19, 2024 Ignite blog as a constraint on elevated-context behaviour [@ms-wri-ignite-2024]. That is not the same thing. The Safe Mode boot path that the CrowdStrike incident used to delete `C-00000291*.sys` was the *same* Safe Mode boot path that has existed since Windows NT; nothing in the WRI primary sources commits to changing what Safe Mode does or does not load. Honest reading: WRI re-prices the recovery surface around Safe Mode; it does not (yet) change Safe Mode itself.
**Cross-vendor partition layout.** The Microsoft Learn WinRE Technical Reference [@ms-winre-tech-ref] documents the recommended ICD-media layout but does not enforce it. Clean Windows Setup, OEM-installed Windows, and ICD-media-installed Windows produce different recovery-partition layouts, and the existence of KB5028997 (the well-known workaround for "recovery partition too small for the new `winre.wim`") is a direct consequence. ChromeOS and macOS do not have this problem because Google and Apple control the layout end to end. Microsoft chose, decades ago, not to.
**Third-party MDM support for the WinRE plug-in model.** The November 18, 2025 update describes the WinRE plug-in model as third-party-MDM-adoptable, but no third-party MDM vendor had shipped a plug-in or a QMR management surface as of that announcement [@ms-wri-ignite-2025]. Customers on JAMF, Workspace ONE, Tanium, or similar do not yet have a documented integration path. If the future of recovery is Intune-coupled, WRI's reach is bounded by Intune adoption.
**BitLocker key escrow as a WRI deliverable.** No WRI primary source ([@ms-wri-ignite-2024, @ms-wri-jun-2025, @ms-wri-ignite-2025]) names "BitLocker recovery key flows" as a discrete WRI deliverable. The adjacent items are: *hardware-accelerated BitLocker* on new devices starting spring 2026 [@ms-wri-ignite-2025]; the BitUnlocker CVE patches in July 2025 [@ms-bitunlocker-blog]; and the Entra ID self-service BitLocker recovery flow at `aka.ms/aadrecoverykey` [@aka-ms-aadrecoverykey, @ms-kb5042421]. The current state is that BitLocker key escrow is an Entra ID and Intune feature, not a WRI feature. QMR's value is bounded by BitLocker key availability for the encrypted-volume fraction of any fleet; a WRI deliverable that improved key escrow would compound QMR's benefit. None has been announced.
**Recovery in air-gapped and sovereign environments.** QMR routes through Windows Update. Air-gapped fleets, sovereign-cloud customers, and offline manufacturing networks cannot reach Windows Update from WinRE. The November 18, 2025 update mentions Connected Cache, but no QMR-Connected-Cache integration is committed [@ms-wri-ignite-2025]. For the high-assurance customer who today does not let manufacturing endpoints talk to the public Internet at all, QMR is a feature for someone else.
> **Note:** The six items above are gaps in the *roadmap*, anchored either by what Microsoft has explicitly named as coming-soon or by the absence of a primary source. They are not features. The article distinguishes Microsoft-committed deliverables (cited to a primary source) from adjacent inferences. Readers reviewing WRI for their own fleets should do the same.
These six gaps are where the next year of WRI roadmap will be argued. None of them is closed; some are closed-soon. For the practitioner, the immediate question is what to do, today, with what is shipping right now.
## 11. Practitioner's Guide
Everything above is architecture. This section is the checklist.
**1. Verify WinRE is provisioned.** Run `reagentc /info` from an elevated prompt. The output should say `Windows RE status: Enabled` and point at a sensible WinRE location -- typically `\?\GLOBALROOT\device\harddisk0\partitionN\Recovery\WindowsRE` or `C:\Windows\System32\Recovery\WindowsRE`. If the status is `Disabled`, run `reagentc /enable`. If the recovery partition is too small for a new `winre.wim` (a known issue surfacing with cumulative updates that grow the image, surfaced as a System event ID 4502 with `ErrorPhase 2`), follow KB5028997 [@ms-kb5028997, @ms-winre-tech-ref].
The mitigation, in outline: disable WinRE temporarily (`reagentc /disable`); shrink the OS partition via `diskpart` by enough megabytes (250 MB minimum per Microsoft's published procedure) to host a larger recovery partition; recreate the recovery partition with the GPT Type ID `DE94BBA4-06D1-4D40-A16A-BFD50179D6AC` and the GPT attributes value `0x8000000000000001` that hides it from automounting; re-enable WinRE (`reagentc /enable`) so the new `winre.wim` is copied into the resized partition. The Microsoft Support KB article carries the exact `diskpart` commands [@ms-kb5028997], with the Windows RE Technical Reference as the architectural anchor [@ms-winre-tech-ref]. Test on a representative device first; the resize is not reversible without re-imaging.
**2. Audit your QMR posture before turning it on.** On Enterprise, Education, and managed Pro, cloud remediation is *off* by default [@ms-qmr]. Decide first; ring second; roll out third. The Intune Settings Catalog path is *Remote Remediation > Enable Cloud Remediation*. Pre-stage a WPA/WPA2 Wi-Fi credential via `reagentc.exe /SetRecoverySettings` if your recovery network is wireless.
**3. Use the test-mode dry run.** `reagentc.exe /SetRecoveryTestmode` followed by `reagentc.exe /BootToRe` triggers a *simulated* QMR cycle. The simulated remediation appears in Settings > Windows Update > Update history rather than mutating the production OS. Run it on a pilot ring before depending on QMR in a real incident [@ms-qmr].
**4. Plan for BitLocker key availability.** Ensure recovery keys are escrowed to Entra ID, not just printed on a card in a drawer. Enable the Entra ID self-service flow at `aka.ms/aadrecoverykey` so an unattended user can retrieve their own key during an incident [@ms-kb5042421, @aka-ms-aadrecoverykey].
**5. Know the difference between Cloud Reset, QMR, and Autopilot Reset.** Cloud Reset (in-Windows *Reset this PC > Cloud download*) reinstalls a running OS [@ms-pbr-overview]. QMR runs in WinRE *before* the OS boots, applying targeted patches from Windows Update [@ms-qmr]. Autopilot Reset re-provisions a *bootable* device via Intune. Three different tools, three different scenarios; do not confuse them in your runbook.
**6. Watch for the November 2025 Intune signals.** Once Intune surfaces WinRE state in the admin centre, build the muscle of looking for it. The roll-up that tells you "12 devices are in WinRE right now" is the operational primitive Microsoft did not have through July 2024 [@ms-wri-ignite-2025].
> **Note:** Promote step 3 (the test-mode dry run) into your incident-response runbook now [@ms-qmr]. The time to discover that the recovery Wi-Fi SSID changed last quarter is not in the middle of a fleet-down event.
> **Note:** QMR cannot decrypt the OS volume. It applies Windows Update patches that take effect on the next boot, but it cannot run against an encrypted volume's contents without the BitLocker recovery key being available [@ms-qmr]. If a device's BitLocker key is not escrowed to Entra ID and the user is not available to read it from a printout, QMR cannot help. Key escrow is upstream of recovery; treat it that way.
The `reagentc /info` output is short and uniform enough that a small script can classify the device's WinRE health. The block below sketches one in JavaScript pseudocode.
{`
// reagentc /info is a small, deterministic text block. Parse it.
const sampleOutput = \`
Windows Recovery Environment (Windows RE) and system reset configuration
Information:
Windows RE status: Enabled
Windows RE location: \\\\?\\\\GLOBALROOT\\\\device\\\\harddisk0\\\\partition4\\\\Recovery\\\\WindowsRE
Boot Configuration Data (BCD) identifier: a1b2c3d4-...-winre-guid
Recovery image location:
Recovery image index: 0
Custom image location:
Custom image index: 0
REAGENTC.EXE: Operation Successful.
\`;
function classify(output) {
const status = /Windows RE status:\\s+(\\w+)/.exec(output)?.[1];
const location = /Windows RE location:\\s+(\\S+)/.exec(output)?.[1] || '';
const partitionMatch = /partition(\\d+)\\\\Recovery\\\\WindowsRE/.exec(location);
const onPartition = !!partitionMatch;
const onOsVolume = /^[A-Z]:\\\\Recovery\\\\WindowsRE/.test(location);
if (status !== 'Enabled') {
return { status, action: 'reagentc /enable -- WinRE is not active' };
}
if (!onPartition && !onOsVolume) {
return { status, action: 'Unknown layout; verify with diskpart and reagentc' };
}
if (onPartition) {
return {
status,
layout: 'recovery-partition',
partition: partitionMatch[1],
note: 'If cumulative updates fail with insufficient-space errors, see KB5028997',
};
}
return { status, layout: 'os-volume-recovery-folder', note: 'OEM-style layout; some Intune' +
' policies assume a separate partition. Confirm before relying on remote remediation.' };
}
console.log(classify(sampleOutput));
`}
The practical questions answered, the article closes with a set of FAQs that catch the common misconceptions.
## 12. Frequently Asked Questions and Closing Thoughts
No. WRI's *Windows endpoint security platform* gives MVI partners a user-mode runtime so their detection logic does not have to live in a kernel-mode `.sys` file [@ms-wri-jun-2025, @ms-wri-ignite-2025]. Kernel-mode drivers as a class are not retired: the November 18, 2025 update is explicit that "graphics drivers, for example, will continue to run in kernel mode for performance reasons" [@ms-wri-ignite-2025], and the driver-resiliency playbook (compiler safeguards, driver isolation, DMA-remapping, higher signing bar) is precisely for the kernel-mode surface that will remain.
No. The Microsoft Learn QMR page is explicit that the recovery flow does not decrypt the OS volume [@ms-qmr]. If the BitLocker recovery key is unavailable, QMR cannot help. The recommended escrow path is Entra ID, with the user-facing self-service flow at `aka.ms/aadrecoverykey` [@aka-ms-aadrecoverykey, @ms-kb5042421].
No. The BCD Boot Options Reference enumerates every legal element on a boot entry, and there is no `/recovery` flag on `winload.efi` or `winload.exe` [@ms-bcd]. WinRE is selected by following the `recoverysequence` element of the OS-loader entry to a separate BCD entry whose `winpe` is `Yes` and whose `osdevice` mounts `winre.wim` from a `boot.sdi`-backed RAM disk. The entire handoff is inside the boot manager, before `winload.efi` runs.
No. The four CVE-2025-48800/-48003/-48804/-48818 advisories were patched in the July 8, 2025 cumulative update before QMR went generally available in August 2025 [@ms-bitunlocker-blog, @nvd-cve-48800, @ms-wri-ignite-2025]. The patches addressed parser and debugger code paths inside WinRE; they did not remove WinRE's ability to read the OS volume's BitLocker recovery material, which is a feature WinRE needs in order to perform any repair on an encrypted volume.
No. The Secure Future Initiative (SFI), announced in November 2023, is Microsoft's company-wide security program. WRI is the Windows-specific workstream inside SFI that owns Windows availability, kernel resilience, and the recovery surface; the published WRI blogs frame it as the Windows pillar of SFI rather than a stand-alone effort [@ms-wri-ignite-2024, @ms-wri-jun-2025].
QMR will not connect. The Microsoft Learn page is explicit that only wired Ethernet and WPA/WPA2 password-based Wi-Fi are supported [@ms-qmr]. The November 18, 2025 update commits to WPA3-Enterprise with device certificates as part of the WinRE-reads-from-Windows networking work and the *Wi-Fi 7 for Enterprise* line, but it does not give a shipping date [@ms-wri-ignite-2025]. For now, enterprises whose recovery story depends on QMR over Wi-Fi must either stand up a dedicated WPA2-PSK recovery SSID or rely on wired recovery.
The code is mostly the same. What changed is the *policy* that lets WinRE call Windows Update without an operator at the keyboard. WinPE has shipped networking drivers since 2002 [@ms-winpe-intro], and `winre.wim` has been bootable from a recovery partition since 2009. The breakthrough is the commitment that the recovery environment is allowed to phone home -- and the surrounding program (MVI 3.0, the user-mode AV platform, Intune visibility) that makes it usable as a fleet-scale primitive.
### Closing
The Windows Recovery Environment that worked perfectly on July 19, 2024 is the same Windows Recovery Environment that became Microsoft's most important security surface on August 1, 2025. The architecture did not change in the year between. The question we ask of it did.
The CrowdStrike incident did not invent the case for resilience as a security property. It priced it. Two months after the bug check signature `csagent+0xe14ed` made the rounds, Microsoft and the MVI cohort sat down at WESES to argue out what would become MVI 3.0 [@ms-weses]. Three months after that, the Ignite 2024 keynote committed to Quick Machine Recovery and to a user-mode antimalware platform [@ms-wri-ignite-2024]. Five months after *that*, the first QMR code shipped on the Beta Channel [@ms-qmr-insider-mar2025]. Twelve months after the incident, MVI 3.0 was binding [@ms-wri-ignite-2025]. Thirteen months after, QMR went generally available -- and BitUnlocker had been patched a month earlier in the July 2025 cumulative update. Sixteen months after, Microsoft published the rebuild-without-shipping-hardware roadmap [@ms-wri-ignite-2025].
WRI does not eliminate the trade-off between recoverability and attack surface. It moves the trade-off to a curve where the per-device cost of a fleet-down event is not bounded by human attention, and where the recovery code path is hardened by the same vendor's offensive-research team. Those are different curves than the ones the platform was on in July 2024. They are not the curves a textbook chapter on Windows internals would have predicted in 2014. They are also still the curves of a single vendor's program, anchored on a small number of blog posts and Microsoft Learn pages, and the work of validating them belongs in every fleet that depends on Windows for availability.
If WinRE worked perfectly on July 19, 2024 and that was the problem, the test of WRI is whether the next *July 19, 2026* never makes the news.