The Fingerprint Two Files Shared: A Field Guide to Cryptographic Hashes

1. The Fingerprint Two Files Shared#

In February 2017, researchers at Google and CWI Amsterdam published two PDF files that looked nothing alike, with different visible content and different bytes, yet when a computer fingerprinted each of them with SHA-1 it returned the very same 40-hex-digit answer: 38762cf7f55934b34d179ae6a4c80cadccbb7f0a ^[3]. Two different files, one identical fingerprint. Five years earlier, malware named Flame had used the same species of trick against the older MD5 hash to forge a code-signing certificate that chained up to Microsoft and rode Windows Update onto target machines as if Microsoft itself had signed it ^[2].

Neither of those is a stunt. They are the failure mode of one of the quietest, most load-bearing primitives in computing. A cryptographic hash function takes an input of any size and returns a short, fixed-length string of bits, its digest or fingerprint, and it promises something enormous: that fingerprint stands in for the whole input unforgeably and unambiguously.

Almost everything you trust online leans on that promise. A certificate authority does not sign your certificate; it signs a hash of it. Code signing signs a hash of the program. A TLS 1.3 handshake authenticates a hash of the transcript. Every Git commit names its parent by hash, and every blockchain block commits to its transactions through a tree of hashes. When the promise holds, all of that machinery works. When it breaks, it breaks silently and all at once.

A cryptographic hash is infrastructure hiding in plain sight. Because signatures, certificates, and integrity checks all sign the digest rather than the data, a break in the hash silently breaks every signature built on top of it. The attacker never has to touch your private key.

That is what makes the two colliding PDFs unsettling rather than merely clever. If an adversary can produce two inputs with the same fingerprint, then any signature over one of them is also a valid signature over the other, and the entire chain of trust resting on that hash quietly inverts. This guide answers four questions in order: what exactly is the promise a hash makes, how is the fingerprint built, how has it broken in the real world, and how do you choose one today that will not break under you.

flowchart TD
A["One promise: a short, unforgeable, unambiguous fingerprint"] --> B["How it is built: the Merkle-Damgard chain and the sponge"]
B --> C["How it broke: MD5 and SHA-1 collisions weaponized into forgery"]
C --> D["What survived: SHA-2 widened the same margins"]
D --> E["What diversified: the SHA-3 sponge, the BLAKE3 tree, KangarooTwelve"]
E --> F["How to choose: match the function and mode to the clause you rely on"]

A preview of the plot, because it corrects the single most common misconception in the whole subject. Some hashes died and were superseded: MD5 gave way to SHA-1, which gave way to SHA-2. But SHA-2 did not then give way to SHA-3. Instead the field forked. One arc is supersession by failure; the other is deliberate diversification, where several unbroken constructions coexist on purpose so the internet's trust machinery never again rests on a single design. Telling those two arcs apart is most of what it means to understand this topic.

Before we can understand how the fingerprint was forged, we have to say, precisely, what it was ever supposed to guarantee.

2. What a Hash Actually Promises#

Ask three engineers what a "secure hash" means and you will get three answers: it is one-way, it has no collisions, it looks random. Cryptographers are more disciplined. They name exactly three properties, plus one more that everyone wishes were true and no real function actually delivers. Getting these four straight is the whole vocabulary, because every break later in this guide is best described as one specific property falling while the others stand.

Start with the object itself. A hash function $H$ maps an input of essentially unbounded length to an output of some fixed length $n$ bits: 256 bits for SHA-256, 160 for SHA-1, 128 for MD5. Because the input space is infinite and the output space has only $2^n$ points, collisions must exist by counting alone. Security is never about avoiding collisions in principle; it is about making them, and two related feats, computationally unreachable.

Phillip Rogaway and Thomas Shrimpton gave the field its rigorous, separated definitions of these properties in 2004, and it is worth stating them the way they did, because the separations matter ^[8].

The gap between $2^{n}$ and $2^{n/2}$ is the single most important number in this article. Preimage and second-preimage pit you against a fixed target, so you are searching a space of size $2^n$. Collision lets you range freely over pairs, and pairs accumulate quadratically.

The name comes from the party trick: in a room of just 23 people, two of them probably share a birthday, even though there are 365 candidate days, because 23 people form 253 pairs. For a 256-bit hash the same arithmetic says a collision surfaces after about $2^{128}$ digests rather than $2^{256}$, so a 256-bit hash buys you only 128 bits of collision resistance. This is why collision is the weakest joint, and, as the failure catalog will show, always the first to fall.

The following simulator makes the gap tangible. It uses a toy 24-bit hash so both effects fit in a browser tab: watch how few inputs it takes to stumble into a collision, and how hopeless it is to hit one fixed target by search.

// A toy 24-bit "hash": mix the input, then keep the low 24 bits (N = 2^24).
const N = 1 << 24;                 // 16,777,216 possible digests
function h(x) {
let v = Math.imul(x ^ 0x9e3779b1, 2654435761) >>> 0;
v ^= v >>> 15;                    // fold high bits down so the low bits mix
v = Math.imul(v, 0x85ebca6b) >>> 0;
v ^= v >>> 13;
return v & (N - 1);
}

// Collision search: hash 1, 2, 3, ... and stop at the first repeat.
const seen = new Map();
let collisionAt = null;
for (let i = 1; i < 300000 && collisionAt === null; i++) {
const d = h(i);
if (seen.has(d)) collisionAt = i;
else seen.set(d, i);
}
console.log("birthday bound, sqrt(N), is about", Math.round(Math.sqrt(N)));
console.log("first collision appeared after only", collisionAt, "inputs");

// Preimage search: how many tries to hit ONE fixed target digest?
const target = h(42), budget = 500000;
let tries = 0, hit = false;
for (let x = 5000000; x < 5000000 + budget; x++) {
tries++;
if (h(x) === target) { hit = true; break; }
}
console.log("preimage search used", tries, "tries and", hit ? "got lucky" : "found nothing");
console.log("average preimage cost would be about N/2 =", N / 2, "tries");

Collisions surface after only a couple of thousand inputs, while the fixed-target search burns through half a million tries and, on average, would need eight million. That is the $2^{n/2}$ versus $2^{n}$ split in miniature.

There is a fourth property people silently assume: that a good hash behaves like a random oracle, a magic box that returns a fresh uniformly random answer for every distinct input and is otherwise consistent. It is a wonderful modeling fiction, and much of applied cryptography is proved secure by pretending it is real.

But no fixed, finite, publicly specified function can actually be a random oracle, and Canetti, Goldreich, and Halevi proved the gap is not merely cosmetic: there exist schemes secure with an ideal oracle yet insecure under every concrete instantiation ^[9]. We make that precise in Section 10. For now, hold the three real clauses in mind, plus the knowledge that the fourth is a heuristic.

Two consequences follow immediately and set up everything after. First, you never sign a document; you sign its digest, so a signature is only as unforgeable as the hash is collision resistant. The formal security goals for signatures and message authentication codes are the subject of Part 1 of this series, the security-definitions field guide; here we only need the consequence. Second, the weakest clause is collision resistance, sitting at $2^{n/2}$. To see how that clause snapped in practice, we first have to see how the fingerprint is assembled, because the assembly leaks in ways the three clauses alone do not reveal.

3. How the Fingerprint Was Built: The Merkle-Damgard Era#

The hash function became load-bearing infrastructure almost by accident. When Whitfield Diffie and Martin Hellman introduced public-key cryptography in 1976, their signatures could only operate on short numbers, roughly the size of the key ^[10]. But people wanted to sign whole documents, contracts, and programs. The fix was quiet and consequential: do not sign the document, sign a short fingerprint of it. That single move loaded the entire weight of a signature onto the hash. If two documents can share a fingerprint, one signature covers both, so the signature scheme is only as sound as the hash beneath it.

Ralph Merkle's 1979 Stanford thesis formalized one-way hash functions and, in its "Tree Authentication" section, introduced the hash tree that now underlies Git, blockchains, and verified downloads ^[11].

The next question was how to build such a fingerprint for an input of any length out of a small, analyzable building block. In 1989, Ralph Merkle and, independently, Ivan Damgard supplied the answer that defined the next two decades ^{[12, 13]}.

That theorem is the reason the 1990s could build hashes with confidence: analyze one small function, get a hash for arbitrary inputs for free. The digest is simply the chaining state after the last block has been absorbed.

flowchart LR
IV["IV: fixed start state"] --> C1["compress"]
M1["block m1"] --> C1
C1 --> C2["compress"]
M2["block m2"] --> C2
C2 --> C3["compress"]
MP["final block: message plus length padding"] --> C3
C3 --> D["digest: the final chaining state"]

Ronald Rivest built the first widely used family on this shape. MD4 arrived in 1990 and the strengthened MD5 in 1992, both producing 128-bit digests from an add-rotate-xor compression function ^{[14, 15]}. RFC 1321 states MD5's purpose plainly: it compresses a message "in a secure manner before being signed with a private key" ^[16]. The design DNA, a message schedule feeding an ARX round function, propagated forward. In 1993 the U.S. government standardized a 160-bit hash of the same lineage in FIPS 180, retroactively called SHA-0, and in 1995 replaced it with SHA-1, a one-bit tweak of the same design ^[17]. SHA-1's specification survives in the current Secure Hash Standard ^[18], and it went on to hash most of the internet for roughly twenty years.

The Merkle-Damgard theorem promised something precise and limited: the whole hash is as collision resistant as its little compression function. It never promised that the shape wrapped around that function, the chaining, the padding, the exposed final state, was itself sound. That distinction is exactly where the first cracks opened, and they opened even for a hypothetically perfect compression function.

4. The Structural Cracks: Why Merkle-Damgard Leaks#

Here is the unsettling part. You can hand the Merkle-Damgard construction a perfect compression function, one with no weakness whatsoever, and the assembled hash still leaks. It leaks four separate ways, and none of them is a bug in any particular function. They are properties of the shape.

The first leak is the most practical, and it follows directly from the diagram in the previous section. The digest is the final chaining state. Nothing hides it. So if you know a digest and the length of the message that produced it, you can pick up the computation exactly where it left off.

The mechanism has exactly three moving parts, and it needs neither the message nor any secret. Publishing $H(m)$ publishes a fully loaded, resumable hash engine. From the length of $m$ the attacker rebuilds the glue padding, the 0x80 byte, the run of zeros, and the encoded bit-length that Merkle-Damgard appends before the final block ^[18]. The attacker reloads $H(m)$ as the IV, folds in the glue and then an arbitrary suffix, and reads off a valid digest for the longer message. When the server later hashes secret || m || glue || suffix it walks the identical chain, passes through the intermediate state $H(m)$, folds in the same suffix, and confirms the forgery.

flowchart LR
PUB["Public: digest H(m) and length of m"] --> REBUILD["Rebuild glue padding from the length alone"]
REBUILD --> RELOAD["Reload H(m) as the compression IV"]
RELOAD --> FOLD["Fold in glue, then attacker suffix"]
FOLD --> FORGE["Forged digest for m plus glue plus suffix"]
SERVER["Server hashes secret plus m plus glue plus suffix"] -.walks the identical chain.-> FORGE

This turns a natural-looking construction into a trap. Many engineers reach for H(secret || message) as a homemade message authentication code, reasoning that you cannot forge a tag without knowing the secret. Length extension refutes that: given a valid tag for one message, an attacker appends chosen data and computes a valid tag for the longer message, secret still unknown. The toy hash below is deliberately tiny and insecure, but it has the one property that matters, its digest is its whole internal state, so it forges end to end exactly as MD5 or SHA-256 would.

// A toy Merkle-Damgard hash. Like MD5 and SHA-256, its DIGEST IS ITS FINAL
// INTERNAL STATE -- that single fact is the entire length-extension bug.
const BLK = 8;
const IV = 0x1234abcd >>> 0;
const enc = s => [...s].map(c => c.charCodeAt(0));

function compress(state, byte) {          // one toy ARX step (not secure)
let s = (state + byte) >>> 0;
s = ((s << 7) | (s >>> 25)) >>> 0;
return Math.imul(s ^ 0x9e3779b1, 2654435761) >>> 0;
}
function fold(state, bytes) {             // raw fold, adds no padding
let s = state >>> 0;
for (const b of bytes) s = compress(s, b);
return s >>> 0;
}
function pad(len) {                       // MD strengthening from the LENGTH ALONE
const p = [0x80];
while ((len + p.length + 1) % BLK !== 0) p.push(0x00);
p.push(len & 0xff);                     // 1-byte toy length encoding
return p;
}
function toyHash(bytes) {                 // full hash = fold over message + padding
return fold(IV, bytes.concat(pad(bytes.length)));
}

// The server publishes (message, tag) where tag = H(secret || message).
const secret  = enc("SECRET");            // 6 bytes the attacker never sees
const message = enc("user=guest");
const tag = toyHash(secret.concat(message));

// The attacker knows tag, message, and only the LENGTH of the secret (6).
const secretLen = 6;
const glue   = pad(secretLen + message.length);   // rebuilt from lengths, no secret
const suffix = enc("&role=admin");
const total  = secretLen + message.length + glue.length + suffix.length;
const forgedTag = fold(tag, suffix.concat(pad(total)));    // resume FROM the tag
const forgedMessage = message.concat(glue).concat(suffix);

// The server verifies the forged message with its real secret.
const check = toyHash(secret.concat(forgedMessage));
console.log("forged tag (secret never used):", forgedTag.toString(16));
console.log("server recomputation          :", check.toString(16));
console.log("forgery accepted?", forgedTag === check);
console.log("smuggled 'role=admin' present?", String.fromCharCode(...forgedMessage).includes("role=admin"));

Real tools do this against real hashes. The canonical hash_extender demonstrates it on MD5: publish signature = MD5(secret || "data") = 6036708eba0d11f6ef52ad44e8b74d5b, and knowing only data, the algorithm, and that signature, an attacker loads the signature back into MD5's state and continues hashing a chosen suffix to forge a valid tag for secret || data || glue || suffix, never learning the secret. Its own summary is the whole lesson: "100% of the state needed to continue a hash is in the output of most hashing algorithms" ^[20].

A precise note you should carry forward, because it is the sharpest discriminator between functions in this whole article. Length extension afflicts hashes whose digest exposes the full final state: MD5, SHA-1, and the full-width SHA-256 and SHA-512. Truncation can close the leak, but only if it hides enough of the state. The truncated SHA-2 variants that hide at least 128 bits (SHA-384, SHA-512/224, SHA-512/256), the SHA-3 family and SHAKE, HMAC, and the BLAKE hashes are all immune. The dangerous middle case is SHA-224, which discards only 32 bits and is therefore still vulnerable at about $2^{32}$ work, a distinction Section 6 makes exact. Defer the full table there; just remember that "truncated" does not automatically mean "safe."

The other three leaks are subtler but drove the field toward new designs. In 2004, Antoine Joux showed that in any iterated hash, finding $2^k$ messages that all share one digest costs only about $k$ times the work of a single collision, not the vastly larger amount naive counting suggests ^[21]. The construction is almost embarrassingly direct: from the starting state, one birthday search finds a colliding pair of blocks that both drive the chain to the same next state; repeat from that state $k$ times; then every one of the $2^k$ ways to pick one block or the other at each step traces the identical chain to the identical final digest.

These multicollisions demolish a piece of folklore that had reassured a generation of engineers: that concatenating two independent hashes, $H_1(m) \,\|\, H_2(m)$, multiplies their strength. It does not. Build a large multicollision in the weaker half -- many messages that all share one digest under it, which is cheap for a Merkle-Damgard hash -- then birthday-search that set for a pair that also collides under the stronger half. The combined collision resistance is essentially that of the stronger single hash, not the sum. Hashing twice buys almost nothing.

Joux's multicollisions were also the engine for two deeper results. In 2005, John Kelsey and Bruce Schneier used expandable messages built from multicollisions to find second preimages on Merkle-Damgard hashes for far less than the ideal $2^n$: for a target of $2^k$ blocks, the cost falls to roughly $2^{n-k}$ plus overhead ^[22]. That matters because second-preimage resistance was supposed to be the sturdy $2^n$ clause, yet the shape leaks it for long messages. In 2006, Kelsey and Tadayoshi Kohno went further with the herding, or Nostradamus, attack: an attacker precomputes a branching diamond structure, publicly commits to a digest as if it were a prediction, and later herds almost any chosen prefix into that exact digest ^[23]. It weaponizes the construction into proof-of-prediction fraud.

Length extension, Joux multicollisions, long-message second preimages, and herding are flaws in the shape, not in the function. A stronger or longer compression function cannot fix them, because they do not depend on any weakness in that function. Only a different construction can. Hold that thought: it is exactly why, after the failures, the field did not just build a bigger Merkle-Damgard hash. It built a different machine.

Those four are leaks in an idealized world, where the compression function is flawless. They were serious enough to motivate truncated designs and, eventually, entirely new constructions. But they were still theoretical. What turned this subject from a seminar into a decade of emergency migrations was the other kind of failure: the compression function itself fell, and someone chose to weaponize it.

5. The Failure Catalog: Every Notable Break, Named#

A broken hash is a lab curiosity until someone forges a certificate with it. Twice, someone did, and once it was a nation-state. This section is the heart of the guide, because the two failure arcs, MD5 and SHA-1, are the evidence for everything the rest of it recommends. Each arc runs the same course: a theoretical crack, a practical collision, a technique that makes the collision meaningful, and finally a real forged trust anchor.

The MD5 arc opens with an early warning that the field under-heeded. In 1996, Hans Dobbertin found practical full collisions for MD4 and collisions in MD5's compression function under a chosen initial value, not yet the full hash but a clear structural crack ^[24]. MD5 stayed deployed anyway.

The warning became undeniable in August 2004, when Xiaoyun Wang, Dengguo Feng, Xuejia Lai, and Hongbo Yu announced full MD5 collisions computable in minutes, alongside collisions for MD4, HAVAL-128, and RIPEMD ^[25]. The following year, Wang and Yu published the peer-reviewed differential method behind it ^[26]. That is the theoretical death of MD5. Everything after is engineering.

The engineering that mattered was learning to choose the colliding inputs.

In 2007, Marc Stevens, Arjen Lenstra, and Benne de Weger built exactly that for MD5, and used it to construct two X.509 certificates with different identities but the same MD5 digest ^[27].

A year later the attack left the lab. At the 2008 Chaos Communication Congress, a seven-person team used a chosen-prefix MD5 collision, about $2^{49}$ MD5 compressions on a cluster of roughly 200 PlayStation 3 consoles, to obtain a genuine commercial certificate-authority signature over a benign certificate that simultaneously validated a forged intermediate certificate authority ^[1]. The forged certificate carried the basic-constraints CA flag set to TRUE, which meant every browser on earth would trust certificates it issued.

flowchart TD
P1["Benign prefix: an ordinary certificate request"] --> COL["Collision blocks appended to both sides"]
P2["Malicious prefix: a rogue CA certificate"] --> COL
COL --> SAME["Both certificates share one MD5 digest"]
SAME --> SIG["Commercial CA signs the benign certificate"]
SIG --> VALID["The same signature validates the rogue CA:TRUE certificate"]

The arc peaked in June 2012 with Flame, espionage malware that carried a code-signing certificate chaining to Microsoft. Flame's authors mounted a new chosen-prefix MD5 collision against a Microsoft sub-authority still using MD5 in its licensing service, letting them sign malware as "Microsoft" and distribute it through Windows Update. Marc Stevens later reconstructed the previously unknown collision variant from the certificate itself ^[2].

A nation-state had used a hash collision to subvert the update trust root of a billion machines. After Flame, MD5 in signatures was not deprecated as a matter of hygiene; it was radioactive.

SHA-1 traveled the same road a decade behind. In 2005, Xiaoyun Wang, Yiqun Lisa Yin, and Hongbo Yu published the first collision attack on the full SHA-1, at about $2^{69}$ operations, comfortably under the $2^{80}$ birthday bound its 160-bit output was supposed to guarantee ^[28]. That theoretical break is what prompted NIST to open the SHA-3 competition. The practical break took twelve more years of hardware progress. In February 2017, a team from CWI Amsterdam and Google announced SHAttered: the first practical identical-prefix SHA-1 collision, at roughly $2^{63.1}$ SHA-1 computations, about nine quintillion of them, costing the equivalent of 6,500 CPU-years plus 110 GPU-years ^[3]. The proof was the pair of PDFs from this article's opening, sharing the digest 38762cf7f55934b34d179ae6a4c80cadccbb7f0a.

In January 2020, Gaetan Leurent and Thomas Peyrin closed the arc with "SHA-1 is a Shambles," the first practical chosen-prefix SHA-1 collision at about $2^{63.4}$, which along the way cut the identical-prefix cost to about $2^{61.2}$ ^{[4, 29]}. To prove it mattered, they forged a PGP/GnuPG key certification, tracked as CVE-2019-14855, building on their own EUROCRYPT 2019 precursor that first pushed collisions toward chosen prefixes ^[30].

"All attacks that are practical on MD5 are now also practical on SHA-1." That is the Shambles team's own summary of what a chosen-prefix collision means: SHA-1 has caught down to MD5 ^[4].

timeline
title Two failure arcs, MD5 and SHA-1
1996 : Dobbertin cracks MD4, dents the MD5 compression function
2004 : Wang team announces practical full MD5 collisions
2005 : Wang, Yin, Yu attack full SHA-1 near 2 to the 69
2007 : Stevens builds chosen-prefix MD5 collisions
2008 : Rogue certificate authority forged near 2 to the 49
2012 : Flame forges a Microsoft-chained code-signing certificate
2017 : SHAttered, first identical-prefix SHA-1 collision
2020 : Shambles, first chosen-prefix SHA-1 collision

The attack costs are worth tabulating, because they show how a break moves from theory to a weekend on rented hardware.

There is a discipline the headlines miss, and it protects you from over-correcting. "MD5 and SHA-1 are dead" is true only for uses that depend on collision resistance.

Every hash in this catalog shared one design shape, the Merkle-Damgard chain of Section 3. The obvious question, and the one the field actually asked, was whether anything built the same way survived. One did, and understanding why is the turn from despair to design.

6. The Survivor: SHA-2, and Why It Did Not Fall#

Here is a paradox worth sitting with. SHA-2 is built the same way as SHA-1: the same Merkle-Damgard skeleton, the same add-rotate-xor compression style, designed by the same institution. By the logic of the last two sections it should have fallen too. Instead, in 2026, SHA-256 is the hash under most of the internet. What did the survivors do differently?

The family, standardized in FIPS 180-4, has six members: SHA-224, SHA-256, SHA-384, and SHA-512, plus the truncated SHA-512/224 and SHA-512/256 ^[18]. And the answer to the paradox is almost anticlimactic. SHA-2 did not adopt a new idea. It widened the margins of the old one.

Larger chaining state, a longer and more diffusing message schedule, more rounds, and distinct per-round constants together mean that the differential paths Wang's team rode through MD5 and SHA-1 do not propagate far enough to reach the full function. The evidence is two decades of cryptanalysis that keeps stalling short: the best published collision attacks reach only 31 of SHA-256's 64 rounds in practice ^[32], and 39 rounds in the weaker semi-free-start setting ^[33]. A 2024 effort using programmatic SAT solvers likewise reaches only reduced-round or modified-initial-value variants, never the full function ^[34]. That wide margin, 33 rounds to the deepest true collision and 25 even against the weaker semi-free-start attack, is the entire reason to trust it.

There is one wrinkle the designers half-anticipated, and it is the single most useful discriminator you can memorize. Full-width SHA-256 and SHA-512 are length-extension-vulnerable, exactly as Section 4 warned, because their digest is the whole final state. Most of the truncated members are not. But one truncated member still is, and it is the one people most often get wrong.

The catch is in that last clause. Truncation defeats length extension only when the hidden part of the state is too large to guess. SHA-384 hides 128 bits and SHA-512/256 hides 256; recovering either by brute force is infeasible. SHA-224, however, is SHA-256 with a different initial value and the 256-bit result cut to 224 bits, so it discards only one 32-bit word of state ^[35].

An attacker brute-forces that missing 32 bits in about $2^{32}$ compression calls, seconds on commodity hardware, recovers the full internal state, and then length-extends exactly as if nothing were hidden. SHA-224 is therefore length-extension-vulnerable with only a 32-bit margin, categorically different from the truncated variants that hide 128 bits or more.

The table below is the sharpest single reference in this guide. When someone asks whether a hash is safe to use as H(secret || message), this is the answer.

That hardware acceleration, combined with universal library support and FIPS validation, is why SHA-256 is the correct default almost everywhere its deployment footprint shows: TLS 1.3 computes its handshake transcript hash with the negotiated SHA-256 or SHA-384 ^[38], Bitcoin hashes every block header with double SHA-256 ^[39], and Git's next-generation object format is built on SHA-256 ^[40], alongside code-signing digests and certificate fingerprints. It earned that position not by being newest but by being unbroken and fast where it counts.

Keep one framing straight, because the whole next act depends on it. SHA-2 is a survivor of the supersession arc that ran MD5 to SHA-1 to SHA-2. It is not something a later hash replaced. But a healthy survivor carrying most of the world's traffic is still a monoculture, and the field had just learned, viscerally, what happens when everything rests on one design. So even though SHA-2 was fine, cryptographers deliberately built something that shared none of its DNA.

7. The Different Shape: The Sponge and SHA-3#

Imagine a hash with no exposed chaining state to steal, that hands you arbitrary-length output for free, and that shares no mathematical DNA with SHA-2. That is not a patch on Merkle-Damgard. It is a different machine.

NIST went looking for that machine deliberately. In November 2007, stung by the 2004 and 2005 collision attacks, it opened a public SHA-3 competition, explicitly asking for a hash built on a construction unlike SHA-2 so that a future break of one would not endanger the other ^[41]. Five finalists emerged from dozens of submissions: BLAKE, Grostl, JH, Keccak, and Skein. In October 2012, NIST selected Keccak, designed by Guido Bertoni, Joan Daemen, Michael Peeters, and Gilles Van Assche, and standardized it as FIPS 202 in August 2015 ^{[41, 6]}. Its engine is the sponge.

The capacity is the whole trick. Because those $c$ bits never touch the input or the output, an attacker who sees the digest learns nothing about them, and cannot reconstruct the internal state.

flowchart LR
M["Message blocks"] --> ABS["Absorb: XOR into rate, then permute"]
ABS --> STATE["State: rate lane plus capacity lane"]
STATE --> SQ["Squeeze: read the rate, permute, repeat"]
SQ --> OUT["Output: fixed digest or an XOF stream"]
CAP["Capacity lane, never read or written directly"] -.-> STATE

Two payoffs fall out of that single design decision. First, the sponge is length-extension-immune by construction, not by truncation: there is no exposed final state to resume from, so H(secret || message) is not forgeable the way it is on full-width SHA-2. Second, the squeeze phase can simply keep going, so the sponge is natively an extendable-output function.

The security is not folklore. Bertoni, Daemen, Peeters, and Van Assche proved the sponge is indifferentiable from a random oracle up to the capacity bound, which is the strongest positive statement you can make about a concrete construction ^{[42, 43]}. In two decades of analysis, practical collisions have reached only about 5 of Keccak's 24 rounds, leaving an enormous margin ^[44].

Now the load-bearing correction this whole guide exists to make. SHA-3 is not the successor to SHA-2 the way SHA-2 succeeded SHA-1. FIPS 202 chooses its words carefully.

FIPS 202 states that the four SHA-3 functions "supplement the hash functions ... in FIPS 180-4: SHA-1 and the SHA-2 family," and that together the two standards "provide resilience against future advances" because they "rely on fundamentally different design principles" ^[6].

The word is supplement, not supersede, and NIST is even blunter elsewhere: its Policy on Hash Functions says there is "no need to transition applications from SHA-2 to SHA-3" ^[45], and the superseded 2012 version of that policy put it more strongly still, with "no need or plan to transition" ^[46].

SHA-3 is not the replacement for SHA-2. It is the insurance policy. After watching one construction, Merkle-Damgard, carry MD5, SHA-1, and SHA-2 all at once, the field deliberately funded a second, unrelated construction so that the internet's trust machinery would never again rest on a single design. SHA-2 and SHA-3 coexist on purpose. That is diversification, not supersession, and it is a different axis from the MD5-to-SHA-1-to-SHA-2 ladder entirely.

The sponge bought genuine algorithmic diversity and length-extension immunity for free. What it did not buy, at least in plain software, was raw speed: on a Zen 4 core without special instructions, SHA3-256 runs at about 6.8 cycles per byte, roughly three times slower than hardware-accelerated SHA-256 ^[37]. That gap set the agenda for the last decade. Could you keep the security and the length-extension immunity and still go an order of magnitude faster? Two answers arrived, from two different shapes.

8. The Fast Hashes: BLAKE3, the Tree, and Where K12 Sits#

The command b3sum can hash a large file roughly ten times faster than sha256sum, and it does not cheat to get there ^[7]. It uses every CPU core and every SIMD lane at once. Understanding how it does that, and when you should let it, means separating two ideas people constantly conflate.

There are two axes here, and they are independent. The construction axis says how a hash is built: Merkle-Damgard (SHA-2), sponge (SHA-3, SHAKE, and, importantly, KangarooTwelve), and the BLAKE tree family derived from the ChaCha stream cipher. The use-case axis says what job you are hiring it for, and "modern high-throughput hash" is a job, currently filled by BLAKE3 and KangarooTwelve. Notice that those two fast hashes sit on different construction axes: one is a tree over an ARX core, the other is a sponge. Speed is not a construction.

The BLAKE line began as a SHA-3 finalist, a ChaCha-derived design ^[48]. It became practical with BLAKE2, specified in RFC 7693, which is directly keyable: feed it a key and it is a message authentication code with no HMAC wrapper needed ^[49]. Then in 2020, Jack O'Connor, Jean-Philippe Aumasson, Samuel Neves, and Zooko Wilcox-O'Hearn released BLAKE3, which took the decisive structural step: it arranges a round-reduced BLAKE2-style compression function inside a binary Merkle tree ^[7].

flowchart TD
ROOT["Root chaining value, extensible to an XOF"] --> P0["Parent node"]
ROOT --> P1["Parent node"]
P0 --> L0["Chunk 0, 1 KiB"]
P0 --> L1["Chunk 1, 1 KiB"]
P1 --> L2["Chunk 2, 1 KiB"]
P1 --> L3["Chunk 3, 1 KiB"]

The tree is what unlocks the speed. Because each 1 KiB chunk hashes independently, any number of SIMD lanes and threads can work in parallel, and the results combine afterward. The same structure gives BLAKE3 incremental updates and verified streaming, and it collapses five roles into one function: plain hash, keyed MAC, pseudorandom function, key-derivation function via its derive_key mode, and extendable output. On a Zen 4 core it runs at about 0.60 cycles per byte single-threaded, the fastest secure hash in the eBASH benchmarks, before you even add threads ^[37].

BLAKE3's own documentation states it plainly: it is "secure against length extension, unlike SHA-2" ^[7].

The other fast hash keeps you on the sponge. KangarooTwelve, with its TurboSHAKE core, is a tree hash over a round-reduced Keccak permutation, Keccak-p with 12 rounds instead of 24, published as the Informational RFC 9861 in October 2025 ^{[50, 51]}. It is genuinely fast: the vendor reports about 0.51 cycles per byte using AVX-512 and 0.75 on an Apple M1, though a generic build measures closer to 3.5 cycles per byte on Zen 4 ^{[52, 37]}. The point to hold onto, and the reason it gets a signpost rather than a section, is that KangarooTwelve is not a third construction family. It is the sponge again, tuned for throughput. It is standardized but still niche, and, like BLAKE3, not FIPS-approved.

Everything else is a one-line signpost. RIPEMD-160, Whirlpool, China's SM3, and Russia's Streebog are real and occasionally mandated regionally, but they are not the default anywhere the reader is likely to design for. The losing SHA-3 finalists, Grostl, JH, and Skein, are well-analyzed but standardized nowhere. None of them changes the decision you are about to make.

Fast, parallel, and diverse, yes. But so far we have only hashed raw data. Real protocols almost never use a bare hash. They key it, they separate its domains, and they stretch its output, and that is precisely where most real bugs live.

9. Beyond the Bare Hash: Keyed, Domain-Separated, and XOF Modes#

The most common hash bug in production is not a broken algorithm. It is a correct algorithm used in the wrong mode, and the specific mistake, again and again, is pressing H(secret || message) into service as a message authentication code. Section 4 showed why that is forgeable on full-width SHA-2. Here is what to use instead, and why it holds.

The canonical fix is HMAC, from Mihir Bellare, Ran Canetti, and Hugo Krawczyk in 1996 and RFC 2104 in 1997 ^{[53, 5]}.

Two things make HMAC the right tool. First, the outer hash of an inner hash structurally defeats length extension: the attacker never sees a resumable state. Second, and this is the payoff of the scoping discipline from Section 5, HMAC does not depend on the collision resistance of its hash. RFC 2104 states directly that the known collision weaknesses of MD5 "do not compromise the use of MD5 within HMAC" ^[5]. That is why HMAC-SHA-1 and HMAC-MD5 remain unbroken even though bare SHA-1 and MD5 are dead for signatures.

The keys HMAC needs have to come from somewhere with real entropy, which is the subject of Part 2 of this series, the field guide to cryptographic randomness.

HMAC is also the foundation of the most widely deployed key-derivation function, HKDF, which follows an extract-then-expand pattern: an extract step distills a uniform key from messy input keying material, and an expand step stretches it to any needed length ^[54]. HKDF is worth naming precisely because it is a mode built on a hash by way of HMAC, not a hash itself; when you need to turn a shared secret into keys, reach for it rather than hashing by hand.

The sponge offers a native alternative that needs no nesting, because it already resists length extension. NIST SP 800-185 defines four such functions: cSHAKE, KMAC, TupleHash, and ParallelHash ^[55]. KMAC is a keyed MAC that simply absorbs the key into the sponge. The other three are about domain separation.

XOFs change how you get short outputs, too. If you need a 20-byte value at a 128-bit security level, do not hash and chop arbitrary bytes off SHA-256; ask SHAKE128 or a BLAKE3 XOF for exactly 20 bytes. The security level is governed by the capacity, not by how many bytes you happen to read, so requesting the exact length is both cleaner and safer than ad hoc truncation. NIST is even revising SP 800-185 to add streaming specifications for SHAKE, a sign of how central XOFs have become ^[56].

Finally, the oldest mode of all scales integrity to enormous data: the Merkle tree from Ralph Merkle's 1979 thesis ^[11]. Hash the leaves, hash pairs of hashes upward to a single root, and you can prove any one leaf belongs to the root with a path of only $O(\log n)$ hashes. Git object trees, blockchain state commitments, Certificate Transparency logs, and BLAKE3's verified streaming are all this idea.

The one rule that separates a correct Merkle tree from a broken one is to domain-separate leaf nodes from internal nodes, so an attacker cannot pass an internal node off as a leaf, exactly the flag-bit discipline BLAKE3 bakes in. The sponge and BLAKE families also power keyed authenticated-encryption modes, which belong to the block-cipher and nonce-reuse parts of this series and are out of scope here.

We now have functions and modes that keep every clause of the promise. But every one of them still bows to a few hard mathematical ceilings, and to one machine that does not yet exist. How much should you actually worry?

10. Theoretical Limits: Bounds, the Random Oracle, and Quantum#

There is a best-possible hash, and you cannot build it. Understanding exactly why is what separates fear from calibrated caution, especially about quantum computers.

Start with the ceilings. For an ideal $n$-bit hash, collision resistance cannot exceed about $2^{n/2}$ work and preimage resistance cannot exceed about $2^{n}$. Those are not merely the best known attacks; they are the best any hash of that output length can offer, because the birthday and search arguments apply to a perfect random function. The good news, easy to miss, is that the deployed survivors meet those bounds exactly. The best published attack on full SHA-256, SHA-512, SHA3-256, BLAKE2, and BLAKE3 is the generic one, so there is no gap between what an attacker can do and the theoretical floor.

When cryptographers talk about a function's "security margin," they mean something narrower and attack-specific: the unbroken rounds between the deepest cryptanalysis of a given kind and the full round count. For SHA-256 collisions, published attacks reach 31 of 64 rounds (39 in the weaker semi-free-start setting), leaving a wide margin ^{[32, 33]}; for Keccak collisions, only about 5 of 24 rounds fall ^[44]. Wide margins are why these functions are trusted, not a gap in their strength.

The one clause where an ideal is genuinely out of reach is the random oracle. A fixed, finite, publicly specified function cannot actually behave like a fresh random answer for every input, for the plain reason that its code is right there to analyze. Ran Canetti, Oded Goldreich, and Shai Halevi made this precise: there exist signature and encryption schemes provably secure in the random-oracle model for which every concrete instantiation of the oracle by a real hash yields an insecure scheme ^{[9, 57]}. So the random-oracle model is a proof heuristic, not a guarantee.

The reachable substitute is indifferentiability: the sponge is provably indifferentiable from a random oracle up to its capacity bound, which is the strongest honest claim a real construction can make ^[42]. And a related limit from Section 4 still bites: Antoine Joux's multicollisions mean concatenating two hashes does not multiply their strength, so there is no cheap "hash twice" route to a stronger oracle ^[21].

Now quantum, where the folklore is loudest and least accurate. Grover's algorithm gives a square-root speedup for search, so a quantum preimage on an $n$-bit hash costs about $2^{n/2}$ instead of $2^n$ ^[58]. That halves the preimage exponent, which sounds alarming until you plug in numbers: SHA-256 keeps about 128 bits of preimage resistance against an ideal quantum adversary, still far beyond reach.

Collisions are the subtler story. Brassard, Hoyer, and Tapp gave a quantum collision algorithm at about $2^{n/3}$ queries, but it demands a comparable amount of quantum-accessible memory ^[59]. Daniel Bernstein's cost analysis shows that once you charge honestly for that memory, plain parallel classical collision search at $2^{n/2}$ with cheap hardware wins, so the practical quantum speedup for collisions is marginal ^[60]. SHA-256 therefore keeps roughly 128-bit collision resistance in practice, quantum or not.

Two frontiers sit just outside this guide's scope but deserve a pointer, because they are where hashes are heading. The first is arithmetization-oriented hashes such as Poseidon ^[61] and the Rescue and Vision family ^[62], designed to minimize arithmetic-circuit cost inside zero-knowledge proof systems rather than CPU cycles. They are younger, with far less mature cryptanalysis than SHA-2 or SHA-3, so they are a higher-risk choice outside their niche.

The second runs the other direction: hash-based signatures such as LMS ^[63], XMSS ^[64], and the standardized stateless SLH-DSA ^[65] build post-quantum signatures purely from hash calls, so they consume this primitive rather than being one. Both belong to the post-quantum posts in this series, not here.

So the mathematics delivers a calm verdict. The survivors are as strong as any hash of their output length can be, and quantum computers sharpen the analysis rather than overturning it. That means the last question is not "which hash is strongest," because among the survivors none is meaningfully stronger than another. The question is which function, in which mode, for your specific job.

11. The Practical Guide: Choose X with These Params in Case Y#

Everything so far collapses into one habit. Name the clause your protocol actually leans on, then pick the function and mode that keeps it. Here is that decision on a single page.

flowchart TD
START["What is the job?"] --> Q0{"Passwords or low-entropy secrets?"}
Q0 -->|Yes| ARGON["Argon2id, or scrypt, bcrypt, PBKDF2"]
Q0 -->|No| Q1{"Keyed authentication?"}
Q1 -->|Yes| HMAC["HMAC-SHA-256, or KMAC for the sponge"]
Q1 -->|No| Q2{"FIPS approval required?"}
Q2 -->|No| BLAKE["BLAKE3 for throughput, streaming, and KDF"]
Q2 -->|Yes| Q3{"Length-extension safety needed?"}
Q3 -->|Yes| TRUNC["SHA-512/256, or SHA-3 and SHAKE"]
Q3 -->|No| SHA256["SHA-256, the default"]

The head-to-head matrix fills in the details behind those leaves. Speeds are single-thread cycles per byte on a 2023 AMD Zen 4, from the eBASH measurements ^[37]; every full function listed is unbroken, so the choice is driven by length-extension behavior, speed, FIPS status, and parallelism, not by any strength gap.

And the same advice as a quick "use / never" card, the version you tape above your desk.

A few sizing rules turn those tables into correct code. For a target security level of $s$ bits against collisions, make the digest at least $2s$ bits, because collision resistance is only half the output length; 128-bit collision security therefore needs a 256-bit hash.

When you need a non-standard output length, request it from an XOF (SHAKE, KMAC, or a BLAKE3 XOF) rather than truncating a fixed digest by hand, and remember that reading more XOF bytes never raises you above the capacity ceiling. Add margin, SHA-384, SHA-512, SHAKE256, or KT256, when you want a quantum hedge, per Section 10. And keep the deprecation discipline precise: MD5 and CRC are fine as non-security checksums against accidental corruption, and nowhere an adversary can influence the input.

One operational detail sinks more code than any exotic attack: comparing digests or tags with an ordinary equality check. A loop that returns the moment two bytes differ leaks, through its timing, how many leading bytes matched, which is enough for an attacker to recover a secret tag byte by byte. The fix is a constant-time comparison that always scans the whole buffer. This adjacency between hashing and timing side channels is the same family of bug explored in the padding-oracle field guide.

// Comparing a secret tag with early-exit leaks how many bytes matched.
function naiveEqual(a, b) {
if (a.length !== b.length) return false;
for (let i = 0; i < a.length; i++) {
  if (a[i] !== b[i]) return false;      // returns as soon as bytes differ
}
return true;
}

// Constant-time: always scan the whole array, OR the differences together.
function constantTimeEqual(a, b) {
if (a.length !== b.length) return false;
let diff = 0;
for (let i = 0; i < a.length; i++) {
  diff |= a[i] ^ b[i];                   // no branch on the secret bytes
}
return diff === 0;
}

const tag   = [0x9f, 0x1c, 0x00, 0x42, 0xab];
const guess = [0x9f, 0x1c, 0xff, 0x00, 0x00];
console.log("naive:", naiveEqual(tag, guess));
console.log("constant-time:", constantTimeEqual(tag, guess));
console.log("In production call a vetted primitive: crypto.timingSafeEqual, CRYPTO_memcmp, or sodium_memcmp.");

Two rules cover most real disasters, and they have nothing to do with picking the "strongest" hash. Never use a bare fast hash for passwords, and never use a bare hash where you meant HMAC. The rest of choosing is about axes that are not strength at all: length-extension behavior, speed, FIPS status, parallelism, and construction diversity. What remains is to clear up the myths that cause the disasters those two rules do not.

12. Misconceptions and Real-Code Smells#

The field's folklore is mostly half-right, which is worse than fully wrong, because half-right beliefs survive code review. Here are the questions that actually decide those reviews.

The misuse catalog is the FAQ's mirror image: the same mistakes, seen from the code side.

Return, finally, to the two files from the opening, the ones that shared the SHA-1 digest 38762cf7f55934b34d179ae6a4c80cadccbb7f0a ^[3]. You can now read that string precisely. It is not evidence that hashing is broken, or that longer is always safer, or that SHA-3 rode in to replace the fallen. It is a single clause, collision resistance, failing on a single aging construction, and being weaponized because signatures sign the digest, not the file. The answer the field actually gave was not one replacement but two moves at once: widen the margins of the survivor, SHA-2, and diversify into unrelated constructions, the SHA-3 sponge and the BLAKE3 tree, so no future break can take everything down together. A cryptographic hash makes one promise. Mastery is knowing exactly which clause of that promise your protocol leans on, and choosing the function and the mode that keeps it.