Two Identical Requests: How Web Bot Auth and HTTP Message Signatures Let Servers Trust the Right Bots

1. Two Identical Requests#

Two HTTP requests hit your server in the same second. Both carry the identical Chrome User-Agent. Both arrive from the same shared cloud IP range. Both ask for the same page. One is OpenAI's Operator, fetching that page on behalf of a paying customer. The other is a scraper, harvesting your content to resell. You have milliseconds to decide which one to admit -- and with everything the web handed you across its first thirty years, you literally cannot tell them apart.

That is not a failure of effort. It is a failure of design. For decades the web learned to authenticate humans with steadily stronger cryptography: passwords gave way to asymmetric keys, and asymmetric keys gave way to passkeys bound to a specific origin. Machines got none of that. The web still governs automated clients the way a library governs its patrons -- with a posted set of rules and the hope that visitors behave. This article is the story of the layer that finally closes the gap -- and the one thing it still refuses to prove.

Start with why the problem is genuinely hard. Cloudflare, whose network sits in front of a large fraction of the web, framed it bluntly in a 2025 proposal titled "Forget IPs": the three signals an origin has today are each broken in a different way ^[3].

The User-Agent string is spoofable -- just text the client types, and agents deliberately send a real Chrome string. The source IP is brittle -- one address is routinely shared across unrelated users, privacy proxies, and whole companies renting the same cloud. A pre-shared secret, the classic API-key answer, is pairwise -- it forces every bot to arrange a separate credential with every site it might visit. None of the three answers the only question that matters: who is actually sending this request?

Before going further, fix the cast of characters. An AI agent here means an orchestrated browser -- often a real, automated instance of Chrome -- acting on a person's behalf: booking a flight, comparing prices, reading a page and summarizing it. From the server's side it looks exactly like a human's browser, because it frequently is one. That is precisely why the old signals collapse: the agent is not pretending to be Chrome, it is Chrome, driven by software instead of hands.

flowchart TD
A["AI agent acting for a user
real Chrome User-Agent, shared cloud IP"] --> O{"Origin: which do I admit?"}
B["Scraper reselling content
real Chrome User-Agent, shared cloud IP"] --> O
O --> Q["No verifiable identity signal
the two requests are indistinguishable"]

There is a deeper idea hiding in that question, and the whole article turns on it. "Who is sending this?" is a question about identity. "Should I let them in?" is a question about authorization. They feel like one decision at the door, but they are two, and conflating them is the mistake that makes bot management so miserable. Everything that follows gives the web a way to answer the first question with cryptographic certainty -- while being scrupulously honest that it never answers the second.

A signature can prove who sent a request without proving the sender is allowed to make it. Verifiable identity and authorization are different problems. Web Bot Auth solves the first and, by design, leaves the second to policy. Hold on to that distinction -- it is the hinge every later section turns on.

So the refined claim this article defends is narrower and more interesting than the headline. After decades of hardening how it authenticates people, the web is finally giving agents a cryptographic identity that is standardized and discoverable -- not the first-ever cryptography for machines, but the first kind any origin can simply look up. If you cannot trust what a bot says it is, and you cannot trust where it comes from, what is left to trust? The answer took the web thirty years to build. To see why it was so hard, we have to go back to 1994.

2. Identity for Humans, Politeness for Machines#

In 1993 and 1994, the first web robots "swamped servers with rapid-fire requests" and wandered into corners of sites they should have left alone ^[4]. There was no way for a server to tell a robot where not to go. Martijn Koster, working with a mailing list of the era's robot authors, proposed a fix and the group reached consensus on it on 30 June 1994. His fix was not a lock. It was a polite note taped to the door.

That note is robots.txt: a single file at a well-known path listing the URL paths a crawler is requested not to fetch. A well-behaved crawler downloads it once, reads the section addressed to its own self-declared name, and voluntarily stays out of the listed areas.

The founding text is remarkably candid about what it is not. The protocol, it says, "is not an official standard backed by a standards body," and it "is not enforced by anybody, and there [is] no guarantee that all current and future robots will use it" ^[4]. It was standardized twenty-eight years later as RFC 9309, which records that the protocol was "originally defined by Martijn Koster in 1994" ^[5].

Read those words carefully, because they contain the distinction the rest of this article depends on. robots.txt never authenticates anyone. The server advertises a policy and hopes; it never learns who is asking. RFC 9309 draws the line in one sentence: its rules "are not a form of access authorization" ^[5]. This is etiquette, not identity, and certainly not enforcement. A hostile scraper reads Disallow: /, crawls the whole site anyway, and the origin is left with neither a barrier nor a name.

If a policy file cannot identify anyone, the obvious next step is to let the client say who it is. The web already had a place for that: the User-Agent header, present since HTTP/1.0 in 1996 ^[6] and given its current normative definition in RFC 9110, section 10.1.5, in June 2022 ^[7].

A crawler announces itself -- User-Agent: MyBotCrawler/1.1 -- and the origin can, in principle, allow the crawlers it trusts and block the rest. For the first time the server has an identity claim it can act on, per request, without relying on the client to police its own behavior.

There is one catch, and it is fatal: the claim is unauthenticated. A header is just a string, and anyone can send any string.

The specification itself concedes the point. If a user agent "masquerades as a different user agent," it notes, recipients simply have to assume the client meant to be treated as whatever it claimed ^[7]. Cloudflare states the operational consequence without euphemism: user-agent strings "alone are easily spoofed and are therefore insufficient for reliable identification," and agents "often use the Chrome user agent for this very reason" ^[3]. The masquerade language lives in RFC 9110 section 10.1.5; the same sentence is reproduced, with that section citation, in Cloudflare's "Forget IPs" post ^[3], which is where many practitioners first meet it.

So the second generation of bot identity answers a subtly wrong question. User-Agent tells you who the client says it is, never who it is. And that is exactly the hook from the previous section: a legitimate agent and a scraper both send a real Chrome string, and the origin cannot separate them. A name anyone can type is not an identity. So perhaps the answer is to stop trusting what the bot writes and start trusting something it cannot simply type -- the network it comes from.

3. Anchoring Identity to the Network#

If a bot can lie about its name, tie its identity to something it cannot simply write down: the network it lives on. A random impostor can set any User-Agent, but it cannot easily forge control of a vendor's DNS or occupy a specific block of IP addresses. For roughly a decade, this was the strongest bot-verification signal the web had.

Google pioneered the technique for Googlebot, and the canonical version has two composable checks. The first is forward-confirmed reverse DNS.

Forging that requires controlling both directions of DNS for the vendor's domain, which a casual spoofer cannot do ^[8]. The second check is a published IP allowlist: the vendor publishes its address ranges as CIDR blocks in a file, and the origin checks whether the source IP falls inside them ^[3]. Google documents both paths today -- common crawlers reverse-resolve under googlebot.com and appear in a published common-crawlers.json, while special-case fetchers use a separate google.com mask ^[8].

This was a real improvement. Identity was now anchored to something the requester could not self-assert; the request had to originate from Google's footprint. "Verify Googlebot" became standard operating procedure, and for years it worked. The mechanism is a little older than any dated write-up, but Google's "How to verify Googlebot" post, dated 20 September 2006, is the first primary source that spells it out and calls reverse-plus-forward DNS "an official way to authenticate Googlebot" ^[9]. That is the debut this article pins to.

And then it broke -- not at the edges, but at its foundation. Reverse DNS and IP lists prove network location, and location is not identity. The entire premise is that a source IP maps to a single, identifiable operator, and modern infrastructure dissolves that mapping.

Cloudflare puts the failure plainly: the same IP address "might be shared by multiple users or multiple services within the same company, or even by multiple companies when hosting infrastructure is shared (like Cloudflare Workers)" ^[3]. Privacy proxies and VPNs interpose addresses that belong to neither the operator nor any stable identity. Serverless platforms hand out egress addresses that churn with the underlying infrastructure, which is why vendors resort to constantly-updated CIDR lists ^[3].

Picture the concrete case that matters now. An AI agent runs on a serverless platform and shares its outbound address with thousands of unrelated tenants. A reverse lookup on that address resolves to the cloud provider, not to the agent's operator. The origin can confirm "this came from a big cloud" and learns nothing about which automated client sent it.

Worse, the scheme is per-vendor and brittle in both directions: every operator must publish and maintain its own ranges, and every origin must track every operator's ever-shifting lists. Onboarding a new verified crawler historically meant hand-checking IP ranges and autonomous-system numbers -- exactly the manual toil the cryptographic approach was later built to retire ^[10].

The flaw, then, is not the mechanism. It is the premise. An IP address was never an identity; it was a stand-in for one, and stand-ins leak. To bind identity to the operator itself -- independent of whatever address it happens to borrow today -- you need something only that operator holds and can present on demand. You need a secret. Or, better still, a key.

4. Generation by Generation#

Step back from the individual mechanisms and a pattern appears: each one failed in a way that named its successor. robots.txt had no identity, so we added a self-declared name. The name was spoofable, so we anchored to the network. The network proved location instead of identity, so we reached for a secret. Six generations, each a better answer to the same stubborn question, each breaking in a way that forced the next.

Generation 4 deserves a moment, because it is where cryptography first almost arrived. A Bearer token or API key is a secret string issued out of band; the bot presents it on every request and the origin checks it against what it handed out. Identity now travels with the credential instead of the address, so changing IPs no longer breaks it.

But RFC 6750 is unusually honest about the model's ceiling: "Any party in possession of a bearer token (a 'bearer') can use it to get access to the associated resources (without demonstrating possession of a cryptographic key)" ^[11]. Possession is not proof -- anyone who intercepts or leaks the token can replay it.

And the secret is shared between exactly two parties, so a bot must arrange a distinct one with every origin it visits. Cloudflare names the wall directly: "You go to every website and share a secret ... This is impractical at scale because it requires developers to maintain separate tokens for each website their bot will visit" ^[3].

flowchart LR
G1["1994
robots.txt
honor system"] --> G2["1996
User-Agent
self-declared"]
G2 --> G3["2006
reverse DNS + IP
network-anchored"]
G3 --> G4["2012
Bearer / API keys
shared secret"]
G4 --> G5["2013-2019
SigV4, cavage, Stripe, mTLS
fragmented crypto"]
G5 --> G6["2024-2026
RFC 9421 + Web Bot Auth
standardized + discoverable"]

The fix for a leakable, pairwise secret is an asymmetric keypair: the bot signs with a private key it never discloses, and any origin verifies with the matching public key. One keypair works everywhere, interception no longer grants impersonation, and the secret never leaves the signer.

And here is the part most summaries get wrong -- the industry built exactly this, repeatedly, more than a decade before Web Bot Auth. AWS Signature Version 4 signed canonical requests at cloud scale ^[12]. Stripe authenticated webhooks with an HMAC over a timestamped payload ^[13]. draft-cavage-http-signatures, first published in May 2013, signed selected HTTP headers with an asymmetric key and was adopted de-facto across the ActivityPub and Mastodon Fediverse ^[14]. Mutual TLS presented client certificates. Cryptographic request identity for machines was not missing. It was everywhere -- and useless for the open web.

"It's not the only way that developers can sign requests ... AWS has used Signature v4, and Stripe has a framework for authenticating webhooks -- but Message Signatures is a published standard." -- Cloudflare, "Forget IPs" ^[3]

Why useless, if the cryptography was sound? Because every scheme was an island, and two gaps blocked open-web identity. First, no shared signature format: SigV4, Stripe, and draft-cavage each canonicalize and encode differently, so a stranger origin has no common way even to parse an unknown crawler's signature.

Second -- and this is the decisive gap -- no interoperable key discovery. The symmetric schemes hide a shared secret, which drags Generation 4's pairwise problem right back in. mTLS demands client-certificate provisioning that is famously awkward on the open web. And draft-cavage, though genuinely asymmetric, discovered keys in an application-specific way: a Mastodon server fetches the sender's actor document and reads its publicKeyPem field, which works only inside the Fediverse, not for an arbitrary origin meeting an arbitrary bot ^[15].

The successor spec states the gap in standards language: RFC 9421 "assumes verifiers have prior knowledge of signers' key material, requiring out-of-band key distribution ... This creates deployment friction" ^[16]. AWS's documentation describes the SigV4 protocol but never states a launch year; it is widely dated to roughly 2013-2014, and this article keeps that hedge rather than asserting a precise date ^[12].

Read the chain along two axes and the story sharpens. Unforgeability climbs the whole way down: from a string anyone can write, to a network position, to a leakable secret, to an unleakable private key. Interoperability does the opposite -- it regresses at Generation 5, which has the best cryptography of any generation before it and yet fails precisely because it has no common standard and no discovery.

The entire arc is a migration in key-distribution cost, from $O(N \times M)$ pairwise secrets and brittle per-vendor state toward a single $O(1)$ keypair backed by an $O(k)$ public directory. There is even a literal human through-line: Manu Sporny co-authored both the 2013 draft-cavage ^[14] and the 2024 RFC 9421 ^[1] -- the same person carrying the same idea from pre-standard sketch to real standard.

The breakthrough was never "add cryptography to bot requests" -- machines had signed HTTP requests since 2013. The novelty is two acts of standardization: a signature construction any verifier can parse (RFC 9421), and a public directory any origin can look up (Web Bot Auth). Interoperability and discovery, not signing, are the news.

Generation 5 had the cryptography right a full decade early. So why did signing a request fail to give bots an open-web identity? Because no two islands agreed on how to build a signature -- and the fix starts with one deceptively simple decision about what a signature should even cover.

5. RFC 9421 and the Signature Base#

The genius of RFC 9421 is a single, almost counterintuitive decision: do not sign the message. Sign a description of the message. Once you see why, the whole design of Generation 6 falls into place.

Here is the problem with signing raw bytes. An HTTP request rarely reaches its destination unchanged. Proxies reorder headers, add their own, re-encode bodies, and normalize whitespace. A signature over the literal bytes would shatter the moment any intermediary touched the request. So RFC 9421 inverts the approach: the signer names exactly which parts of the message it wants to commit to, and signs a canonical rendering of just those parts.

The signer chooses its covered components -- any mix of ordinary fields and derived components -- and appends one trailing line, @signature-params, that carries the metadata: created, expires, keyid, an optional alg, an optional nonce, and a tag. All of it serializes into a canonical string.

A signature base for a simple request looks like this. Each covered component is one line; the last line records the recipe itself:

"@method": GET
"@authority": example.com
"@path": /articles/latest
"@signature-params": ("@method" "@authority" "@path");created=1618884473;expires=1618884773;keyid="poqkLGiymh_W0uP6PZFw-dvez3QJT5SolqXBCW38r0U";tag="web-bot-auth"

That block is the entire mechanism in miniature. The private key signs exactly those four lines. The result travels in two Structured-Field headers: Signature-Input, the @signature-params recipe so a verifier knows what to rebuild, and Signature, which carries the signature bytes ^[1].

When the request has a body worth protecting, the signer adds a content-digest field -- a hash of the body defined by RFC 9530 -- to the covered set, so the body is committed to as well ^[17]. The serialization rules all come from Structured Field Values, RFC 8941, which give headers a strict, unambiguous grammar of items, lists, and dictionaries ^[18]. ※ Without a strict grammar, two implementations could serialize the same header differently and never agree on the signature base -- which is precisely how the Generation 5 islands failed to interoperate.

flowchart TD
R["HTTP request"] --> C["Pick covered components
@method, @authority, @path, chosen fields"]
C --> P["Append @signature-params
created, expires, keyid, tag"]
P --> B["Canonical signature base
one line per covered component"]
B --> S["Ed25519 sign the base"]
S --> H["Emit Signature-Input (recipe) and Signature (bytes)"]

Now the payoff, and it is the aha of the whole standard. Because the signature commits to chosen components rather than raw bytes, it survives intermediaries, re-encodings, and proxies -- the signer decides exactly what the signature guarantees and what it deliberately leaves free to change.

This is the precise property every Generation 5 island lacked. SigV4 and draft-cavage each had their own private notion of what to canonicalize; RFC 9421 makes that notion a published standard, so any conforming verifier -- in any language, at any origin -- can reconstruct the base and check the signature without prior agreement with the signer.

RFC 9421 is deliberately algorithm-agnostic. Ed25519 is the deployed default because it is fast, deterministic, and small enough to run per request at internet scale, but the standard commits to no single scheme.

The IANA "HTTP Signature Algorithms" registry -- Specification Required, so it can grow -- also profiles RSA-PSS-SHA512, RSA-PKCS1-v1.5-SHA256, ECDSA over P-256 and P-384, and HMAC-SHA256 ^[19]. The alg is a property of the key, not a mandate of the protocol, which matters later when quantum-resistant schemes need a home. RFC 9421 was edited by Annabelle Backman (Amazon), Justin Richer (Bespoke Engineering), and Manu Sporny (Digital Bazaar) -- the same Sporny who co-authored the 2013 draft-cavage, which is why the standard is best read as a revival rather than an invention ^[1].

One more precision, because the record is muddled. RFC 9421 is a Standards-Track document published in February 2024. Several AI-generated summaries date RFC 9421 to June 2023; the RFC Editor and IETF Datatracker both settle it as February 2024 ^[1]. When a source and a chatbot disagree on a standard's date, trust the RFC Editor.

RFC 9421 tells you how to sign a request so that anyone can verify it. But "anyone can verify it" hides a trapdoor. To check a signature, a verifier needs the signer's public key -- and RFC 9421 says nothing about where to find it. It "assumes verifiers have prior knowledge of signers' key material, requiring out-of-band key distribution" ^[16]. For a paying customer's agent meeting an origin it has never contacted, that missing half is not a detail. It is the entire problem. Solving it is what turns a signature into an identity.

6. Web Bot Auth: From a General Tool to a Bot Identity#

RFC 9421 is a general engine. Web Bot Auth is what happens when Cloudflare and Google point that engine at one specific job -- proving which bot sent a request -- and bolt on the piece the RFC deliberately left out. The result is authored by Thibault Meunier of Cloudflare and Sandor Major of Google, and it answers the trapdoor question from the last section head-on: how does a stranger find your key?

Start with the cast. Web Bot Auth defines a three-actor model. A User delegates a task. An Agent carries it out and signs each outbound request with its private key. An Origin receives the request and verifies the signature. The clean separation matters: the party that signs is the agent operator, not the human, and that distinction returns to haunt the governance discussion later. ※ Because the operator signs, one key can sit in front of thousands of different end users -- the seed of the intermediary-trust problem that resurfaces in the open problems.

On top of RFC 9421's freedom, the profile pins down the parts that must be fixed for strangers to interoperate.

The last row is the whole point -- the genuine novelty of the entire effort.

A signer publishes its public keys as a JWKS at the well-known path /.well-known/http-message-signatures-directory, and points to that directory in-band with a request header.

That self-protecting loop is the elegant part. The header says where the keys live; the signature covers the header; so the pointer to the keys is itself signed. And the directory response must in turn be signed -- with tag="http-message-signatures-directory", one signature per listed key -- so no one can mirror your directory at their own host and enroll as you ^[21]. The keyid closes the loop by naming the exact key without any external registry.

You can watch a keyid derive itself. The snippet below takes the real Ed25519 public key served by Cloudflare's live debug directory, canonicalizes its required members exactly as RFC 7638 prescribes, hashes, and base64url-encodes the result -- and out falls the same kid the directory advertises ^[22].

const NL = String.fromCharCode(10);
function rotr(n, x) { return (x >>> n) | (x << (32 - n)); }
function sha256(bytes) {
const K = [
  0x428a2f98,0x71374491,0xb5c0fbcf,0xe9b5dba5,0x3956c25b,0x59f111f1,0x923f82a4,0xab1c5ed5,
  0xd807aa98,0x12835b01,0x243185be,0x550c7dc3,0x72be5d74,0x80deb1fe,0x9bdc06a7,0xc19bf174,
  0xe49b69c1,0xefbe4786,0x0fc19dc6,0x240ca1cc,0x2de92c6f,0x4a7484aa,0x5cb0a9dc,0x76f988da,
  0x983e5152,0xa831c66d,0xb00327c8,0xbf597fc7,0xc6e00bf3,0xd5a79147,0x06ca6351,0x14292967,
  0x27b70a85,0x2e1b2138,0x4d2c6dfc,0x53380d13,0x650a7354,0x766a0abb,0x81c2c92e,0x92722c85,
  0xa2bfe8a1,0xa81a664b,0xc24b8b70,0xc76c51a3,0xd192e819,0xd6990624,0xf40e3585,0x106aa070,
  0x19a4c116,0x1e376c08,0x2748774c,0x34b0bcb5,0x391c0cb3,0x4ed8aa4a,0x5b9cca4f,0x682e6ff3,
  0x748f82ee,0x78a5636f,0x84c87814,0x8cc70208,0x90befffa,0xa4506ceb,0xbef9a3f7,0xc67178f2];
let h = [0x6a09e667,0xbb67ae85,0x3c6ef372,0xa54ff53a,0x510e527f,0x9b05688c,0x1f83d9ab,0x5be0cd19];
const l = bytes.length, withOne = l + 1;
const pad = (56 - (withOne % 64) + 64) % 64, total = withOne + pad + 8;
const m = new Uint8Array(total); m.set(bytes); m[l] = 0x80;
const dv = new DataView(m.buffer);
dv.setUint32(total - 4, (l * 8) >>> 0);
dv.setUint32(total - 8, Math.floor(l * 8 / 0x100000000));
const w = new Uint32Array(64);
for (let i = 0; i < total; i += 64) {
  for (let t = 0; t < 16; t++) w[t] = dv.getUint32(i + t * 4);
  for (let t = 16; t < 64; t++) {
    const s0 = rotr(7,w[t-15]) ^ rotr(18,w[t-15]) ^ (w[t-15] >>> 3);
    const s1 = rotr(17,w[t-2]) ^ rotr(19,w[t-2]) ^ (w[t-2] >>> 10);
    w[t] = (w[t-16] + s0 + w[t-7] + s1) >>> 0;
  }
  let a = h[0], b = h[1], c = h[2], d = h[3], e = h[4], f = h[5], g = h[6], hh = h[7];
  for (let t = 0; t < 64; t++) {
    const S1 = rotr(6,e) ^ rotr(11,e) ^ rotr(25,e);
    const t1 = (hh + S1 + ((e & f) ^ (~e & g)) + K[t] + w[t]) >>> 0;
    const S0 = rotr(2,a) ^ rotr(13,a) ^ rotr(22,a);
    const t2 = (S0 + ((a & b) ^ (a & c) ^ (b & c))) >>> 0;
    hh=g; g=f; f=e; e=(d+t1)>>>0; d=c; c=b; b=a; a=(t1+t2)>>>0;
  }
  h = [(h[0]+a)>>>0,(h[1]+b)>>>0,(h[2]+c)>>>0,(h[3]+d)>>>0,(h[4]+e)>>>0,(h[5]+f)>>>0,(h[6]+g)>>>0,(h[7]+hh)>>>0];
}
const out = new Uint8Array(32), odv = new DataView(out.buffer);
for (let i = 0; i < 8; i++) odv.setUint32(i*4, h[i]);
return out;
}
function base64url(bytes) {
let s = '';
for (const b of bytes) s += String.fromCharCode(b);
return btoa(s).split('+').join('-').split('/').join('_').split('=').join('');
}
// The Ed25519 public key served by Cloudflare's live Web Bot Auth directory:
const jwk = { crv: 'Ed25519', kty: 'OKP', x: 'JrQLj5P_89iXES9-vFgrIy29clF9CC_oPPsw3c5D0bs' };
// RFC 7638: keep only the required members, sort them, no whitespace.
const canonical = '{"crv":"' + jwk.crv + '","kty":"' + jwk.kty + '","x":"' + jwk.x + '"}';
const thumbprint = base64url(sha256(new TextEncoder().encode(canonical)));
console.log('Canonical JWK : ' + canonical);
console.log('Thumbprint    : ' + thumbprint);
console.log('Directory kid : poqkLGiymh_W0uP6PZFw-dvez3QJT5SolqXBCW38r0U');
console.log('Identical?    : ' + (thumbprint === 'poqkLGiymh_W0uP6PZFw-dvez3QJT5SolqXBCW38r0U'));

With the signature standard and the directory in place, the verifier's job is mechanical: fetch the directory, match the keyid thumbprint, reconstruct the signature base, verify the Ed25519 signature, and enforce the validity window. Only then -- as a wholly separate step -- does it decide policy.

sequenceDiagram
participant U as User
participant A as Agent
participant O as Origin
participant D as Key directory
U->>A: Delegate a task, fetch this page
A->>A: Build signature base, Ed25519 sign
A->>O: Send request with Signature-Input, Signature, Signature-Agent
O->>D: Fetch or read cached directory at the well-known path
D-->>O: Return signed JWKS of public keys
O->>O: Match keyid thumbprint, reconstruct base, verify
O->>O: Enforce created ≤ now ≤ expires
O-->>A: Identity established, then apply policy separately

Ed25519 is the deployed default here, but the mechanism stays algorithm-agnostic: the same registry that lets RFC 9421 carry RSA-PSS or ECDSA lets Web Bot Auth carry them too ^[19]. Deployment is real but early, and precision matters. The directory draft registers the plural path /.well-known/http-message-signatures-directory ^[16], while some Cloudflare prose uses a singular alias; the live server answers on both, and this article uses the registered plural form ^[10]. The strongest independent signal that this is not vendor theater is OpenAI.

"With HTTP Message Signatures (RFC 9421), OpenAI signs all Operator requests so site owners can verify they genuinely originate from Operator and haven't been tampered with." -- Eugenio, Engineer, OpenAI ^[3]

With a signature and a directory, an origin can verify a bot it has never met, on first contact, with zero prior arrangement. That is the headline. But Cloudflare shipped a second proposal in the very same blog, and there is a whole family of rival ways to prove a machine's identity. Why did signatures win?

7. Signatures, Certificates, and the Other Signers#

Cloudflare proposed two mechanisms in the same post, and the one it did not prioritize explains why the one it did is pulling ahead. The road not taken is request mTLS.

Classic mTLS is decades old and rock-solid inside a managed setting where both sides already run a private certificate authority. The problem is the open web. If an origin asks for a client certificate and the client has none, the browser shows "an inscrutable" and unskippable error ^[3]. To soften that, draft-jhoyla-req-mtls-flag adds a TLS flag -- an experimental extension -- by which a client signals it holds a suitable certificate, so the server can request one only when it will not break an ordinary visitor ^[25].

It is a real, live parallel track. But Cloudflare is candid about ranking it second: it prioritizes Message Signatures because that approach "relies on the previously adopted RFC 9421 with several reference implementations, and works at the HTTP layer" ^[3]. The reference-implementation point is concrete -- RFC 9421 already has an implementations hub and independent libraries to build on ^[26].

flowchart TD
subgraph HL["HTTP layer -- Web Bot Auth"]
A1["Signature travels inside headers"] --> A2["Survives CDNs and reverse proxies"]
end
subgraph TL["Transport layer -- request mTLS"]
B1["Identity bound to the TLS connection"] --> B2["Lost when a proxy terminates TLS upstream"]
end

The layer difference is not cosmetic. A signature lives in HTTP headers, so it rides through content delivery networks and reverse proxies untouched and can be verified at the edge or the origin. A client certificate binds to the connection, so a reverse proxy that terminates TLS -- how much of the web is commonly served -- strips the identity before the request reaches the application. For an internet built on intermediaries, HTTP-layer identity is simply the one that arrives.

Web Bot Auth also clarifies its relationship to the Generation 5 signers. It keeps draft-cavage's asymmetric idea but replaces the Fediverse-specific publicKeyPem lookup with a standard format and an open-web directory ^{[14] [15]}; and unlike SigV4 or Stripe, it needs no shared secret, only a public key the origin can fetch. Every path so far -- reverse DNS, mTLS, SigV4, Web Bot Auth -- differs less in its cryptography than in the row of the table that has no cryptography at all: how a verifier discovers the key.

Method D in that table is not really a rival at all -- it answers the opposite question.

Line the methods up by how a verifier learns the key and the pattern is stark. mTLS and SigV4 distribute keys out of band. IP schemes distribute a public list of ranges. Only Web Bot Auth carries an in-band pointer to a standardized, discoverable directory -- the one design choice, containing no new cryptography, that makes stranger-to-stranger verification possible on the open web.

And yet notice what unites every row, signatures included. Each mechanism proves exactly one kind of thing: who, or what network, or what property. Not one of them, however strong its cryptography, can tell you whether the bot should be admitted. That gap is not an engineering oversight to be patched in the next draft. It is a limit on what a signature is -- and it is time to look at it directly.

8. What a Signature Cannot Prove#

Here is the sentence the whole article has been walking toward. A valid signature proves who signed and what they covered -- and nothing else. Not that the bot is well-behaved. Not that it respects your rate limits. Not that it is allowed anywhere near your origin.

This is not a shortcoming to be engineered away in a later draft; it is the definition of what a signature is. A digital signature is a proof of key possession over a covered message. It can establish authenticity -- which key signed -- and integrity -- that the covered components were not altered. It is information-theoretically silent on everything else.

"Allowed" is a predicate over policy and behavior, not over bytes, so no amount of cryptographic strengthening can make a signature answer it. The web already learned this once: RFC 9309 says the rules in a robots.txt file "are not a form of access authorization" ^[5]. Web Bot Auth simply makes the identity half verifiable while leaving that same authorization line exactly where it was.

flowchart TD
R["Signed request arrives"] --> V["Verify signature and validity window"]
V --> I["Identity established: who signed, what was covered"]
I --> G{"Policy gate, not cryptography"}
G -->|trusted and well-behaved| Allow["Admit"]
G -->|unknown or abusive| Deny["Deny or rate-limit"]

Three concrete limits make the boundary vivid, and each is fundamental rather than incidental.

First, replay. A signature with a validity window can be captured and replayed by anyone until it expires. The only defenses are a short window and a server-tracked nonce -- and the moment the server tracks a nonce, it is keeping state. This is a genuine lower bound, not a missing feature: stateless anti-replay is impossible. You trade window size against verifier memory, and Web Bot Auth's mandatory created/expires plus optional 64-byte nonce sit exactly on that trade-off. A verifier that admits a request only when $created \le now \le expires$ has narrowed the replay opportunity, never closed it ^[3].

Second, uncovered components. A signature commits only to what it lists. Anything the signer left out of the covered set can be altered in flight without breaking verification. That is why the profile insists on covering @authority and Signature-Agent -- so the target host and the pointer to the key directory cannot be quietly swapped. Coverage is a choice, and everything outside it is unprotected by construction.

Third, the trust-anchor problem, which is the deepest of the three. A verified signature does not create trust; it relocates it. "Do I trust this request?" becomes "do I trust the directory behind this key?" Cryptography cannot manufacture the first bit of trust in which signers matter -- that decision has to come from somewhere outside the math. It is the same unanswered question that sits under the entire web public-key infrastructure: who gets to be a certificate authority? Web Bot Auth relocates that question to "whose directory do I honor?" and, by design, leaves it open ^[16].

One more frontier the standards make concrete by existing side by side: Web Bot Auth points toward accountable, linkable identity, Privacy Pass toward unlinkable anonymity ^[27]. You can pick a point on that axis, but not both endpoints in one token -- a design decision, not a bug awaiting a fix.

A valid signature proves who signed and what was covered -- never whether the bot is allowed. Signing makes a bad bot accountable, not good.

It is worth naming what these limits are not. This domain has no impossibility theorems in the Turing sense, no proven complexity lower bounds, no hardness results. The honest "bounds" here are conceptual: they describe what a signature can attest, not how expensive it is to compute. Pretending otherwise -- dressing a category boundary up as a big-O result -- would be a fabrication. The real limit is cleaner and more absolute than any asymptotic one.

Identity and authorization are not two ends of one dial. They are different categories. A signature is silent on "allowed" the way a photograph is silent on "true" -- not because it is a weak signature, but because "allowed" is not the kind of thing a signature can encode. This is a boundary, not a backlog item.

So where does "allowed" get decided? Not here.

If a signature only proves who, then "who do we trust, and who decides?" becomes the entire game. And nobody has fully answered it. That unfinished question opens onto the frontier of governance.

9. Governance, Revocation, and the Legitimacy Gap#

Verifiable identity solves the easy half of the problem. The hard half -- who deserves trust, and how you take it back -- is wide open, and almost none of it is a cryptography question. Here are the eight problems that will decide whether Web Bot Auth becomes the default or a footnote.

1. Signer accreditation. A signature proves who signed, never whether that signer deserves trust. Someone still has to decide which directories an origin honors. Cloudflare's Verified Bots Program, backed by its BotBase and Radar data, is a de-facto answer -- but a single-vendor one ^[23]. The open problem is a neutral, multi-party accreditation model, without which every origin rebuilds the per-vendor allowlist that this whole effort set out to retire.

2. Revocation and rotation at web scale. A leaked key needs to be pulled fast and globally. Short expires windows bound the exposure, and a signer can drop a key from its JWKS, but there is no low-latency revocation or transparency mechanism -- nothing like the certificate world's revocation lists or transparency logs -- for these directories yet ^[16].

3. Directory availability and denial of service. Verification depends on fetching a third party's well-known directory. When that directory is slow, down, or flooded, the request hot path inherits a new availability and amplification surface. Signing the directory response makes cached copies tamper-evident, but cache-lifetime and denial-of-service guidance is still thin ^[21].

4. The legitimacy gap. A signed bot can still scrape aggressively, ignore robots.txt, or blow past rate limits. Signing makes it accountable, not good. Cloudflare's response is to bundle identity with a behavioral bar -- a Verified bot must both self-identify honestly and be non-abusive, obeying robots.txt at reasonable rates ^[23]. That is policy bolted on top of identity, exactly as the identity-versus-authorization spine predicts.

5. Transitive trust for intermediary agents. An "agent mode" browser is driven by many different end users, so trusting the operator's key is not the same as trusting every person behind it. The party that signs is not the party ultimately responsible for a given action. Cloudflare's July 2026 taxonomy adds a Direct-versus-Intermediary label and is experimenting with forwarding end-user information in the standard Forwarded header, so an origin can apply policy to the responsible party ^[23].

6. Adoption bootstrapping. Signers gain little until origins verify, and origins gain little until signers sign -- the classic two-sided chicken-and-egg. The escape is asymmetric scale: a large reverse proxy verifying by default seeds millions of origins at once ^[10], while a marquee signer like OpenAI's Operator seeds the client side ^[3].

7. Standardization status and version skew. The profile is a set of Individual Internet-Drafts, not RFCs, and it is a moving target: the architecture draft was renamed mid-flight to draft-meunier-webbotauth-httpsig-protocol ^[24], while production pins to earlier revisions than the latest. Interop can break silently between draft versions until a working group freezes the text.

8. Post-quantum migration. Ed25519, RSA, and ECDSA are all breakable by a sufficiently large quantum computer. A long-lived signing infrastructure should become crypto-agile before it must. RFC 9421's algorithm registry is extensible by design, and post-quantum signature schemes are being explored for it, but none is the deployed default today ^[19].

None of these are reasons to wait. The identity layer they sit on is already real, already standardized at its base, and already signing live traffic from a marquee agent. Which leaves one intensely practical question, the one an engineer actually needs answered: how do you do this, on the wire?

10. Doing It on the Wire#

Enough theory. Here is the whole protocol as an engineer meets it, in two roles.

If you are building a bot, there are four steps. Generate an Ed25519 keypair. Publish your public keys as a JWKS at /.well-known/http-message-signatures-directory, and sign that directory response itself with tag="http-message-signatures-directory", one signature per key ^[21]. On each outbound request, build a signature base that covers at least @authority and signature-agent, set a short created/expires window, set keyid to your key's JWK Thumbprint ^[20], and set tag="web-bot-auth". Then emit three headers -- Signature-Agent, Signature-Input, and Signature. On the wire, using the current directory draft's Structured-Dictionary URI for Signature-Agent ^[16], that is:

Signature-Agent: sig="https://signer.example.com"
Signature-Input: sig=("@authority" "signature-agent";key="sig");created=1700000000;expires=1700011111;keyid="ba3e64==";tag="web-bot-auth"
Signature: sig=abc==

Three lines carry the entire identity claim ^[3]. Signature-Agent says where the keys live, Signature-Input is the recipe, and Signature is the proof.

If you are the origin, the flow reverses. Parse Signature-Agent to learn where the directory is. Fetch it -- or reuse a cached copy -- and match the request's keyid to a published key. Reconstruct the signature base byte-for-byte from the same covered components, verify the Ed25519 signature against the matched key, and enforce that the request falls inside its created/expires window. Only after all of that succeeds do you reach the separate, non-cryptographic step: apply policy.

The one part worth running yourself is the signature base, because byte-identical reconstruction is where implementations most often fail. The snippet below assembles exactly the string a signer signs, then runs the validity-window check the verifier performs.

const NL = String.fromCharCode(10);
// Build the RFC 9421 signature base: the exact string a signer signs
// and a verifier must reconstruct byte-for-byte.
function componentId(c) {
return '"' + c.name + '"' + (c.key ? ';key="' + c.key + '"' : '');
}
function buildSignatureBase(covered, values, params) {
const lines = [];
for (const c of covered) {
  lines.push(componentId(c) + ': ' + values[c.name]);
}
const innerList = '(' + covered.map(componentId).join(' ') + ')';
const sigParams = innerList
  + ';created=' + params.created
  + ';expires=' + params.expires
  + ';keyid="' + params.keyid + '"'
  + ';tag="' + params.tag + '"';
lines.push('"@signature-params": ' + sigParams);
return lines.join(NL);
}
// Web Bot Auth MUST cover @authority; draft-05 covers the signature-agent ;key="sig" member.
const covered = [
{ name: '@authority' },
{ name: 'signature-agent', key: 'sig' }
];
const values = { '@authority': 'example.com', 'signature-agent': '"https://signer.example.com"' };
const params = {
created: 1700000000,
expires: 1700000300,
keyid: 'poqkLGiymh_W0uP6PZFw-dvez3QJT5SolqXBCW38r0U',
tag: 'web-bot-auth'
};
const base = buildSignatureBase(covered, values, params);
console.log('--- signature base (input to Ed25519) ---');
console.log(base);
// The origin enforces the validity window BEFORE trusting the signature.
const now = 1700000300;
const withinWindow = params.created <= now && now <= params.expires;
console.log('');
console.log('created <= now <= expires : ' + (withinWindow ? 'VALID' : 'REJECT'));
console.log('the real signing is then one call: ed25519.sign(privateKey, base)');

That is genuinely the whole computation. The actual signing is a single library call on the string the snippet builds, and verification is the same string plus one verify call.

You do not have to build this from scratch. The reference implementations are first-party: an npm web-bot-auth package ^[30], a Rust crate, a Cloudflare Workers verifier, and a Caddy plugin, all in the public repositories ^{[31] [32]}, plus the broader RFC 9421 implementations hub ^[26]. Best of all, there is a live directory you can hit right now. Cloudflare runs a live debug directory at http-message-signatures-example.research.cloudflare.com; its well-known path serves a real Ed25519 JWK whose kid is poqkLGiymh_W0uP6PZFw-dvez3QJT5SolqXBCW38r0U -- the very key the earlier thumbprint snippet reproduces ^[22].

Fetch that directory, feed its key into the verifier, and you have reproduced the entire mechanism end to end. The discovery layer that RFC 9421 left out is the part you can now watch working, live ^[16].

Stand back and the mechanics are almost anticlimactic: a keypair, a JSON file at a well-known URL, and two request headers. The hard part was never the code. It was getting the entire web to agree on the same keypair format, the same JSON document, the same two headers -- so that an origin and a bot that have never met can still understand each other. That agreement is the thirty-year story you just read.

11. Frequently Asked Questions#

After Thirty Years, an Identity#

Return to the two requests from the opening -- the agent and the scraper, identical down to the byte. The web spent its first three decades unable to separate them, not for lack of trying but because every tool it reached for answered the wrong question. robots.txt asked bots to behave. User-Agent let them name themselves. Reverse DNS pinned them to a network. Bearer tokens and per-vendor signatures added real cryptography but locked it inside islands no stranger could reach. Each was a better answer to "which bot is this?" and each broke in a way that named its successor.

RFC 9421 and Web Bot Auth close the loop with two acts of standardization the previous generations never managed: a signature construction any verifier can parse, and a public directory any origin can look up. That is the honest shape of the claim. Not the first cryptography for machines -- the first cryptographic identity for machines that is standardized, discoverable, and open to the whole web. A bot generates one keypair, publishes one signed JSON document at one well-known URL, and any origin on earth can verify it on first contact, with no prior arrangement.

After decades of hardening how it authenticates humans, the web has finally given agents a verifiable cryptographic identity -- finally standardized and discoverable, not first-ever. And it proves exactly one thing: who signed, never whether they are allowed.

That last clause is the discipline the whole article kept. A signature makes a bot accountable, not good. It relocates trust, it does not manufacture it. Everything above the identity floor -- who deserves trust, how you revoke it, what "allowed" even means for an agent acting for a thousand different users -- is governance, and governance is unfinished. The door now knows who is knocking. Deciding whether to open it is a different, still-unwritten story, and it begins exactly where this one ends.