# Two Identical Requests: How Web Bot Auth and HTTP Message Signatures Let Servers Trust the Right Bots

> How RFC 9421 and Web Bot Auth give bots and AI agents a standardized, directory-backed cryptographic identity -- and why it proves who, never whether-allowed.

*Published: 2026-07-20*
*Canonical: https://paragmali.com/blog/two-identical-requests-how-web-bot-auth-and-http-message-sig*
*© Parag Mali. All rights reserved.*

---
<TLDR>
For thirty years the web authenticated humans with ever-stronger cryptography while governing bots with etiquette (`robots.txt`), self-declared names (`User-Agent`), and network guesswork (IP allowlists) -- none of which can tell a legitimate AI agent apart from a scraper. RFC 9421 (HTTP Message Signatures, February 2024) [@rfc9421] and the Cloudflare and Google **Web Bot Auth** drafts [@draft-webbotauth-arch] fix this: a bot signs selected *components* of each request with an Ed25519 key and publishes its public keys in a discoverable JWKS directory, so any origin can verify a stranger bot on first contact. The genuine novelty is not the cryptography -- machines have signed requests since AWS SigV4 and ActivityPub [@cf-blog-webbotauth] -- but a **standardized, directory-backed, open-web identity anyone can look up**. And it proves exactly one thing: *who signed*, never *whether they are allowed*.
</TLDR>

## 1. Two Identical Requests

Two HTTP requests hit your server in the same second. Both carry the identical Chrome `User-Agent`. Both arrive from the same shared cloud IP range. Both ask for the same page. One is OpenAI's Operator, fetching that page on behalf of a paying customer. The other is a scraper, harvesting your content to resell. You have milliseconds to decide which one to admit -- and with everything the web handed you across its first thirty years, you literally cannot tell them apart.

That is not a failure of effort. It is a failure of design. For decades the web learned to authenticate *humans* with steadily stronger cryptography: passwords gave way to asymmetric keys, and asymmetric keys gave way to passkeys bound to a specific origin. Machines got none of that. The web still governs automated clients the way a library governs its patrons -- with a posted set of rules and the hope that visitors behave. This article is the story of the layer that finally closes the gap -- and the one thing it still refuses to prove.

Start with why the problem is genuinely hard. Cloudflare, whose network sits in front of a large fraction of the web, framed it bluntly in a 2025 proposal titled "Forget IPs": the three signals an origin has today are each broken in a different way [@cf-blog-webbotauth].

The `User-Agent` string is *spoofable* -- just text the client types, and agents deliberately send a real Chrome string. The source IP is *brittle* -- one address is routinely shared across unrelated users, privacy proxies, and whole companies renting the same cloud. A pre-shared secret, the classic API-key answer, is *pairwise* -- it forces every bot to arrange a separate credential with every site it might visit. None of the three answers the only question that matters: *who is actually sending this request?*

<Definition term="Web Bot Auth">
The set of Cloudflare- and Google-authored IETF Internet-Drafts that profile RFC 9421 (HTTP Message Signatures) for the specific job of bot identity, and add the piece the RFC leaves out -- a discoverable public key directory. Together they let an origin cryptographically verify *which* automated client sent a request, with no prior arrangement.
</Definition>

Before going further, fix the cast of characters. An **AI agent** here means an orchestrated browser -- often a real, automated instance of Chrome -- acting on a person's behalf: booking a flight, comparing prices, reading a page and summarizing it. From the server's side it looks exactly like a human's browser, because it frequently *is* one. That is precisely why the old signals collapse: the agent is not pretending to be Chrome, it *is* Chrome, driven by software instead of hands.

<Mermaid caption="Two byte-identical requests reach an origin with the same User-Agent and the same shared IP range. Nothing in either request lets the server tell the legitimate agent from the scraper.">
flowchart TD
    A["AI agent acting for a user<br/>real Chrome User-Agent, shared cloud IP"] --> O&#123;"Origin: which do I admit?"&#125;
    B["Scraper reselling content<br/>real Chrome User-Agent, shared cloud IP"] --> O
    O --> Q["No verifiable identity signal<br/>the two requests are indistinguishable"]
</Mermaid>

There is a deeper idea hiding in that question, and the whole article turns on it. "Who is sending this?" is a question about *identity*. "Should I let them in?" is a question about *authorization*. They feel like one decision at the door, but they are two, and conflating them is the mistake that makes bot management so miserable. Everything that follows gives the web a way to answer the first question with cryptographic certainty -- while being scrupulously honest that it never answers the second.

> **Key idea:** A signature can prove *who* sent a request without proving the sender is *allowed* to make it. Verifiable identity and authorization are different problems. Web Bot Auth solves the first and, by design, leaves the second to policy. Hold on to that distinction -- it is the hinge every later section turns on.

So the refined claim this article defends is narrower and more interesting than the headline. After decades of hardening how it authenticates people, the web is finally giving agents a cryptographic identity that is *standardized and discoverable* -- not the first-ever cryptography for machines, but the first kind any origin can simply look up. If you cannot trust what a bot *says* it is, and you cannot trust *where* it comes from, what is left to trust? The answer took the web thirty years to build. To see why it was so hard, we have to go back to 1994.

## 2. Identity for Humans, Politeness for Machines

In 1993 and 1994, the first web robots "swamped servers with rapid-fire requests" and wandered into corners of sites they should have left alone [@robotstxt-orig]. There was no way for a server to tell a robot where not to go. Martijn Koster, working with a mailing list of the era's robot authors, proposed a fix and the group reached consensus on it on 30 June 1994. His fix was not a lock. It was a polite note taped to the door.

That note is `robots.txt`: a single file at a well-known path listing the URL paths a crawler is *requested* not to fetch. A well-behaved crawler downloads it once, reads the section addressed to its own self-declared name, and voluntarily stays out of the listed areas.

The founding text is remarkably candid about what it is not. The protocol, it says, "is not an official standard backed by a standards body," and it "is not enforced by anybody, and there [is] no guarantee that all current and future robots will use it" [@robotstxt-orig]. It was standardized twenty-eight years later as RFC 9309, which records that the protocol was "originally defined by Martijn Koster in 1994" [@rfc9309].

Read those words carefully, because they contain the distinction the rest of this article depends on. `robots.txt` never authenticates anyone. The server advertises a policy and hopes; it never learns *who* is asking. RFC 9309 draws the line in one sentence: its rules "are not a form of access authorization" [@rfc9309]. This is etiquette, not identity, and certainly not enforcement. A hostile scraper reads `Disallow: /`, crawls the whole site anyway, and the origin is left with neither a barrier nor a name.

<Aside label="The other half of the story: authenticating humans">
While machines got etiquette, people got cryptography -- and steadily more of it. The web moved from passwords (a shared secret the server stores and the user retypes), to asymmetric second factors, to [WebAuthn and passkeys](/blog/webauthn-and-passkeys-on-windows-from-ctap-to-the-credential/), where the browser holds a private key bound to a specific origin and proves possession without ever transmitting the secret. That is the exact shape of guarantee -- a private key that signs, a public key that verifies, tied to an identity -- that bots never received. This blog's field guide on passkeys covers the human side; Web Bot Auth is, in a real sense, the same idea arriving decades late for machines.
</Aside>

If a policy file cannot identify anyone, the obvious next step is to let the client *say* who it is. The web already had a place for that: the `User-Agent` header, present since HTTP/1.0 in 1996 [@rfc1945] and given its current normative definition in RFC 9110, section 10.1.5, in June 2022 [@rfc9110].

A crawler announces itself -- `User-Agent: MyBotCrawler/1.1` -- and the origin can, in principle, allow the crawlers it trusts and block the rest. For the first time the server has an identity claim it can act on, per request, without relying on the client to police its own behavior.

There is one catch, and it is fatal: the claim is unauthenticated. A header is just a string, and anyone can send any string.

The specification itself concedes the point. If a user agent "masquerades as a different user agent," it notes, recipients simply have to assume the client meant to be treated as whatever it claimed [@rfc9110]. Cloudflare states the operational consequence without euphemism: user-agent strings "alone are easily spoofed and are therefore insufficient for reliable identification," and agents "often use the Chrome user agent for this very reason" [@cf-blog-webbotauth].<Sidenote>The masquerade language lives in RFC 9110 section 10.1.5; the same sentence is reproduced, with that section citation, in Cloudflare's "Forget IPs" post [@cf-blog-webbotauth], which is where many practitioners first meet it.</Sidenote>

So the second generation of bot identity answers a subtly wrong question. `User-Agent` tells you *who the client says it is*, never *who it is*. And that is exactly the hook from the previous section: a legitimate agent and a scraper both send a real Chrome string, and the origin cannot separate them. A name anyone can type is not an identity. So perhaps the answer is to stop trusting what the bot writes and start trusting something it cannot simply type -- the network it comes from.

## 3. Anchoring Identity to the Network

If a bot can lie about its name, tie its identity to something it cannot simply write down: the network it lives on. A random impostor can set any `User-Agent`, but it cannot easily forge control of a vendor's DNS or occupy a specific block of IP addresses. For roughly a decade, this was the strongest bot-verification signal the web had.

Google pioneered the technique for Googlebot, and the canonical version has two composable checks. The first is forward-confirmed reverse DNS.

<Definition term="Forward-confirmed reverse DNS (FCrDNS)">
Run a reverse (PTR) lookup on the source IP to get a hostname -- for Googlebot, something like `crawl-66-249-66-1.googlebot.com` -- then forward-resolve that hostname and confirm it maps back to the same IP. Passing both directions proves the request came from network infrastructure the claimed operator controls. It proves network control, not identity.
</Definition>

Forging that requires controlling both directions of DNS for the vendor's domain, which a casual spoofer cannot do [@google-verify]. The second check is a published IP allowlist: the vendor publishes its address ranges as CIDR blocks in a file, and the origin checks whether the source IP falls inside them [@cf-blog-webbotauth]. Google documents both paths today -- common crawlers reverse-resolve under `googlebot.com` and appear in a published `common-crawlers.json`, while special-case fetchers use a separate `google.com` mask [@google-verify].

This was a real improvement. Identity was now anchored to something the requester could not self-assert; the request had to originate from Google's footprint. "Verify Googlebot" became standard operating procedure, and for years it worked.<Sidenote>The mechanism is a little older than any dated write-up, but Google's "How to verify Googlebot" post, dated 20 September 2006, is the first primary source that spells it out and calls reverse-plus-forward DNS "an official way to authenticate Googlebot" [@google-2006-blog]. That is the debut this article pins to.</Sidenote>

And then it broke -- not at the edges, but at its foundation. Reverse DNS and IP lists prove *network location*, and location is not identity. The entire premise is that a source IP maps to a single, identifiable operator, and modern infrastructure dissolves that mapping.

Cloudflare puts the failure plainly: the same IP address "might be shared by multiple users or multiple services within the same company, or even by multiple companies when hosting infrastructure is shared (like Cloudflare Workers)" [@cf-blog-webbotauth]. Privacy proxies and VPNs interpose addresses that belong to neither the operator nor any stable identity. Serverless platforms hand out egress addresses that churn with the underlying infrastructure, which is why vendors resort to constantly-updated CIDR lists [@cf-blog-webbotauth].

> **Note:** Every network-anchored check answers the question "what network did this packet come from?" That was a useful proxy for identity when an operator owned its addresses. The moment agents began running on shared clouds, privacy relays, and serverless platforms, the proxy stopped pointing at anything. The signal did not weaken -- it started measuring the wrong thing.

Picture the concrete case that matters now. An AI agent runs on a serverless platform and shares its outbound address with thousands of unrelated tenants. A reverse lookup on that address resolves to the cloud provider, not to the agent's operator. The origin can confirm "this came from a big cloud" and learns nothing about *which* automated client sent it.

Worse, the scheme is per-vendor and brittle in both directions: every operator must publish and maintain its own ranges, and every origin must track every operator's ever-shifting lists. Onboarding a new verified crawler historically meant hand-checking IP ranges and autonomous-system numbers -- exactly the manual toil the cryptographic approach was later built to retire [@cf-blog-verifiedbots].

The flaw, then, is not the mechanism. It is the premise. An IP address was never an identity; it was a stand-in for one, and stand-ins leak. To bind identity to the *operator itself* -- independent of whatever address it happens to borrow today -- you need something only that operator holds and can present on demand. You need a secret. Or, better still, a key.

## 4. Generation by Generation

Step back from the individual mechanisms and a pattern appears: each one failed in a way that named its successor. `robots.txt` had no identity, so we added a self-declared name. The name was spoofable, so we anchored to the network. The network proved location instead of identity, so we reached for a secret. Six generations, each a better answer to the same stubborn question, each breaking in a way that forced the next.

| Generation | Mechanism | What it added | The failure that forced the next |
|---|---|---|---|
| 1 (1994) | `robots.txt` honor system | A shared vocabulary for addressing crawlers | No authentication, no enforcement -- the server never learns who is asking [@robotstxt-orig] |
| 2 (1996) | `User-Agent` header | A per-request identity claim to act on | Self-declared and trivially spoofed; a real Chrome string hides everything [@rfc9110] |
| 3 (~2006) | Reverse DNS + IP allowlists | Identity anchored to network infrastructure | Proves location, not who; collapses on shared clouds, VPNs, serverless [@cf-blog-webbotauth] |
| 4 (2012) | Bearer tokens / API keys | Portable, IP-independent possession | Possession is not proof, and secrets are pairwise -- N x M distribution [@rfc6750] |
| 5 (2013-19) | Per-vendor request signing (SigV4, draft-cavage, Stripe, mTLS) | Real asymmetric cryptography plus integrity | No shared format, no interoperable key discovery -- every scheme an island [@cf-blog-webbotauth] |
| 6 (2024-26) | RFC 9421 + Web Bot Auth + JWKS directory | A standard signature and a discoverable public key | Not superseded; its limit is fundamental -- it proves who, never whether-allowed [@draft-webbotauth-arch] |

Generation 4 deserves a moment, because it is where cryptography first almost arrived. A Bearer token or API key is a secret string issued out of band; the bot presents it on every request and the origin checks it against what it handed out. Identity now travels with the credential instead of the address, so changing IPs no longer breaks it.

But RFC 6750 is unusually honest about the model's ceiling: "Any party in possession of a bearer token (a 'bearer') can use it to get access to the associated resources (without demonstrating possession of a cryptographic key)" [@rfc6750]. Possession is not proof -- anyone who intercepts or leaks the token can replay it.

And the secret is *shared* between exactly two parties, so a bot must arrange a distinct one with every origin it visits. Cloudflare names the wall directly: "You go to every website and share a secret ... This is impractical at scale because it requires developers to maintain separate tokens for each website their bot will visit" [@cf-blog-webbotauth].

<Mermaid caption="Thirty years of bot identity: each generation is a better answer to the same question, and each fails in a way that forces the next.">
flowchart LR
    G1["1994<br/>robots.txt<br/>honor system"] --> G2["1996<br/>User-Agent<br/>self-declared"]
    G2 --> G3["2006<br/>reverse DNS + IP<br/>network-anchored"]
    G3 --> G4["2012<br/>Bearer / API keys<br/>shared secret"]
    G4 --> G5["2013-2019<br/>SigV4, cavage, Stripe, mTLS<br/>fragmented crypto"]
    G5 --> G6["2024-2026<br/>RFC 9421 + Web Bot Auth<br/>standardized + discoverable"]
</Mermaid>

The fix for a leakable, pairwise secret is an asymmetric keypair: the bot signs with a private key it never discloses, and any origin verifies with the matching public key. One keypair works everywhere, interception no longer grants impersonation, and the secret never leaves the signer.

And here is the part most summaries get wrong -- **the industry built exactly this, repeatedly, more than a decade before Web Bot Auth.** AWS Signature Version 4 signed canonical requests at cloud scale [@aws-sigv4]. Stripe authenticated webhooks with an HMAC over a timestamped payload [@stripe-webhooks]. `draft-cavage-http-signatures`, first published in May 2013, signed selected HTTP headers with an asymmetric key and was adopted de-facto across the ActivityPub and Mastodon Fediverse [@draft-cavage]. Mutual TLS presented client certificates. Cryptographic request identity for machines was not missing. It was everywhere -- and useless for the open web.

<PullQuote>
"It's not the only way that developers can sign requests ... AWS has used Signature v4, and Stripe has a framework for authenticating webhooks -- but Message Signatures is a published standard." -- Cloudflare, "Forget IPs" [@cf-blog-webbotauth]
</PullQuote>

Why useless, if the cryptography was sound? Because every scheme was an island, and two gaps blocked open-web identity. First, no shared signature format: SigV4, Stripe, and draft-cavage each canonicalize and encode differently, so a stranger origin has no common way even to *parse* an unknown crawler's signature.

Second -- and this is the decisive gap -- no interoperable key discovery. The symmetric schemes hide a shared secret, which drags Generation 4's pairwise problem right back in. mTLS demands client-certificate provisioning that is famously awkward on the open web. And draft-cavage, though genuinely asymmetric, discovered keys in an application-specific way: a Mastodon server fetches the sender's actor document and reads its `publicKeyPem` field, which works only inside the Fediverse, not for an arbitrary origin meeting an arbitrary bot [@w3c-activitypub].

The successor spec states the gap in standards language: RFC 9421 "assumes verifiers have prior knowledge of signers' key material, requiring out-of-band key distribution ... This creates deployment friction" [@draft-httpsig-directory].<Sidenote>AWS's documentation describes the SigV4 protocol but never states a launch year; it is widely dated to roughly 2013-2014, and this article keeps that hedge rather than asserting a precise date [@aws-sigv4].</Sidenote>

Read the chain along two axes and the story sharpens. **Unforgeability** climbs the whole way down: from a string anyone can write, to a network position, to a leakable secret, to an unleakable private key. **Interoperability** does the opposite -- it *regresses* at Generation 5, which has the best cryptography of any generation before it and yet fails precisely because it has no common standard and no discovery.

The entire arc is a migration in key-distribution cost, from $O(N \times M)$ pairwise secrets and brittle per-vendor state toward a single $O(1)$ keypair backed by an $O(k)$ public directory. There is even a literal human through-line: Manu Sporny co-authored both the 2013 `draft-cavage` [@draft-cavage] and the 2024 RFC 9421 [@rfc9421] -- the same person carrying the same idea from pre-standard sketch to real standard.

> **Key idea:** The breakthrough was never "add cryptography to bot requests" -- machines had signed HTTP requests since 2013. The novelty is two acts of standardization: a signature construction any verifier can parse (RFC 9421), and a public directory any origin can look up (Web Bot Auth). Interoperability and discovery, not signing, are the news.

Generation 5 had the cryptography right a full decade early. So why did signing a request fail to give bots an open-web identity? Because no two islands agreed on *how* to build a signature -- and the fix starts with one deceptively simple decision about what a signature should even cover.

## 5. RFC 9421 and the Signature Base

The genius of RFC 9421 is a single, almost counterintuitive decision: do not sign the message. Sign a *description* of the message. Once you see why, the whole design of Generation 6 falls into place.

<Definition term="HTTP Message Signatures (RFC 9421)">
An IETF Standards-Track mechanism, published February 2024, for creating and verifying signatures over *selected components* of an HTTP message -- specific fields and derived values the signer chooses -- rather than over the raw byte stream of the request.
</Definition>

Here is the problem with signing raw bytes. An HTTP request rarely reaches its destination unchanged. Proxies reorder headers, add their own, re-encode bodies, and normalize whitespace. A signature over the literal bytes would shatter the moment any intermediary touched the request. So RFC 9421 inverts the approach: the signer names exactly which parts of the message it wants to commit to, and signs a canonical rendering of just those parts.

<Definition term="Derived Component">
A signable pseudo-field computed from the message itself rather than sent as a header -- `@method`, `@authority`, `@path`, `@query`, `@target-uri`, `@status`. The leading `@` marks it as derived, distinguishing it from ordinary HTTP fields like `content-type`.
</Definition>

The signer chooses its **covered components** -- any mix of ordinary fields and derived components -- and appends one trailing line, `@signature-params`, that carries the metadata: `created`, `expires`, `keyid`, an optional `alg`, an optional `nonce`, and a `tag`. All of it serializes into a canonical string.

<Definition term="Signature Base">
The canonical, newline-delimited string assembled from the covered components plus the trailing `@signature-params` line. The signature base -- not the raw request -- is precisely what the private key signs and what the verifier independently reconstructs to check.
</Definition>

A signature base for a simple request looks like this. Each covered component is one line; the last line records the recipe itself:

```
"@method": GET
"@authority": example.com
"@path": /articles/latest
"@signature-params": ("@method" "@authority" "@path");created=1618884473;expires=1618884773;keyid="poqkLGiymh_W0uP6PZFw-dvez3QJT5SolqXBCW38r0U";tag="web-bot-auth"
```

That block is the entire mechanism in miniature. The private key signs exactly those four lines. The result travels in two Structured-Field headers: `Signature-Input`, the `@signature-params` recipe so a verifier knows what to rebuild, and `Signature`, which carries the signature bytes [@rfc9421].

When the request has a body worth protecting, the signer adds a `content-digest` field -- a hash of the body defined by RFC 9530 -- to the covered set, so the body is committed to as well [@rfc9530]. The serialization rules all come from Structured Field Values, RFC 8941, which give headers a strict, unambiguous grammar of items, lists, and dictionaries [@rfc8941].<MarginNote>Without a strict grammar, two implementations could serialize the same header differently and never agree on the signature base -- which is precisely how the Generation 5 islands failed to interoperate.</MarginNote>

<Mermaid caption="RFC 9421 signs a description of the request -- the signature base -- not its raw bytes, so the signature survives proxies and re-encoding.">
flowchart TD
    R["HTTP request"] --> C["Pick covered components<br/>@method, @authority, @path, chosen fields"]
    C --> P["Append @signature-params<br/>created, expires, keyid, tag"]
    P --> B["Canonical signature base<br/>one line per covered component"]
    B --> S["Ed25519 sign the base"]
    S --> H["Emit Signature-Input (recipe) and Signature (bytes)"]
</Mermaid>

Now the payoff, and it is the aha of the whole standard. Because the signature commits to *chosen components* rather than raw bytes, it survives intermediaries, re-encodings, and proxies -- the signer decides exactly what the signature guarantees and what it deliberately leaves free to change.

This is the precise property every Generation 5 island lacked. SigV4 and draft-cavage each had their own private notion of what to canonicalize; RFC 9421 makes that notion a published standard, so any conforming verifier -- in any language, at any origin -- can reconstruct the base and check the signature without prior agreement with the signer.

<Definition term="Ed25519">
A fast, deterministic elliptic-curve signature scheme -- EdDSA using the edwards25519 curve, the twisted-Edwards curve birationally equivalent to Curve25519 -- with compact 32-byte keys. It is Web Bot Auth's deployed default; this blog's field guide on digital signatures covers the underlying curve mathematics.
</Definition>

RFC 9421 is deliberately algorithm-agnostic. [Ed25519](/blog/the-math-held-the-interface-leaked-a-field-guide-to-digital-/) is the deployed default because it is fast, deterministic, and small enough to run per request at internet scale, but the standard commits to no single scheme.

The IANA "HTTP Signature Algorithms" registry -- Specification Required, so it can grow -- also profiles RSA-PSS-SHA512, RSA-PKCS1-v1.5-SHA256, ECDSA over P-256 and P-384, and HMAC-SHA256 [@iana-httpsig-registry]. The `alg` is a property of the key, not a mandate of the protocol, which matters later when quantum-resistant schemes need a home.<Sidenote>RFC 9421 was edited by Annabelle Backman (Amazon), Justin Richer (Bespoke Engineering), and Manu Sporny (Digital Bazaar) -- the same Sporny who co-authored the 2013 `draft-cavage`, which is why the standard is best read as a revival rather than an invention [@rfc9421].</Sidenote>

One more precision, because the record is muddled. RFC 9421 is a Standards-Track document published in February 2024.<Sidenote>Several AI-generated summaries date RFC 9421 to June 2023; the RFC Editor and IETF Datatracker both settle it as February 2024 [@rfc9421]. When a source and a chatbot disagree on a standard's date, trust the RFC Editor.</Sidenote>

RFC 9421 tells you *how* to sign a request so that anyone can verify it. But "anyone can verify it" hides a trapdoor. To check a signature, a verifier needs the signer's public key -- and RFC 9421 says nothing about where to find it. It "assumes verifiers have prior knowledge of signers' key material, requiring out-of-band key distribution" [@draft-httpsig-directory]. For a paying customer's agent meeting an origin it has never contacted, that missing half is not a detail. It is the entire problem. Solving it is what turns a signature into an identity.

## 6. Web Bot Auth: From a General Tool to a Bot Identity

RFC 9421 is a general engine. Web Bot Auth is what happens when Cloudflare and Google point that engine at one specific job -- proving which bot sent a request -- and bolt on the piece the RFC deliberately left out. The result is authored by Thibault Meunier of Cloudflare and Sandor Major of Google, and it answers the trapdoor question from the last section head-on: how does a stranger find your key?

Start with the cast. Web Bot Auth defines a **three-actor model**. A *User* delegates a task. An *Agent* carries it out and signs each outbound request with its private key. An *Origin* receives the request and verifies the signature. The clean separation matters: the party that signs is the agent operator, not the human, and that distinction returns to haunt the governance discussion later.<MarginNote>Because the operator signs, one key can sit in front of thousands of different end users -- the seed of the intermediary-trust problem that resurfaces in the open problems.</MarginNote>

On top of RFC 9421's freedom, the profile pins down the parts that must be fixed for strangers to interoperate.

| Component or parameter | RFC 9421 says | Web Bot Auth requires |
|---|---|---|
| Covered components | Signer's free choice | MUST cover at least `@authority` (or `@target-uri`) [@draft-webbotauth-arch] |
| `created` / `expires` | Optional metadata | Both MUST be present; expiry RECOMMENDED ≤ 24 hours [@draft-webbotauth-arch] |
| `keyid` | Opaque signer-chosen string | MUST be the base64url JWK SHA-256 Thumbprint [@rfc7638] |
| `tag` | Optional label | MUST be `web-bot-auth` [@draft-webbotauth-arch] |
| `nonce` | Optional | A 64-byte nonce is the recommended anti-replay tool [@draft-webbotauth-arch] |
| Key discovery | Out of scope | JWKS at a well-known path, found via `Signature-Agent` [@draft-httpsig-directory] |

The last row is the whole point -- the genuine novelty of the entire effort.

<Definition term="JWKS (JSON Web Key Set)">
A JSON document listing public keys, each expressed as a JSON Web Key. The Web Bot Auth directory is a JWKS: a signer's public keys, published for anyone to fetch.
</Definition>

A signer publishes its public keys as a JWKS at the well-known path `/.well-known/http-message-signatures-directory`, and points to that directory in-band with a request header.

<Definition term="Signature-Agent header">
An in-band request header whose value is a Structured-Dictionary URI pointing at the signer's key directory. Because Web Bot Auth lists `Signature-Agent` among the covered components, an attacker cannot swap the pointer without invalidating the signature.
</Definition>

That self-protecting loop is the elegant part. The header says where the keys live; the signature covers the header; so the pointer to the keys is itself signed. And the directory response must in turn be signed -- with `tag="http-message-signatures-directory"`, one signature per listed key -- so no one can mirror your directory at their own host and enroll as you [@cf-docs-webbotauth]. The `keyid` closes the loop by naming the exact key without any external registry.

<Definition term="JWK Thumbprint (RFC 7638)">
A stable, self-certifying key identifier: the base64url-encoded SHA-256 hash over a JSON Web Key's required members in canonical form. It is what `keyid` carries, so a key effectively names itself.
</Definition>

You can watch a `keyid` derive itself. The snippet below takes the real Ed25519 public key served by Cloudflare's live debug directory, canonicalizes its required members exactly as RFC 7638 prescribes, hashes, and base64url-encodes the result -- and out falls the same `kid` the directory advertises [@cf-live-directory].

<RunnableCode lang="js" title="Deriving a keyid: the JWK Thumbprint (RFC 7638)">{`
const NL = String.fromCharCode(10);
function rotr(n, x) { return (x >>> n) | (x << (32 - n)); }
function sha256(bytes) {
  const K = [
    0x428a2f98,0x71374491,0xb5c0fbcf,0xe9b5dba5,0x3956c25b,0x59f111f1,0x923f82a4,0xab1c5ed5,
    0xd807aa98,0x12835b01,0x243185be,0x550c7dc3,0x72be5d74,0x80deb1fe,0x9bdc06a7,0xc19bf174,
    0xe49b69c1,0xefbe4786,0x0fc19dc6,0x240ca1cc,0x2de92c6f,0x4a7484aa,0x5cb0a9dc,0x76f988da,
    0x983e5152,0xa831c66d,0xb00327c8,0xbf597fc7,0xc6e00bf3,0xd5a79147,0x06ca6351,0x14292967,
    0x27b70a85,0x2e1b2138,0x4d2c6dfc,0x53380d13,0x650a7354,0x766a0abb,0x81c2c92e,0x92722c85,
    0xa2bfe8a1,0xa81a664b,0xc24b8b70,0xc76c51a3,0xd192e819,0xd6990624,0xf40e3585,0x106aa070,
    0x19a4c116,0x1e376c08,0x2748774c,0x34b0bcb5,0x391c0cb3,0x4ed8aa4a,0x5b9cca4f,0x682e6ff3,
    0x748f82ee,0x78a5636f,0x84c87814,0x8cc70208,0x90befffa,0xa4506ceb,0xbef9a3f7,0xc67178f2];
  let h = [0x6a09e667,0xbb67ae85,0x3c6ef372,0xa54ff53a,0x510e527f,0x9b05688c,0x1f83d9ab,0x5be0cd19];
  const l = bytes.length, withOne = l + 1;
  const pad = (56 - (withOne % 64) + 64) % 64, total = withOne + pad + 8;
  const m = new Uint8Array(total); m.set(bytes); m[l] = 0x80;
  const dv = new DataView(m.buffer);
  dv.setUint32(total - 4, (l * 8) >>> 0);
  dv.setUint32(total - 8, Math.floor(l * 8 / 0x100000000));
  const w = new Uint32Array(64);
  for (let i = 0; i < total; i += 64) {
    for (let t = 0; t < 16; t++) w[t] = dv.getUint32(i + t * 4);
    for (let t = 16; t < 64; t++) {
      const s0 = rotr(7,w[t-15]) ^ rotr(18,w[t-15]) ^ (w[t-15] >>> 3);
      const s1 = rotr(17,w[t-2]) ^ rotr(19,w[t-2]) ^ (w[t-2] >>> 10);
      w[t] = (w[t-16] + s0 + w[t-7] + s1) >>> 0;
    }
    let a = h[0], b = h[1], c = h[2], d = h[3], e = h[4], f = h[5], g = h[6], hh = h[7];
    for (let t = 0; t < 64; t++) {
      const S1 = rotr(6,e) ^ rotr(11,e) ^ rotr(25,e);
      const t1 = (hh + S1 + ((e & f) ^ (~e & g)) + K[t] + w[t]) >>> 0;
      const S0 = rotr(2,a) ^ rotr(13,a) ^ rotr(22,a);
      const t2 = (S0 + ((a & b) ^ (a & c) ^ (b & c))) >>> 0;
      hh=g; g=f; f=e; e=(d+t1)>>>0; d=c; c=b; b=a; a=(t1+t2)>>>0;
    }
    h = [(h[0]+a)>>>0,(h[1]+b)>>>0,(h[2]+c)>>>0,(h[3]+d)>>>0,(h[4]+e)>>>0,(h[5]+f)>>>0,(h[6]+g)>>>0,(h[7]+hh)>>>0];
  }
  const out = new Uint8Array(32), odv = new DataView(out.buffer);
  for (let i = 0; i < 8; i++) odv.setUint32(i*4, h[i]);
  return out;
}
function base64url(bytes) {
  let s = '';
  for (const b of bytes) s += String.fromCharCode(b);
  return btoa(s).split('+').join('-').split('/').join('_').split('=').join('');
}
// The Ed25519 public key served by Cloudflare's live Web Bot Auth directory:
const jwk = { crv: 'Ed25519', kty: 'OKP', x: 'JrQLj5P_89iXES9-vFgrIy29clF9CC_oPPsw3c5D0bs' };
// RFC 7638: keep only the required members, sort them, no whitespace.
const canonical = '{"crv":"' + jwk.crv + '","kty":"' + jwk.kty + '","x":"' + jwk.x + '"}';
const thumbprint = base64url(sha256(new TextEncoder().encode(canonical)));
console.log('Canonical JWK : ' + canonical);
console.log('Thumbprint    : ' + thumbprint);
console.log('Directory kid : poqkLGiymh_W0uP6PZFw-dvez3QJT5SolqXBCW38r0U');
console.log('Identical?    : ' + (thumbprint === 'poqkLGiymh_W0uP6PZFw-dvez3QJT5SolqXBCW38r0U'));
`}</RunnableCode>

With the signature standard and the directory in place, the verifier's job is mechanical: fetch the directory, match the `keyid` thumbprint, reconstruct the signature base, verify the Ed25519 signature, and enforce the validity window. Only then -- as a wholly separate step -- does it decide policy.

<Mermaid caption="Web Bot Auth's discovery-and-verify handshake: the origin reads Signature-Agent, fetches the signer's key directory, matches the keyid thumbprint, and verifies. Policy is applied as a separate step.">
sequenceDiagram
    participant U as User
    participant A as Agent
    participant O as Origin
    participant D as Key directory
    U->>A: Delegate a task, fetch this page
    A->>A: Build signature base, Ed25519 sign
    A->>O: Send request with Signature-Input, Signature, Signature-Agent
    O->>D: Fetch or read cached directory at the well-known path
    D-->>O: Return signed JWKS of public keys
    O->>O: Match keyid thumbprint, reconstruct base, verify
    O->>O: Enforce created ≤ now ≤ expires
    O-->>A: Identity established, then apply policy separately
</Mermaid>

Ed25519 is the deployed default here, but the mechanism stays algorithm-agnostic: the same registry that lets RFC 9421 carry RSA-PSS or ECDSA lets Web Bot Auth carry them too [@iana-httpsig-registry]. Deployment is real but early, and precision matters.<Sidenote>The directory draft registers the plural path `/.well-known/http-message-signatures-directory` [@draft-httpsig-directory], while some Cloudflare prose uses a singular alias; the live server answers on both, and this article uses the registered plural form [@cf-blog-verifiedbots].</Sidenote> The strongest independent signal that this is not vendor theater is OpenAI.

<PullQuote>
"With HTTP Message Signatures (RFC 9421), OpenAI signs all Operator requests so site owners can verify they genuinely originate from Operator and haven't been tampered with." -- Eugenio, Engineer, OpenAI [@cf-blog-webbotauth]
</PullQuote>

<Aside label="Deployment maturity, stated precisely">
Cloudflare moved this from a closed beta announced in May 2025 [@cf-blog-webbotauth] to integration in its Verified Bots Program in July 2025, where its edge automatically validates signatures and marks matching traffic as verified [@cf-blog-verifiedbots]. As of a July 2026 taxonomy update, Web Bot Auth is one of two accepted validation methods for becoming a Verified bot [@cf-docs-verifiedbots]. But the profile is still a set of Individual Internet-Drafts, not RFCs, and edge validation is ramping rather than generally available. This is production-adjacent, not finished.
</Aside>

> **Note:** Cloudflare's production implementation pins to specific earlier draft revisions -- roughly `directory-03` and `architecture-02` -- while the latest published drafts are at `-05`, and the architecture draft was renamed mid-flight to `draft-meunier-webbotauth-httpsig-protocol` [@cf-docs-webbotauth] [@draft-webbotauth-protocol]. Interop can break silently across revisions, so verify which draft your counterparty implements before assuming compatibility.

With a signature *and* a directory, an origin can verify a bot it has never met, on first contact, with zero prior arrangement. That is the headline. But Cloudflare shipped a *second* proposal in the very same blog, and there is a whole family of rival ways to prove a machine's identity. Why did signatures win?

## 7. Signatures, Certificates, and the Other Signers

Cloudflare proposed two mechanisms in the same post, and the one it did *not* prioritize explains why the one it did is pulling ahead. The road not taken is request mTLS.

<Definition term="Mutual TLS (mTLS)">
A handshake in which both peers present X.509 certificates, so each authenticates the other at the transport layer, beneath HTTP. The resulting identity is bound to the TLS *connection*, not to any individual request on it.
</Definition>

Classic mTLS is decades old and rock-solid inside a managed setting where both sides already run a private certificate authority. The problem is the open web. If an origin asks for a client certificate and the client has none, the browser shows "an inscrutable" and unskippable error [@cf-blog-webbotauth]. To soften that, `draft-jhoyla-req-mtls-flag` adds a TLS flag -- an experimental extension -- by which a client *signals* it holds a suitable certificate, so the server can request one only when it will not break an ordinary visitor [@draft-jhoyla-mtls].

It is a real, live parallel track. But Cloudflare is candid about ranking it second: it prioritizes Message Signatures because that approach "relies on the previously adopted RFC 9421 with several reference implementations, and works at the HTTP layer" [@cf-blog-webbotauth]. The reference-implementation point is concrete -- RFC 9421 already has an implementations hub and independent libraries to build on [@httpsig-org].

<Mermaid caption="Why an HTTP-layer signature reaches the origin when a transport-layer certificate may not: a signature rides in headers through proxies, while a client certificate binds only to the TLS connection.">
flowchart TD
    subgraph HL["HTTP layer -- Web Bot Auth"]
      A1["Signature travels inside headers"] --> A2["Survives CDNs and reverse proxies"]
    end
    subgraph TL["Transport layer -- request mTLS"]
      B1["Identity bound to the TLS connection"] --> B2["Lost when a proxy terminates TLS upstream"]
    end
</Mermaid>

The layer difference is not cosmetic. A signature lives in HTTP headers, so it rides through content delivery networks and reverse proxies untouched and can be verified at the edge or the origin. A client certificate binds to the connection, so a reverse proxy that terminates TLS -- how much of the web is commonly served -- strips the identity before the request reaches the application. For an internet built on intermediaries, HTTP-layer identity is simply the one that arrives.

Web Bot Auth also clarifies its relationship to the Generation 5 signers. It keeps draft-cavage's asymmetric idea but replaces the Fediverse-specific `publicKeyPem` lookup with a standard format and an open-web directory [@draft-cavage] [@w3c-activitypub]; and unlike SigV4 or Stripe, it needs no shared secret, only a public key the origin can fetch. Every path so far -- reverse DNS, mTLS, SigV4, Web Bot Auth -- differs less in its cryptography than in the row of the table that has no cryptography at all: how a verifier discovers the key.

| Dimension | A. Web Bot Auth | B. Request mTLS | C. Reverse DNS + IP | D. Privacy Pass |
|---|---|---|---|---|
| What it proves | Who signed, plus integrity of covered parts | Who holds the client certificate | Which network sent the packet | A property, unlinkably |
| Layer | HTTP (application) | TLS (transport) | Network / DNS | HTTP (application) |
| Cryptography | Asymmetric, Ed25519 default | Asymmetric, X.509 | None | Blind signatures / VOPRF |
| Key discovery | Public JWKS + `Signature-Agent`, in-band | PKI / CA, out-of-band | Published range files, per-vendor | Issuer directory |
| Replay protection | Validity window + optional nonce | TLS channel binding | None at the IP layer | Token single-use rules |
| Survives reverse proxies | Yes, an HTTP header | No, if TLS is terminated upstream | Only with trusted forwarding | Yes, an HTTP header |
| Privacy | Identifying by design | Identifying | Coarsely identifying | Unlinkable by design |
| Best suited for | Open-web stranger identity | Managed B2B and API | Legacy search crawlers | Privacy-preserving attestation |

Method D in that table is not really a rival at all -- it answers the opposite question.

<Aside label="Privacy Pass: the opposite trade-off">
Privacy Pass and Apple's Private Access Tokens, standardized in June 2024 as RFCs 9576, 9577, and 9578, prove a *property* about a request -- "a human or an attested device stands behind this" -- without revealing who, and without letting the origin link one request to another [@rfc9576] [@rfc9577] [@rfc9578]. That is the mirror image of a signature. Web Bot Auth maximizes accountable identity; Privacy Pass maximizes unlinkable anonymity. They sit at opposite ends of the same axis and coexist because they serve different jobs: one names the bot, the other refuses to.
</Aside>

Line the methods up by how a verifier learns the key and the pattern is stark. mTLS and SigV4 distribute keys out of band. IP schemes distribute a public list of ranges. Only Web Bot Auth carries an in-band pointer to a standardized, discoverable directory -- the one design choice, containing no new cryptography, that makes stranger-to-stranger verification possible on the open web.

And yet notice what unites every row, signatures included. Each mechanism proves exactly one kind of thing: *who*, or *what network*, or *what property*. Not one of them, however strong its cryptography, can tell you whether the bot should be admitted. That gap is not an engineering oversight to be patched in the next draft. It is a limit on what a signature *is* -- and it is time to look at it directly.

## 8. What a Signature Cannot Prove

Here is the sentence the whole article has been walking toward. A valid signature proves who signed and what they covered -- and nothing else. Not that the bot is well-behaved. Not that it respects your rate limits. Not that it is allowed anywhere near your origin.

This is not a shortcoming to be engineered away in a later draft; it is the definition of what a signature is. A digital signature is a proof of key possession over a covered message. It can establish *authenticity* -- which key signed -- and *integrity* -- that the covered components were not altered. It is information-theoretically silent on everything else.

"Allowed" is a predicate over policy and behavior, not over bytes, so no amount of cryptographic strengthening can make a signature answer it. The web already learned this once: RFC 9309 says the rules in a `robots.txt` file "are not a form of access authorization" [@rfc9309]. Web Bot Auth simply makes the *identity* half verifiable while leaving that same authorization line exactly where it was.

<Mermaid caption="The dividing line the whole article defends: cryptography establishes identity, and a separate policy gate -- not a signature -- decides allow, deny, or rate-limit.">
flowchart TD
    R["Signed request arrives"] --> V["Verify signature and validity window"]
    V --> I["Identity established: who signed, what was covered"]
    I --> G&#123;"Policy gate, not cryptography"&#125;
    G -->|trusted and well-behaved| Allow["Admit"]
    G -->|unknown or abusive| Deny["Deny or rate-limit"]
</Mermaid>

Three concrete limits make the boundary vivid, and each is fundamental rather than incidental.

First, **replay**. A signature with a validity window can be captured and replayed by anyone until it expires. The only defenses are a short window and a server-tracked `nonce` -- and the moment the server tracks a nonce, it is keeping state. This is a genuine lower bound, not a missing feature: stateless anti-replay is impossible. You trade window size against verifier memory, and Web Bot Auth's mandatory `created`/`expires` plus optional 64-byte nonce sit exactly on that trade-off. A verifier that admits a request only when $created \le now \le expires$ has narrowed the replay opportunity, never closed it [@cf-blog-webbotauth].

Second, **uncovered components**. A signature commits only to what it lists. Anything the signer left out of the covered set can be altered in flight without breaking verification. That is why the profile insists on covering `@authority` and `Signature-Agent` -- so the target host and the pointer to the key directory cannot be quietly swapped. Coverage is a choice, and everything outside it is unprotected by construction.

Third, the **trust-anchor problem**, which is the deepest of the three. A verified signature does not create trust; it *relocates* it. "Do I trust this request?" becomes "do I trust the directory behind this key?" Cryptography cannot manufacture the first bit of trust in *which signers matter* -- that decision has to come from somewhere outside the math. It is the same unanswered question that sits under the entire web public-key infrastructure: who gets to be a certificate authority? Web Bot Auth relocates that question to "whose directory do I honor?" and, by design, leaves it open [@draft-httpsig-directory].

One more frontier the standards make concrete by existing side by side: Web Bot Auth points toward accountable, linkable identity, Privacy Pass toward unlinkable anonymity [@rfc9576]. You can pick a point on that axis, but not both endpoints in one token -- a design decision, not a bug awaiting a fix.

<PullQuote>
A valid signature proves who signed and what was covered -- never whether the bot is allowed. Signing makes a bad bot accountable, not good.
</PullQuote>

It is worth naming what these limits are *not*. This domain has no impossibility theorems in the Turing sense, no proven complexity lower bounds, no hardness results. The honest "bounds" here are conceptual: they describe what a signature can attest, not how expensive it is to compute. Pretending otherwise -- dressing a category boundary up as a big-O result -- would be a fabrication. The real limit is cleaner and more absolute than any asymptotic one.

> **Key idea:** Identity and authorization are not two ends of one dial. They are different categories. A signature is silent on "allowed" the way a photograph is silent on "true" -- not because it is a weak signature, but because "allowed" is not the kind of thing a signature can encode. This is a boundary, not a backlog item.

So where does "allowed" get decided? Not here.

<Aside label="Where identity hands off to authorization">
Deciding whether an identified agent may perform an action is the job of an authorization system -- the concern of standards like [AuthZEN](/blog/who-authorized-this-tool-call-openid-authzen-the-mcp-profile/), covered in this blog's agent-authorization post. The clean handoff is the point: cryptography answers "who," and policy answers "allowed," and keeping them separate is what lets each be reasoned about honestly. A further layer sits beyond "allowed" -- proving *what* a user authorized an agent to do, such as a payment -- the domain of this blog's companion post on agentic payment intent ([FIDO AP2 / Verifiable Intent](/blog/the-mandate-that-replaced-the-checkout-button-how-fido-ap2-a/)). A signed request is the beginning of an access decision, never the whole of it.
</Aside>

If a signature only proves *who*, then "who do we trust, and who decides?" becomes the entire game. And nobody has fully answered it. That unfinished question opens onto the frontier of governance.

## 9. Governance, Revocation, and the Legitimacy Gap

Verifiable identity solves the easy half of the problem. The hard half -- who *deserves* trust, and how you take it back -- is wide open, and almost none of it is a cryptography question. Here are the eight problems that will decide whether Web Bot Auth becomes the default or a footnote.

1. **Signer accreditation.** A signature proves who signed, never whether that signer deserves trust. Someone still has to decide which directories an origin honors. Cloudflare's Verified Bots Program, backed by its BotBase and Radar data, is a de-facto answer -- but a single-vendor one [@cf-docs-verifiedbots]. The open problem is a neutral, multi-party accreditation model, without which every origin rebuilds the per-vendor allowlist that this whole effort set out to retire.

2. **Revocation and rotation at web scale.** A leaked key needs to be pulled fast and globally. Short `expires` windows bound the exposure, and a signer can drop a key from its JWKS, but there is no low-latency revocation or transparency mechanism -- nothing like the certificate world's revocation lists or transparency logs -- for these directories yet [@draft-httpsig-directory].

3. **Directory availability and denial of service.** Verification depends on fetching a third party's well-known directory. When that directory is slow, down, or flooded, the request hot path inherits a new availability and amplification surface. Signing the directory response makes cached copies tamper-evident, but cache-lifetime and denial-of-service guidance is still thin [@cf-docs-webbotauth].

4. **The legitimacy gap.** A signed bot can still scrape aggressively, ignore `robots.txt`, or blow past rate limits. Signing makes it accountable, not good. Cloudflare's response is to bundle identity with a *behavioral* bar -- a Verified bot must both self-identify honestly and be non-abusive, obeying `robots.txt` at reasonable rates [@cf-docs-verifiedbots]. That is policy bolted on top of identity, exactly as the identity-versus-authorization spine predicts.

5. **Transitive trust for [intermediary agents](/blog/agentic-identity-on-windows-when-the-process-acting-on-your-/).** An "agent mode" browser is driven by many different end users, so trusting the operator's key is not the same as trusting every person behind it. The party that signs is not the party ultimately responsible for a given action. Cloudflare's July 2026 taxonomy adds a Direct-versus-Intermediary label and is experimenting with forwarding end-user information in the standard `Forwarded` header, so an origin can apply policy to the responsible party [@cf-docs-verifiedbots].

6. **Adoption bootstrapping.** Signers gain little until origins verify, and origins gain little until signers sign -- the classic two-sided chicken-and-egg. The escape is asymmetric scale: a large reverse proxy verifying by default seeds millions of origins at once [@cf-blog-verifiedbots], while a marquee signer like OpenAI's Operator seeds the client side [@cf-blog-webbotauth].

7. **Standardization status and version skew.** The profile is a set of Individual Internet-Drafts, not RFCs, and it is a moving target: the architecture draft was renamed mid-flight to `draft-meunier-webbotauth-httpsig-protocol` [@draft-webbotauth-protocol], while production pins to earlier revisions than the latest. Interop can break silently between draft versions until a working group freezes the text.

8. **Post-quantum migration.** Ed25519, RSA, and ECDSA are all breakable by a sufficiently large quantum computer. A long-lived signing infrastructure should become crypto-agile before it must. RFC 9421's algorithm registry is extensible by design, and post-quantum signature schemes are being explored for it, but none is the deployed default today [@iana-httpsig-registry].

> **Note:** Notice that not one item on this list is fixed by a better signature. Accreditation, revocation, availability, legitimacy, delegation, adoption, standardization, and quantum-readiness are questions of governance and process. Cryptography set the identity floor; everything above it is human coordination -- and coordination moves slower than code.<Sidenote>The datatracker landing pages for these drafts are version-agnostic and stable even as the documents are renamed, which is why this article cites them rather than any single versioned snapshot [@draft-webbotauth-protocol].</Sidenote>

None of these are reasons to wait. The identity layer they sit on is already real, already standardized at its base, and already signing live traffic from a marquee agent. Which leaves one intensely practical question, the one an engineer actually needs answered: how do you *do* this, on the wire?

## 10. Doing It on the Wire

Enough theory. Here is the whole protocol as an engineer meets it, in two roles.

If you are building a bot, there are four steps. Generate an Ed25519 keypair. Publish your public keys as a JWKS at `/.well-known/http-message-signatures-directory`, and sign that directory response itself with `tag="http-message-signatures-directory"`, one signature per key [@cf-docs-webbotauth]. On each outbound request, build a signature base that covers at least `@authority` and `signature-agent`, set a short `created`/`expires` window, set `keyid` to your key's JWK Thumbprint [@rfc7638], and set `tag="web-bot-auth"`. Then emit three headers -- `Signature-Agent`, `Signature-Input`, and `Signature`. On the wire, using the current directory draft's Structured-Dictionary URI for `Signature-Agent` [@draft-httpsig-directory], that is:

```
Signature-Agent: sig="https://signer.example.com"
Signature-Input: sig=("@authority" "signature-agent";key="sig");created=1700000000;expires=1700011111;keyid="ba3e64==";tag="web-bot-auth"
Signature: sig=abc==
```

Three lines carry the entire identity claim [@cf-blog-webbotauth]. `Signature-Agent` says where the keys live, `Signature-Input` is the recipe, and `Signature` is the proof.

If you are the origin, the flow reverses. Parse `Signature-Agent` to learn where the directory is. Fetch it -- or reuse a cached copy -- and match the request's `keyid` to a published key. Reconstruct the signature base byte-for-byte from the same covered components, verify the Ed25519 signature against the matched key, and enforce that the request falls inside its `created`/`expires` window. Only after all of that succeeds do you reach the separate, non-cryptographic step: apply policy.

The one part worth running yourself is the signature base, because byte-identical reconstruction is where implementations most often fail. The snippet below assembles exactly the string a signer signs, then runs the validity-window check the verifier performs.

<RunnableCode lang="js" title="Building an RFC 9421 signature base">{`
const NL = String.fromCharCode(10);
// Build the RFC 9421 signature base: the exact string a signer signs
// and a verifier must reconstruct byte-for-byte.
function componentId(c) {
  return '"' + c.name + '"' + (c.key ? ';key="' + c.key + '"' : '');
}
function buildSignatureBase(covered, values, params) {
  const lines = [];
  for (const c of covered) {
    lines.push(componentId(c) + ': ' + values[c.name]);
  }
  const innerList = '(' + covered.map(componentId).join(' ') + ')';
  const sigParams = innerList
    + ';created=' + params.created
    + ';expires=' + params.expires
    + ';keyid="' + params.keyid + '"'
    + ';tag="' + params.tag + '"';
  lines.push('"@signature-params": ' + sigParams);
  return lines.join(NL);
}
// Web Bot Auth MUST cover @authority; draft-05 covers the signature-agent ;key="sig" member.
const covered = [
  { name: '@authority' },
  { name: 'signature-agent', key: 'sig' }
];
const values = { '@authority': 'example.com', 'signature-agent': '"https://signer.example.com"' };
const params = {
  created: 1700000000,
  expires: 1700000300,
  keyid: 'poqkLGiymh_W0uP6PZFw-dvez3QJT5SolqXBCW38r0U',
  tag: 'web-bot-auth'
};
const base = buildSignatureBase(covered, values, params);
console.log('--- signature base (input to Ed25519) ---');
console.log(base);
// The origin enforces the validity window BEFORE trusting the signature.
const now = 1700000300;
const withinWindow = params.created <= now && now <= params.expires;
console.log('');
console.log('created <= now <= expires : ' + (withinWindow ? 'VALID' : 'REJECT'));
console.log('the real signing is then one call: ed25519.sign(privateKey, base)');
`}</RunnableCode>

That is genuinely the whole computation. The actual signing is a single library call on the string the snippet builds, and verification is the same string plus one `verify` call.

> **Note:** Sign the directory response itself, or an attacker can mirror your directory and enroll as you [@cf-docs-webbotauth]. Reconstruct the signature base byte-for-byte -- the Structured Field grammar of RFC 8941 is strict, and a stray space or a reordered field breaks verification [@rfc8941]. Always cover `@authority` and `Signature-Agent` so the host and the key-directory pointer cannot be swapped. Watch clock skew against short windows -- a five-minute expiry is safer against replay but less forgiving of unsynchronized clocks. Pin the draft revision your counterparty implements. And deploy Ed25519: the standard is algorithm-agile, but Ed25519 is what interoperates today [@draft-webbotauth-arch].

You do not have to build this from scratch. The reference implementations are first-party: an npm `web-bot-auth` package [@npm-webbotauth], a Rust crate, a Cloudflare Workers verifier, and a Caddy plugin, all in the public repositories [@gh-cloudflareresearch] [@gh-cloudflare-webbotauth], plus the broader RFC 9421 implementations hub [@httpsig-org]. Best of all, there is a live directory you can hit right now.<Sidenote>Cloudflare runs a live debug directory at `http-message-signatures-example.research.cloudflare.com`; its well-known path serves a real Ed25519 JWK whose `kid` is `poqkLGiymh_W0uP6PZFw-dvez3QJT5SolqXBCW38r0U` -- the very key the earlier thumbprint snippet reproduces [@cf-live-directory].</Sidenote>

Fetch that directory, feed its key into the verifier, and you have reproduced the entire mechanism end to end. The discovery layer that RFC 9421 left out is the part you can now watch working, live [@draft-httpsig-directory].

<Spoiler kind="hint" label="Watch it work in one command">
Run `curl` against the debug host's well-known path -- `http-message-signatures-example.research.cloudflare.com/.well-known/http-message-signatures-directory` -- and the response is a signed JWKS containing a single Ed25519 key. Its `kid` is the exact thumbprint the earlier snippet computes [@cf-live-directory].
</Spoiler>

Stand back and the mechanics are almost anticlimactic: a keypair, a JSON file at a well-known URL, and two request headers. The hard part was never the code. It was getting the entire web to agree on the *same* keypair format, the *same* JSON document, the *same* two headers -- so that an origin and a bot that have never met can still understand each other. That agreement is the thirty-year story you just read.

## 11. Frequently Asked Questions

<FAQ title="Frequently asked questions">
<FAQItem question="Isn't this just mTLS?">
No. Mutual TLS authenticates at the transport layer with a per-connection X.509 certificate, and it struggles on the open web: a client without a certificate gets an inscrutable browser error, provisioning and rotation are heavy, and the identity does not survive a reverse proxy that terminates TLS. Web Bot Auth signs at the HTTP layer, per request, so it rides through content delivery networks and can be verified at the edge [@cf-blog-webbotauth].
</FAQItem>
<FAQItem question="Does a valid signature mean the bot is allowed?">
No. A signature proves who signed and what was covered. "Allowed" is a separate policy decision -- identity is not authorization. The web drew this exact line for `robots.txt`, whose rules "are not a form of access authorization" [@rfc9309]. A signature makes the sender *nameable*, not *trustworthy* -- it hands you someone to hold responsible, not a reason to say yes.
</FAQItem>
<FAQItem question="Is Ed25519 required?">
No. RFC 9421 is algorithm-agnostic, and its IANA registry also profiles RSA-PSS, ECDSA over P-256 and P-384, and HMAC [@iana-httpsig-registry]. Ed25519 is the deployed default for interoperability today, not a mandate.
</FAQItem>
<FAQItem question="Is this the first cryptographic identity for machines?">
No, and this is the point most coverage gets wrong. AWS Signature v4, `draft-cavage` HTTP Signatures across the Fediverse, Stripe webhook signatures, and mTLS all predate it [@cf-blog-webbotauth] [@draft-cavage]. The novelty is a standardized, discoverable, open-web identity -- not signing itself.
</FAQItem>
<FAQItem question="Does this replace robots.txt?">
No. `robots.txt` says *what* an origin permits; Web Bot Auth establishes *who* is asking. Identity makes etiquette enforceable against a named party for the first time, so the two are complementary rather than competing [@rfc9309].
</FAQItem>
<FAQItem question="Is it production-ready?">
Partly. RFC 9421 is a published standard, but the Web Bot Auth profile is a set of Individual Internet-Drafts, not RFCs, and Cloudflare's edge validation is ramping from beta into its Verified Bots Program rather than being generally available [@cf-blog-verifiedbots]. OpenAI's Operator, however, signs in production today [@cf-blog-webbotauth].
</FAQItem>
<FAQItem question="What stops someone replaying a signed request?">
A short `created`/`expires` window plus an optional server-tracked 64-byte nonce [@draft-webbotauth-arch]. But stateless anti-replay is impossible: you trade a smaller window against the verifier having to remember nonces. The window shrinks the opportunity, it never eliminates it.
</FAQItem>
</FAQ>

### After Thirty Years, an Identity

Return to the two requests from the opening -- the agent and the scraper, identical down to the byte. The web spent its first three decades unable to separate them, not for lack of trying but because every tool it reached for answered the wrong question. `robots.txt` asked bots to behave. `User-Agent` let them name themselves. Reverse DNS pinned them to a network. Bearer tokens and per-vendor signatures added real cryptography but locked it inside islands no stranger could reach. Each was a better answer to "which bot is this?" and each broke in a way that named its successor.

RFC 9421 and Web Bot Auth close the loop with two acts of standardization the previous generations never managed: a signature construction any verifier can parse, and a public directory any origin can look up. That is the honest shape of the claim. Not the first cryptography for machines -- the first cryptographic identity for machines that is standardized, discoverable, and open to the whole web. A bot generates one keypair, publishes one signed JSON document at one well-known URL, and any origin on earth can verify it on first contact, with no prior arrangement.

<PullQuote>
After decades of hardening how it authenticates humans, the web has finally given agents a verifiable cryptographic identity -- finally standardized and discoverable, not first-ever. And it proves exactly one thing: who signed, never whether they are allowed.
</PullQuote>

That last clause is the discipline the whole article kept. A signature makes a bot accountable, not good. It relocates trust, it does not manufacture it. Everything above the identity floor -- who deserves trust, how you revoke it, what "allowed" even means for an agent acting for a thousand different users -- is governance, and governance is unfinished. The door now knows who is knocking. Deciding whether to open it is a different, still-unwritten story, and it begins exactly where this one ends.

<StudyGuide slug="signed-agents-web-bot-auth" keyTerms={[
  { term: "Web Bot Auth", definition: "Cloudflare and Google IETF drafts that profile RFC 9421 for bot identity and add a discoverable public key directory." },
  { term: "Signature Base", definition: "The canonical string built from covered components plus the @signature-params line; the exact input a private key signs." },
  { term: "Derived Component", definition: "A signable pseudo-field computed from the message, such as @method, @authority, or @path." },
  { term: "JWK Thumbprint (RFC 7638)", definition: "A base64url SHA-256 hash over a JWK's canonical members; what Web Bot Auth's keyid carries." },
  { term: "Signature-Agent header", definition: "An in-band, covered request header pointing to the signer's key directory." },
  { term: "Forward-confirmed reverse DNS", definition: "Reverse-then-forward DNS resolution that proves network control, not identity." },
  { term: "Identity versus authorization", definition: "A signature proves who signed and what was covered, never whether the sender is allowed." }
]} />