Reverse Engineering "Verified Bot": The Six-Layer Identity Chain That Decides Whether Googlebot Sees Your Site (and the September 15, 2026 Default That Quietly Rewired It)

At 09:00 UTC on September 15, 2026, Cloudflare changed the meaning of a checkbox on a network that carries more than a fifth of the web's domains.

In July 2025, "Block AI bots" was a single switch. It meant what it said: stop the crawlers that hoover up your content to train models. A lot of site owners turned it on, felt briefly virtuous, and moved on. Some of them ran display ads. Some of them had spent a decade earning organic rankings on the pages where those ads lived.

Then Cloudflare published its second Content Independence Day announcement on July 1, 2026, and buried the operative sentence in the middle of a post about a bot taxonomy: "Since the defaults will be enforced by the most restrictive applicable rules, multi-purpose crawlers such as Googlebot, Applebot, and BingBot will be blocked by customers who have selected to block Training."

Read that again with the 2025 checkbox in mind. You selected to block Training. Googlebot crawls for Search and for Training. The most restrictive applicable rule wins. Therefore, your ad pages stop serving Googlebot.

Nobody chose that. There was no dialog that said, "this will also block the search engine that pays for your mortgage." There was a taxonomy decision, made by a third party, that a bot has behaviors rather than a purpose, and a rule that a bot carrying several behaviors is governed by the strictest one. A policy you expressed in one vocabulary in 2025 was silently translated into a different action in 2026.

That is the surface story, and by now you have probably read a version of it. This article is about the layer underneath, because the surface story cannot be debugged without it.

Every "block AI crawlers" guide on the internet gives you the same artifact: a list of user-agent strings. GPTBot, ClaudeBot, PerplexityBot, Google-Extended. Paste them into a rule. Done.

A list of strings is not an identity system. It is a name tag. Anyone can write any name on a name tag, and — this is the part that took the industry two years to admit in public — some of the largest companies on the internet have been credibly accused of writing someone else's.

Three mechanisms the industry keeps conflating

Before the teardown, one distinction has to be made, because almost every confused conversation about AI crawlers in 2026 comes from collapsing it. There are three completely different things in play, and they have three completely different trust properties.

The September 15 change was a collision between all three. Cloudflare's category assignment is a claim about a crawler's behavior, stored in Cloudflare's own database. The enforcement is a proof system built on IP ranges and rDNS. The thing site owners believed they were expressing was a preference. When the taxonomy reclassified Googlebot as Search + Training, a preference about training was re-read as a proof-tier decision about search, and a block fired that the site owner never asked for.

That is not a bug in Cloudflare's product. It is what happens when an industry tries to govern a new economic relationship using 1994 primitives and 2026 vocabulary.

Teardown: what is actually in the request

Here is a trimmed header block from a real crawler hit on an ordinary site behind a CDN. I have removed the identifiers. Everything else is what your origin receives.

GET /blog/some-post HTTP/2
Host: example.com
User-Agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible;
    Googlebot/2.1; +http://www.google.com/bot.html) Chrome/124.0.0.0 Safari/537.36
Accept: text/html,application/xhtml+xml
Accept-Encoding: gzip, br
From: googlebot(at)googlebot.com
CF-Connecting-IP: 66.249.66.1
CF-IPCountry: US
CF-Ray: 8f2c1a4b9e6d7f01-IAD
X-Forwarded-Proto: https
Signature-Agent: sig1="https://example-operator.example"
Signature-Input: sig1=("@authority" "signature-agent";key="sig1");created=1761177600;
    keyid="NFcWBst6DXG-N35nHdzMrioWntdzNZghQSkjHNMMSjw";alg="ed25519";
    expires=1761181200;tag="web-bot-auth"
Signature: sig1=:MEUCIQD...:

Six things in that block can be used to identify the sender. They are not equally trustworthy, and they are not even measuring the same property. Tear them apart in order.

Layer 0 — The socket: IP address, ASN, and the TLS handshake

The IP tells you which network the packet left. Not which company, not which product, not which human. On a modern cloud, the answer to "who owns this IP" has layers: the cloud platform owns the block and publishes it, the tenant rents a fraction of it, and the crawler operator rents the tenant. The IETF draft that defines this whole architecture says so plainly in its motivation section, and it is one of the more honest paragraphs in the document:

"IPs on cloud platforms have layers of ownership — the platform owns the IP and registers it in their published IP blocks, only to be re-published by the agent with little to bind the publication to the actual service provider that may be renting infra."

What layer 0 proves: the request came from a network. That is all. It is still the most useful layer for detecting residential proxy pools and datacenter scraping, because those have shapes — a single ASN hammering 4,000 paths with no cookies and no CSS — but shape is a statistical judgment, not an identity.

Layer 1 — Reverse DNS: the first layer with real friction

Forward-confirmed reverse DNS is the oldest working bot verification on the web, and most engineers implement it wrong.

The naive version asks one question: does the PTR record for this IP contain googlebot.com? That check is worthless, because PTR records are controlled by whoever owns the IP block and can say anything. An attacker who owns a block can point a PTR record at crawl-1-2-3-4.googlebot.com and pass the substring test.

The correct version asks three questions and requires all three to agree:

1. Resolve the PTR for the connecting IP. Expect something like crawl-66-249-66-1.googlebot.com.
2. Forward-resolve that hostname. Expect an A or AAAA record.
3. Check that the forward result contains the original IP.

The cost of faking that loop is not a header. It is control of the reverse DNS zone for an IP block, which means owning or leasing the block and having the provider delegate the zone to you. That is genuine friction, and it is why rDNS survived twenty years of spoofing attempts.

What layer 1 proves: the operator of this IP block has configured DNS to claim this identity, and the DNS system agrees in both directions. What it does not prove: which of the operator's crawlers is calling. Google runs several crawlers across the same address space. A verified rDNS lookup tells you "this is Google infrastructure," not "this is Googlebot and not Google-Extended."

Layer 2 — Published prefix lists: the layer that quietly broke this year

Every large crawler operator publishes the IP ranges it crawls from, so firewalls can allowlist without DNS lookups. OpenAI's list at openai.com/gptbot.json is live and machine-readable; when I pulled it while writing this, its creationTime field read 2026-09-22T02:00:07 — the operator had refreshed it within the last month.

Google's list is a different story, and this is the part I have not seen documented anywhere.

The URL that thousands of firewall scripts, nginx configs, Lambda functions and SEO tools have hardcoded — https://developers.google.com/static/search/apis/ipranges/googlebot.json — now returns a 301 to https://developers.google.com/static/crawling/ipranges/common-crawlers.json. The redirect target is a valid 21 KB JSON document, and when I parsed it, its entire top-level structure was two keys:

{ "creationTime": "...", "prefixes": [ ... 317 entries ... ] }

No googlebot key. No google-special-crawlers. No per-crawler attribution of any kind. Google has consolidated its crawler ranges into one undifferentiated set of 317 prefixes, and the endpoint name — common-crawlers — is the only hint that the granularity is gone.

Two consequences follow, and both are operational rather than theoretical:

• Any pipeline that fetched googlebot.json and parsed a single prefix array may still work, if it followed redirects and did not depend on the old schema. If it ran with redirect-following disabled — a reasonable security default — it now receives a 301 and no data, and its allowlist silently stops updating. The failure is invisible: the last-known-good list keeps being enforced, and it keeps drifting out of date.
• IP membership can no longer tell you which Google crawler called. That matters enormously in 2026, because Googlebot is precisely the crawler that Cloudflare's taxonomy classifies as Search + Training. If your policy is "allow search crawling, refuse training crawling," the prefix list cannot express it. The ranges are shared. You have to fall back to rDNS or user-agent, and user-agent is layer 3.

Layer 3 — User-Agent: a name tag, not a credential

This is the layer everyone builds on, and it is the layer that proves nothing.

The definitive public demonstration arrived on August 4, 2025, when Cloudflare published an analysis of Perplexity's crawling behavior. The finding was not that a small scraper lied. It was that a well-funded AI search company was, according to Cloudflare's measurements, repeatedly changing user agents and rotating across IPs and ASNs — including presenting as an ordinary Chrome browser — in order to keep crawling sites that had explicitly blocked its declared bot. Cloudflare's framing was precise: stealth, undeclared crawlers, in direct conflict with explicit no-crawl directives.

You do not need to accept every number in that report to accept the structural conclusion, because the structure is not controversial. A user-agent string is chosen by the client. It costs one header to write Googlebot. And the honest crawler and the dishonest crawler emit the identical bytes, so no amount of string matching can separate them.

There is a second, subtler problem with user-agent rules that nobody mentions: they are overloaded by design. The IETF draft is explicit that an agent running Chromium may need to present as Chromium so that pages render correctly, while still wanting the site to know it is an agent. One string is being asked to carry both "render me like a browser" and "identify me as a bot." Those are contradictory requirements, and the industry resolved them by lying to itself in the first field of every request.

Layer 4 — Signed requests: the only layer that binds a message to a key

Web Bot Auth is the 2025–2026 attempt to fix this properly. It is an IETF work item (the web-bot-auth working group), built on RFC 9421 — HTTP Message Signatures, published February 2024 — and the architecture draft I read is revision -05, dated 2 March 2026, authored by Thibault Meunier of Cloudflare and Sami Major of Google. It is an Informational draft that expired on 3 September 2026, which is worth repeating to anyone planning to build on it: this is a moving target, not a standard.

The mechanics are simple enough to state in one paragraph. The agent generates a key pair, publishes the public half at a well-known path on a domain it controls, and signs each outbound request. The signature covers the request authority and the signature-agent header, carries a keyid that is the base64url SHA-256 thumbprint of the JWK, a created timestamp, an expires timestamp that should be no more than 24 hours out, and a tag that must equal web-bot-auth. The origin parses the headers, fetches the key directory if it does not already know the keyid, and verifies. Shared HMAC secrets are explicitly forbidden, because they break non-repudiation.

Here is the same header block from layer 0, annotated:

Signature-Agent: sig1="https://operator.example"
  # a self-nominated pointer to the key directory FQDN.
  # the verifier MUST decide whether it trusts this domain. the spec
  # does not tell it how. that is the whole ballgame.

Signature-Input: sig1=("@authority" "signature-agent";key="sig1");
    created=1761177600;
    keyid="NFcWBst6DXG-N35nHdzMrioWntdzNZghQSkjHNMMSjw";
    alg="ed25519";
    expires=1761181200;
    tag="web-bot-auth"
  # keyid is a JWK thumbprint, not a company name.
  # "covered components" is the list before the semicolon: only
  # @authority and signature-agent. NOT the path. remember this.

Signature: sig1=:MEUCIQD...:
  # base64 signature over the RFC 9421 signature base.

Note what the covered components list does not contain: the target path, the method, the query string, or the body. The draft requires at least one of @authority or @target-uri, so a conforming agent may sign only the host. We will come back to that in the failure modes, because it is a real replay primitive and almost nobody checks for it.

The Verification Ladder

Here is the framework I use when auditing a site's bot rules. Six tiers, ordered by the cost of forging the evidence. The right-hand column is the one that matters in practice and the one no vendor publishes, because it is the column that tells you what your mistake will cost.

Two things fall out of that table that are worth stating plainly.

First, T4 is a trap. Signature verification feels like the finish line. It is not. A signature with no trust policy proves only that the sender controls a key that the sender published. An attacker can generate an Ed25519 keypair, publish it at their own well-known path, sign a request for your site, and pass every cryptographic check you wrote. The draft knows this — it lists "public list" as an unresolved option for key distribution and immediately notes the governance problem: who gates the list, and who decides who gets on it. Until you answer that question for yourself, T4 is a formality with better marketing than T1.

Second, the cost asymmetry is inverted from how people reason about it. Engineers instinctively optimize for "don't let the bad thing in." For crawler identity, the expensive error is usually the other one: blocking the crawler that pays you. A scraper that gets through costs you content that is already public. A false positive against Googlebot costs you the pages that generate revenue, and it costs you weeks before you notice, because the crawl rate decays before the ranking does. Any policy that does not weight those two errors differently is not a policy. It is a reflex.

Seven ways the signed layer fails in production

Web Bot Auth is a good design. It is also an unfinished one, and the gap between "the draft describes an architecture" and "you shipped a verifier" is where incidents live. These are the seven failure modes I would check in any implementation review, in order of how badly they bite.

1. The verifier becomes an SSRF proxy

The draft says that during validation, "if the keyid is unknown to the origin, they MAY fetch the provider directory as indicated by Signature-Agent header." Read that as an implementer: the request supplies a URL, and your server fetches it.

Consider what an unauthenticated request can now do:

Signature-Agent: sig1="http://169.254.169.254/latest/meta-data/iam/security-credentials/"
Signature-Agent: sig1="http://127.0.0.1:8080/admin"
Signature-Agent: sig1="http://internal-service.prod.svc.cluster.local/"

If your verifier follows the header without a policy, the attacker has turned your edge into a request forwarder with your network position. The draft anticipates signature abuse but not this; it is a consequence of the discovery mechanism, not of the cryptography.

Mitigations that work, in order of importance: allowlist the directory domains you will ever fetch (which you needed for T5 anyway, so this costs nothing extra); refuse non-HTTPS; refuse private, loopback, link-local and metadata address ranges after DNS resolution, not before; disable redirects on the fetch; cap response size; timeout aggressively; cache negative results; and rate-limit directory fetches per source ASN. If you cannot do all of that, do not fetch directories at request time at all — pre-fetch keys for the operators you care about, on a schedule, and reject unknown keyids.

2. Trust-anchor confusion

The signature names a Signature-Agent domain. The verifier must decide whether that domain is allowed to speak for the operator it claims. If the answer is "any domain with a valid key directory," then the identity system is a mirror: it tells you what the sender wants to be called.

The failure looks like this in practice. A crawler presents Signature-Agent: sig1="https://openai-crawler.example", publishes a perfectly valid Ed25519 key at the well-known path, signs correctly, and your verifier returns "verified." Nothing about that chain involved OpenAI. The string similarity did the work, in a human's head, at 2 a.m., during an incident.

The only fix is a hardcoded allowlist mapping directory domain to operator name, reviewed like a firewall rule, with an expiry date on every entry. Treat that allowlist as production configuration with an owner, not as a constant in a source file.

3. Path-unbound replay

Look again at the covered components in the header example: ("@authority" "signature-agent"). The path is not in there. That is conforming — the draft requires at least one of @authority or @target-uri, and the agent chose the cheaper one.

The consequence: a signature captured on /public/sample-report is cryptographically valid for /premium/report on the same host, because the signature base never included the path. An attacker who can observe one signed request — from their own machine, if they run an agent, or from a proxy log — can replay it against any path on your origin for the life of the signature.

You cannot fix this by wishing. You can refuse signatures that do not cover @target-uri (or explicitly include :path as a covered component), and you can require a nonce. The draft supports both; neither is mandatory.

4. The 24-hour window and the missing nonce store

Expiry is a recommendation, not a bound: "It is RECOMMENDED the expiry to be no more than 24 hours." Nonce is optional, and even when present, enforcement is explicitly the origin's problem: the client must make it globally unique, but "it is the origin's responsibility to enforce it."

So the default implementation — check the clock, skip the nonce — accepts a replayable credential with a one-day lifetime. Nothing in the crypto stops it. Add a nonce store or accept that your verifier is a bearer-token validator with extra steps.

5. Intermediaries can relabel the identity pointer

This one is subtle and comes straight out of the draft's security considerations. An intermediary is permitted to relabel an existing signature when processing the message, and the draft gives an explicit example of an intermediary updating Signature-Agent from sig2 to sig3, with the signature input updated to match.

In other words: the entity that tells you who signed can be rewritten in transit by a component you trust. That is sometimes exactly what you want — a gateway attesting on behalf of many operators, which is the "transitive trust" model Cloudflare proposed with the Forwarded header. It is also an assumption you must be conscious of, because your verifier cannot distinguish "the operator said this" from "the gateway said the operator said this" unless you sign and log the whole chain.

6. Key rotation with no revocation list

keyid is a JWK thumbprint. There is no CRL, no OCSP, no published revocation feed in the architecture. If an operator's signing key leaks, the operator is told to stop using it, notify affected origins "if possible," and rotate. Your cache does not know. Your allowlist does not know. Until the old key's exp passes, a compromised key remains a valid identity — and if you cached the directory aggressively, possibly longer.

Practical mitigation: short cache TTLs on key material, alert on keyid changes for operators you trust, and treat a sudden key change as an event worth a human look rather than a silent success.

7. Verification cost lands on the hot path

Per-request signature verification plus a directory fetch is not free. The draft acknowledges this and suggests a session: after a successful verification, issue a cookie and stop verifying every request. Which reintroduces exactly the property signatures were meant to remove — a bearer credential that can be stolen and replayed for the session lifetime. That trade is reasonable, but it is a trade, and the draft says so: "This trades cryptographic verification costs for the security properties of bearer tokens, including susceptibility to credential theft and replay."

If you cache verification results, cache them on keyid plus a normalized signature base, not on the raw header, and keep the TTL shorter than the signature's own expiry.

A short history of the identity layer, 2023 to 2026

None of this arrived in one piece. The pattern is worth seeing as a sequence, because it explains why the controls you inherited are incoherent: each layer was added to solve the previous layer's failure, without retiring the older assumption.

The eight-state access matrix

The industry argues about bot blocking as a binary. It is not a binary. There are three independent behaviors a modern crawler can perform against your site, and each of the eight combinations buys you something different and costs you something different.

Search means the crawler is building or refreshing an index that will answer questions about your content. Training means your content is being absorbed into model weights. Agent means a live request on behalf of a human who is waiting for an answer right now.

Three observations that the table makes obvious but the discourse does not.

Row 3 is the industry default, and it is the most mispriced configuration on the web. Allow Search, block Train, allow Agent: you keep today's referrals and today's agent citations, and you accept permanent absence from model weights. That is a defensible trade for a publisher with a twelve-quarter horizon. It is a bad trade if your assumption is that answer engines will increasingly answer from memory rather than from retrieval, because you are optimizing the channel that shrinks and divesting the one that compounds.

Row 4 is not a policy. It is a bug with a UI. Allow Search, block Train, block Agent is the configuration a rational ad-supported publisher would choose, and for a crawler that also trains, it is unrepresentable. On a unified crawler, "block training" is read as "block," which is precisely what happened on September 15. You cannot reach row 4 through a training-block toggle. You reach it by using a training-specific token such as Google-Extended, or by not blocking at all.

Row 6 is the most common accidental outcome, and it is the worst one: block Search because you wanted to block AI, allow Train because you never configured it, block Agent because agents sound risky. Your content is absorbed into models, cannot be found, and cannot be fetched live. If you have ever seen a site owner say "I blocked AI and my traffic died and nothing changed for the better," this is usually the state they are in.

Why the taxonomy cannot express what site owners want

Cloudflare's 2026 taxonomy is a genuine improvement over a single switch, and the announcement's own framing is honest about the tension it creates. The company's argument is that operators should split their crawlers: if a company builds a search index, acts as an agent, and trains models, it should separate those into three declared crawlers so site owners can choose.

That is the right ask. It also relocates the entire cost of the solution onto the party with the least incentive to pay it.

Consider what splitting means commercially. If Googlebot remains one crawler with two behaviors, then every site that blocks training is also blocking Google Search, which is unacceptable for most businesses. The result is that the safe default for a rational site owner becomes "allow everything," which is exactly the outcome the unified crawler operator prefers. A unified crawler is not an accident of engineering. It is leverage.

Google's counter-move already exists — Google-Extended, which opts a site out of Gemini and Vertex training without touching Googlebot. That is precisely the crawler split the taxonomy asks for, expressed as a robots.txt token instead of a separate network identity. Apple did the same with Applebot-Extended. OpenAI went further and separated the actual fetchers.

So the practical rule for 2026 is this: a purpose-specific opt-out token is worth more than a category toggle, because it survives the vendor's next taxonomy revision. Tokens are contractual and documented; category assignments are vendor judgment and can be redefined in a release note.

Which brings us back to the layer nobody wants to discuss: none of these tokens are enforced. They are read by compliant systems and ignored by non-compliant ones, and the entire 2025 Perplexity episode demonstrated that a determined operator can rotate identity faster than a UA list can be updated. If you want enforcement, you need layer 0 through layer 5, and you need to accept that you are now running an identity system with a maintenance burden.

The measurement problem: how to tell what is actually crawling you

Here is the question that decides whether any of this worked, and the one your analytics cannot answer: of the requests claiming to be a declared crawler this week, what fraction were verified?

Analytics platforms cannot answer it because they were built to count sessions, not to classify clients. Three specific distortions matter:

• Referrer loss. Traffic arriving from an AI chat interface frequently lands as direct or unattributed because the referrer is stripped, the app opens an in-app browser, or the URL was pasted by a human. A block that reduces AI referrals by 100% can show up as a small dip in "direct."
• Zero-click is invisible by construction. If an answer engine summarizes your page and the user never leaves, you have been consumed with no session to record. Your traffic graph and your extraction graph are not the same graph.
• The reporting lag is longer than the feedback loop. Crawl-rate changes appear within days in server logs, ranking effects take weeks, and aggregate field data lags further. Decisions made on a two-week window are decisions made on noise.

The fix is to stop measuring the outcome first and measure the input: classify every request, log the tier, and watch the ratio. On Cloudflare, Logpush gives you the raw material — ClientIP, ClientASN, ClientRequestUserAgent, ClientRequestPath, VerifiedBot, BotScore, BotScoreSrc. If you are not on Cloudflare, your origin access log plus a reverse-proxy tier field is enough.

The single query that matters, in pseudocode:

-- weekly, by claimed operator
SELECT
  claimed_operator,
  COUNT(*)                                        AS requests,
  SUM(CASE WHEN tier < 3 THEN 1 ELSE 0 END)       AS unverified,
  ROUND(100.0 * SUM(CASE WHEN tier < 3 THEN 1 ELSE 0 END) / COUNT(*), 2)
                                                  AS unverified_pct,
  COUNT(DISTINCT client_asn)                      AS distinct_asns
FROM request_log
WHERE claimed_operator IS NOT NULL
  AND ts >= NOW() - INTERVAL '7 days'
GROUP BY 1
ORDER BY unverified DESC;

Read it like this. A declared operator with a high unverified_pct and a small number of ASNs is probably honest infrastructure with a stale prefix list. A declared operator with a high unverified_pct and dozens of ASNs is either a distributed crawler you have not catalogued, or someone wearing its name tag. Those two cases need different responses, and no dashboard you can buy will distinguish them for you.

Build the classifier

What follows is deliberately framework-neutral: it uses the standard Request/Response shapes, so it drops into a Worker, a Hono app, a Next.js middleware file, or an Express handler with a thin adapter. It uses no dependencies.

Step 1: forward-confirmed reverse DNS, with caching

import { promises as dns } from "node:dns";

const rdnsCache = new Map(); // ip -> { ok, host, expires }
const TTL = 10 * 60 * 1000;

export async function forwardConfirmed(ip, suffix) {
  const hit = rdnsCache.get(ip);
  if (hit && hit.expires > Date.now()) return hit;

  let result = { ok: false, host: null, expires: Date.now() + TTL };

  try {
    const ptrs = await dns.reverse(ip);
    for (const raw of ptrs) {
      const host = raw.replace(/\.$/, "");
      if (!host.endsWith(suffix)) continue;

      const [v4, v6] = await Promise.all([
        dns.resolve4(host).catch(() => []),
        dns.resolve6(host).catch(() => []),
      ]);

      // compare canonically: same address family, same value
      if ([...v4, ...v6].some((addr) => sameIp(addr, ip))) {
        result = { ok: true, host, expires: Date.now() + TTL };
        break;
      }
    }
  } catch {
    // NXDOMAIN, SERVFAIL, timeout: treat as unverified, cache the negative
  }

  rdnsCache.set(ip, result);
  return result;
}

Two details that separate a working implementation from a hopeful one. sameIp must normalize IPv6, because 2001:4860:4801:10::1 and 2001:4860:4801:0010:0000:0000:0000:0001 are the same address in two spellings. And negative results must be cached, or a crawler with no PTR can turn your verifier into a DNS amplifier.

Step 2: prefix matching that fails closed

function v4ToInt(ip) {
  const p = ip.split(".").map(Number);
  if (p.length !== 4 || p.some((n) => !Number.isInteger(n) || n < 0 || n > 255)) return null;
  return ((p[0] << 24) | (p[1] << 16) | (p[2] << 8) | p[3]) >>> 0;
}

function inV4Cidr(ip, cidr) {
  const [base, bitsRaw] = cidr.split("/");
  const bits = Number(bitsRaw);
  const a = v4ToInt(ip), b = v4ToInt(base);
  if (a === null || b === null || !Number.isInteger(bits) || bits < 0 || bits > 32) return false;
  const mask = bits === 0 ? 0 : (0xffffffff << (32 - bits)) >>> 0;
  return (a & mask) === (b & mask);
}

Do not skip the validation. A malformed prefix in a fetched list should produce "no match," never a wildcard match, and IPv6 needs a real 128-bit implementation or a tested library rather than a hand-rolled hack. This is also where the Google consolidation bites: you are matching against 317 undifferentiated prefixes, so a match tells you "Google infrastructure," not "Googlebot."

Step 3: the tiered decision

// directory allowlist: the trust anchor nobody gives you
const TRUSTED = new Map([
  // "operator.example": "Operator Name",
]);

export async function classify(req, ip) {
  const ua = req.headers.get("user-agent") ?? "";
  const claimed = matchDeclaredCrawler(ua);      // your catalogue, not a magic list
  if (!claimed) return { tier: 0, operator: null, action: "allow" };

  const base = { operator: claimed.name, claimed: true };

  if (!(await inPublishedRanges(claimed, ip)))
    return { ...base, tier: 1, action: "challenge" };

  const rdns = await forwardConfirmed(ip, claimed.rdnsSuffix);
  if (!rdns.ok) return { ...base, tier: 2, action: "challenge" };

  const sig = await verifyWebBotAuth(req);        // returns { ok, directory, keyid, covered }
  if (!sig.ok) return { ...base, tier: 3, action: "allow" };

  const trusted = TRUSTED.has(sig.directory);
  const pathBound = sig.covered.includes("@target-uri") || sig.covered.includes(":path");

  if (!trusted) return { ...base, tier: 4, action: "allow" };   // signed, unvetted
  if (!pathBound) return { ...base, tier: 4, action: "challenge" };

  return { ...base, tier: 5, action: claimed.policy };         // trusted + scoped
}

Notice what the ladder does to the decision. A user-agent match alone never reaches allow. rDNS is necessary but not sufficient. A valid signature without a trusted directory is explicitly named as tier 4 and handled separately, because "cryptographically valid stranger" is a category that should exist in your code, not be silently merged into "verified."

Step 4: express it at the edge, not in the app

If you are on Cloudflare, the same logic belongs in a WAF rule, where it runs before your origin and can be changed during an incident without a deploy. A reasonable starting expression, using the managed bot fields:

(http.user_agent contains "Googlebot"
  and not cf.bot_management.verified_bot)
or (http.user_agent contains "GPTBot"
  and not cf.bot_management.verified_bot)

Action: managed challenge, not block. A challenge gives an honest crawler a path to prove itself and gives a liar a wall, and it produces a log line you can read. Blocking on a heuristic during an incident is how sites lose a crawler for three weeks without noticing.

The falsification protocol: six commands, run against your own site

Every claim in this article is checkable on infrastructure you control. Run these against your own staging host, never against someone else's.

1. Read your own preferences. curl -s https://yourhost/robots.txt | grep -i content-signal. If there is no output, you are expressing no preference at all, and any vendor dashboard claiming otherwise is describing its own defaults, not your file.
2. Prove your block is name-tag-based. curl -s -o /dev/null -w "%{http_code}" -A "GPTBot/1.2" https://yourhost/. A 200 means the block you believe exists does not apply to that path, and a 403 means it matched on a string you just typed in one flag — which is the same thing an attacker can do. Neither result is proof of verification.
3. Check the DNS loop. dig +short -x <your-ip> then dig +short <that-hostname>. If the forward lookup does not contain your IP, your own rDNS would fail your own verifier — a good way to discover that suffix matching, not forward confirmation, is what you shipped.
4. Test the SSRF surface. Send your staging host a request with Signature-Agent: sig1="http://169.254.169.254/latest/meta-data/" and a garbage Signature. Then check whether your edge made an outbound request. If it did, you have the discovery-layer SSRF, and no amount of signature validation will save you.
5. Test path binding. If you run your own signed agent, capture a signed request for a harmless path and re-issue the same headers against a different path. An 200 is a replay finding.
6. Compute your spoof ratio. Run the weekly query from the previous section. Any nonzero unverified count for a declared operator is the floor of your impersonation exposure.

What to do in Q4 2026, by site type

Seven mistakes that cost real money

1. Pasting a user-agent list and calling it access control. It is a filter with a one-header bypass, and the 2025 evidence is public.
2. Reading "Verified" as "allowed." Cloudflare explicitly decoupled the two in 2026: verified now means allowable within its category, and the category assignment is not yours.
3. Blocking training without checking what else that classification touches. If your crawler is unified, you just blocked your search engine on the pages that earn.
4. Shipping signature verification without a directory allowlist. Tier 4 is a participation trophy. It proves the sender can generate a key.
5. Fetching key directories at request time with no SSRF controls. The discovery mechanism is the vulnerability. Resolve, validate, allowlist, cap, time out.
6. Measuring the result with sessions. Zero-click consumption and stripped referrers make the outcome invisible in exactly the case you care about.
7. Treating robots.txt as enforcement. It is a preference with thirty years of social enforcement behind it and no technical enforcement at all.

Fourteen things to do, with honest hour estimates

How this connects to being cited at all

There is an irony in the middle of all this. The whole reason a site owner cares about crawler identity is that AI-mediated discovery has become a real traffic channel. But the channel runs on the same pipes you are now trying to gate.

If you block Search, you cannot be retrieved. If you block Training, you are absent from the model's parametric memory, which is the version of you that answers when no retrieval happens. If you block Agent, you are absent from the live fetch that produces a citation the user can click. Pick two of those and you have described a business model; pick zero and you have described a preference.

The practical consequence is that crawler policy is no longer an infrastructure chore sitting next to SEO. It is upstream of it. The sites that will win the next two years of AI-mediated discovery are the ones that decided, deliberately and per path, which of the three behaviors they are willing to trade — and then implemented that decision with evidence instead of strings. Our GEO guide covers what to do once you are retrievable; this article covers whether you are retrievable at all, and who decided.

The identity layer is now a governance layer

Step back from the headers and the drafts, and the shape of the last eighteen months is clear. The web spent thirty years with an implicit bargain: crawl me, and send me traffic. That bargain was enforced by mutual interest and a text file. It is now being renegotiated in public, by infrastructure companies, model providers, and publishers, using mechanisms that were never designed for negotiation.

What replaced the bargain is a stack of claims. A category assigned by a CDN. A string chosen by a client. A prefix list published by an operator. A key directory nominated by the sender. Each one is a statement about identity, and only the last two come with evidence — and even those require you to supply the trust anchor that the drafts deliberately leave blank.

Which means the question is no longer "should I block AI crawlers." That question is unanswerable in the abstract, because "AI crawler" is a category someone else defined. The answerable question is narrower and much more useful:

Which specific behaviors am I willing to accept on which specific paths, what evidence will I require before I believe the client is who it says it is, and what will I do when the evidence arrives without a trust anchor attached?

Answer that in writing. Put it in the repo next to the WAF rules. Give it an owner and a review date. Then implement the cheapest tier that expresses it, and measure the ratio instead of the outcome.

Everything else is a name tag.