Should You Block AI Crawlers? Decide by Bot, Enforce by IP

Your access log shows GPTBot, ClaudeBot and a few names you don't recognise. The guides you find all say the same thing: list the User-Agents in robots.txt, then block the same strings in your server config.

That advice has a hole. A User-Agent is a string the client types, and typing "GPTBot" costs nothing. A blocklist of names stops honest bots, which would have obeyed robots.txt anyway, and lets through every scraper that borrows a name you allowed. The name is a claim and the IP is the proof, the same principle as bot detection by IP in general.

Below we sort the bots by job, show what robots.txt can and can't do, run four real crawler addresses through our API, and give you a three-tier policy with tested nginx and Express code. A spoofed GPTBot gets a 403. A real Googlebot never does.

Should you block AI crawlers? It depends on the bot's job

You should block AI crawlers whose job you object to and keep the ones that send you readers. AI bots do three different jobs, and blocking each one costs you something different.

• Training crawlers collect pages that may train future models. OpenAI says disallowing GPTBot "indicates a site's content should not be used in training generative AI foundation models." Blocking them costs nothing in search.
• AI search bots index pages so an assistant can cite them. OpenAI says sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers". Blocking them costs you citations and their clicks.
• User-triggered fetchers load a page because a person asked an assistant to read it. Blocking them breaks that person's request.

A paywalled publisher has a clear reason to refuse training. A brand that wants assistants to recommend it should keep the search bots. Most sites end up with a split: no to training, yes to search, a speed limit on fetchers. We cover which OpenAI bot to block and what each choice costs in a separate post. Agents that shop on a user's behalf are a different problem, covered in AI shopping agent fraud at checkout.

Which AI bots train and which cite you

This table sorts the AI bots you are most likely to see by job, with what each operator says about robots.txt. Every cell comes from the operator's own page, opened on 20 October 2026.

Proof is a separate question from policy. A bot can only be verified if its operator publishes where it crawls from:

Before you copy a list from another site, check its names. Some guides that rank for this topic still name anthropic-ai and Claude-Web, which Anthropic's current page doesn't list, and miss Claude-SearchBot and Claude-User. A stale list blocks strings nobody sends.

What robots.txt can and can't do

robots.txt can tell well-behaved bots what you want, and it can't stop anything. RFC 9309, the standard that defines it, says: "These rules are not a form of access authorization." Whether ignoring it has legal consequences is a question for a lawyer.

It is still the first tier, because most named bots read it. This file refuses training, welcomes AI search and leaves Google Search alone:

Four details decide whether that file does what you think:

• Google-Extended and Applebot-Extended are tokens, not crawlers. Google's common crawlers page, updated 15 October 2026, says "Google-Extended doesn't have a separate HTTP request user agent string" and that it "does not impact a site's inclusion in Google Search". No request ever arrives from Google-Extended, so no IP rule can block it. Apple's Applebot page says Applebot-Extended "does not crawl webpages" either.
• Blocking Googlebot to stop AI use costs you Search. So does a User-agent: * group with Disallow: /.
• User-triggered fetchers may ignore the file. OpenAI says of ChatGPT-User that "robots.txt rules may not apply". Perplexity says Perplexity-User "generally ignores robots.txt rules".
• Changes are not instant. OpenAI says "it can take ~24 hours from a site's robots.txt update for our systems to adjust".

Newer files such as llms.txt and ai.txt are proposals. Like robots.txt, they ask and don't enforce.

The opposite mistake is putting search crawlers on an "AI bot" blocklist. Contentstack's guide to blocking AI crawlers shows an edge function whose bot list includes googlebot and bingbot next to gptbot and claudebot, and answers all of them with "Forbidden: AI crawlers are not allowed." Copied as written, that rule takes a site out of Google and Bing search. Anyone pretending to be Googlebot is a fake Googlebot, a different problem with a different fix.

A real crawler or a spoof: how verification works

You tell a real crawler from a spoof by checking the connecting IP against what the named operator publishes. Which of three methods works depends on the operator.

• Range files. OpenAI, Anthropic, Perplexity, Common Crawl, Apple and Google publish JSON lists of their prefixes. For OpenAI and Anthropic this is the only method, because their addresses have no reverse DNS.
• Forward-confirmed reverse DNS. Google's verification guide has you resolve the IP to a name, check the domain, then resolve the name back and confirm it returns the same IP. Apple and Common Crawl document the same method.
• Signed requests (Web Bot Auth). A bot signs each request with a published key. It is still a draft, covered in the edge section below.

Our guide on how to verify an AI crawler by IP covers all three in detail. We looked up four crawler addresses at 04:37 UTC on 20 October 2026, without a key. This is the ClaudeBot address, trimmed, nesting kept and values unedited:

The other three, from the same run:

The GPTBot address is the first host of 132.196.86.0/24, the first prefix in OpenAI's gptbot.json. Then we asked DNS for names, at 04:45 UTC:

That is the finding most guides miss. ClaudeBot and GPTBot have no PTR record, so reverse DNS cannot verify them. The range file is the only check. no_ptr_datacenter fires on both, yet the score stays at 0, because verified_crawler is an exonerating signal. For these operators a missing PTR is normal. Googlebot passes both ways: its name resolves back to the same address.

Two limits apply. The category field says "ai" for training and non-training AI bots alike, so policy code should key on crawler.name. And the lookup sees only an IP, never the User-Agent, so the API never flags a spoof. Comparing the claimed name with the proven name happens in your code. You can run the same check by hand in our IP lookup tool.

A three-tier policy: ask, verify, throttle

A policy that holds up has three tiers: ask with robots.txt, verify every bot claim against its IP, and rate-limit whatever is left. Block by job, enforce by IP, and never put Googlebot on an AI blocklist.

Tier 1: ask. robots.txt carries your per-bot decision, plus Crawl-delay for operators that honour it.

Tier 2: verify. When a User-Agent names a bot, look up the connecting IP and compare names. If the IP doesn't belong to that bot, the request gets a 403. If it does, apply your tier 1 decision, so a verified GPTBot that ignores your Disallow is still refused.

Tier 3: throttle. Everything else gets a rate limit: requests that claim no bot, bots that can't be verified, and fetchers acting for a person.

The third row teaches the most. 216.73.216.10 is a verified crawler, just not the one the request names, so a check that only asks "is this a verified crawler?" would let it in. Compare names. Likewise, 107.178.192.1 is a verified Google fetcher, but a request from it claiming Googlebot is not Googlebot.

Log every tier 2 failure. An address outside the published ranges usually means a spoof, but it can be a prefix the operator just added. Our range files are re-fetched daily and the lookup database rebuilt daily, so a new prefix can take up to about two days to verify.

Enforce it in nginx and Express, with a spoofed-GPTBot test

Enforcement is one small Express service that does the lookup, plus an nginx config that asks it about every request claiming to be a bot. Stock nginx can't call an HTTP API inline, so the auth_request module sends a subrequest to the verifier and nginx caches the answer. Without nginx, the same file protects an Express app on its own.

The verifier matches the User-Agent against current bot tokens, looks up the IP with GET https://api.geoiphub.com/v1/lookup/{ip} and the x-api-key header, caches the result for an hour, and compares threat.crawler.name with the claimed name. It needs a name map because the API writes some names without the hyphen: OAISearchBot, ChatGPTUser, PerplexityUser. See the crawler fields in the API docs.

Three choices are deliberate. If the lookup fails or takes over 1.5 seconds, the request is never treated as verified; it falls through to the rate limiter. Behind nginx, that answer is sent with no-store, so nginx doesn't cache it and the verifier counts every request against the same limit. The policy list holds User-Agent tokens, so refusing ClaudeBot doesn't refuse Claude-SearchBot, even though both come back from the lookup as ClaudeBot. trust proxy is loopback only, because otherwise a client could send its own X-Forwarded-For and choose the address you check. And only bot claims are looked up, so volume follows bot traffic, not page views. The free tier covers 1,500 lookups a day with a key, every field included.

The nginx side maps the User-Agent to a claimed bot, skips the subrequest when there is no claim, and caches the verifier's 204 or 403 per address and bot for an hour, except answers marked no-store:

nginx -t passed on this config, and we ran it in front of the Express file on 20 October 2026, at 05:05 UTC. First the spoofed GPTBot test, through nginx, from the same machine:

That GPTBot string is the current example on OpenAI's bots page. From your own machine the request comes from 127.0.0.1, which no crawler uses, so the verifier refuses it before any lookup. To watch the lookup, call the Express file directly with the address nginx would forward:

Follow the first line end to end. The User-Agent claims GPTBot. The verifier looks up 216.73.216.10, and the API answers with a verified crawler named ClaudeBot. ClaudeBot is not GPTBot, so the request gets a 403. The real GPTBot address, 132.196.86.1, passes verification and is refused only because this policy refuses training; with BLOCKED_BOTS=CCBot it got a 200. A Claude-SearchBot claim from 216.73.216.10 got a 200, because this policy refuses the ClaudeBot token, not Anthropic's search bot. A test-only copy of the nginx config that forwarded the same four addresses returned 403, 403, 403 and 200. With the API made unreachable on purpose, 130 bingbot claims from one address through nginx got 120 responses with status 200, then 403s: an outage throttles bot claims instead of waving them through.

If you test without a key, keep it short. Anonymous lookups are limited per minute, and the 30th call inside one 60-second window is already a 429 that locks your IP out for two hours.

Rate-limit what you can't verify instead of blocking it

Rate limiting is the right tool for traffic you can't prove either way: unverifiable bots, user-triggered fetchers and anything that claims no bot. A limit stops a scraper copying your site in an afternoon without locking out the people behind a fetcher.

Both snippets already do this. nginx's limit_req keys on the client address for requests with no bot claim, at 10 a second with a burst of 20. In our test, 40 back-to-back requests returned 25 successes and 15 responses with status 429. Tune the numbers, and exclude static assets if one page load fetches dozens. The Express limiter keys IPv6 clients on their /64, because one subscriber usually controls a whole /64; see rate limiting IPv6 by /64 prefix for when to widen to /56 or /48.

Crawl-delay is the polite version for bots that honour it. Anthropic's crawler help page says "we support the non-standard Crawl-delay extension to robots.txt", and Common Crawl's FAQ says "We obey the Crawl-delay parameter". Others don't. Google's robots.txt specification says fields "such as crawl-delay aren't supported", and Apple says "Applebot does not follow crawl-delay". OpenAI's and Perplexity's pages don't mention it.

Tier 2 only catches requests that name a bot. To block AI scrapers that never claim to be bots, add behaviour to tier 3: request rates, paths, and the network the request comes from. Our post on how to detect web scraping from your logs covers those signals.

Edge options: Cloudflare's AI blocking and pay per crawl

Behind Cloudflare, you can block AI crawlers with a setting instead of code, or charge them per request. Neither removes the need to know which bots you are dealing with.

Cloudflare announced on 1 July 2025, in a post last modified 15 July 2026, that it was "changing the default to block AI crawlers unless they pay creators for their content." If you run your own rules too, check what Cloudflare already blocks on your zone so the two layers agree.

With pay per crawl, a crawler requesting a priced page gets "a 402 Payment Required response, accompanied by a crawler-price header", and can retry with payment intent. Crawlers sign requests under the Web Bot Auth proposals so nobody can pay as someone else. Cloudflare says pay per crawl "is currently in private beta".

Web Bot Auth is not a standard yet. On 20 October 2026 the IETF webbotauth working group had one working-group document, draft-ietf-webbotauth-httpsig-protocol-00 from 1 September 2026, and no RFC; its charter's April 2026 milestone for a standards-track specification has passed. An individual draft, draft-illyes-webbotauth-jafar, proposes a common JSON format for crawler range files. Until those land, IP ranges and reverse DNS are what you can check.

Measure what the policy changed

Measure the policy by logging the verification result on every bot request, then comparing before and after.

• Share by verified bot. Log crawler.name and crawler.verified next to each request that claimed a bot. The share of failed claims is your spoofing rate. The share by name shows which operators actually crawl you.
• AI referrals. Count visits referred by chatgpt.com, perplexity.ai, claude.ai, copilot.microsoft.com and gemini.google.com. Treat that count as a floor, since some assistant traffic arrives with no referrer.
• Crawl volume. Compare requests per day from each bot before and after a change. Search Console's Crawl stats report covers Google's crawlers only, so the other bots have to come from your own logs.

If citations drop after you block an AI search bot, you have your answer on that bot.

When to revisit your AI crawler policy

Revisit your AI crawler policy whenever an operator adds a bot, changes a range file or retires a token, and at least once a quarter. OpenAI's bots page now lists OAI-AdsBot, with its own adsbot.json. Our crawler feeds don't include that file and the code above doesn't name it, so an OAI-AdsBot claim falls to tier 3 until you decide.

Be clear about what this setup can't do:

• It only proves bots that publish ranges. Bytespider and Meta-ExternalAgent can't be verified, and a spoof of either looks exactly like the real thing.
• New prefixes lag. A prefix added today can take up to about two days to verify, so a real bot can briefly fail tier 2.
• IP blocks can backfire. Anthropic warns that blocking its IPs "may not work correctly or persistently guarantee an opt-out, as doing so impedes our ability to read your robots.txt file." Keep the Disallow in robots.txt even when tier 2 enforces it.
• The lookup can't see the User-Agent. The spoof check is the name comparison in your code.
• Tier 2 misses scrapers that never claim a name. Those are tier 3's job.

Start with your own logs. Pick one address that claimed GPTBot or ClaudeBot this week, run it through a lookup, and compare crawler.name with the name it claimed.