How to Verify AI Crawlers and Agents by IP: Reverse DNS, CIDR Ranges, and Web Bot Auth
How to verify an AI crawler by IP: forward-confirmed reverse DNS, published CIDR ranges, and Web Bot Auth (RFC 9421) for GPTBot, Googlebot and AI agents.

You see GPTBot, ClaudeBot, or PerplexityBot in your logs and you have to make a call: allow it, rate-limit it, or block it. But before any of that, you have to answer a more basic question: is it actually that bot? Because a User-Agent header is a claim, not proof. Anyone can run curl -A "GPTBot/1.3" https://your-site.com and your logs will dutifully record "GPTBot."
This isn't theoretical. HUMAN Security's Satori team found that 5.7% of all traffic claiming to be a well-known AI crawler was spoofed, peaking at 7.7% on some days (~2 million requests/day), with the ChatGPT-User agent impersonated in roughly one of every six requests. This guide covers the three real ways to verify a crawler at the IP and crypto layers (forward-confirmed reverse DNS, operator-published CIDR allowlists, and Web Bot Auth), plus the failure modes that trip up almost every other guide.

Why a User-Agent of "GPTBot" proves nothing
The User-Agent is self-reported metadata. There is no authentication in it, nothing signed, nothing an origin server can check. Verifying a crawler means tying the request to infrastructure or a key the operator actually controls. There are exactly three regimes that do that:
- Forward-confirmed reverse DNS (FCrDNS): prove the IP resolves to and from the operator's domain.
- Published CIDR allowlists: match the IP against ranges the operator publishes.
- Web Bot Auth (RFC 9421): verify a cryptographic signature on the request itself.
Everything below is about doing these correctly, and knowing what each one can and cannot prove.
The 2026 stakes: bots are the majority of the web
Crawler verification stopped being an SEO niche the moment automated traffic became the bulk of the internet. Imperva's 2026 Bad Bot Report found that automated traffic made up 53% of the web traffic it observed in 2025, and that bad bots alone made up 40%. When most requests are automated and AI crawlers keep growing in volume, "is this bot real?" becomes a question at your login, signup, and checkout endpoints, not just your robots.txt. At checkout it is now a fraud question too: see how to stop AI shopping agent fraud without blocking real agent sales. For which IP signal answers which bot question, from crawler ranges to scanner flags, see our hub on bot detection by IP.
Method 1: Forward-confirmed reverse DNS, done right
FCrDNS is the canonical method Google documents for verifying Googlebot. Four steps:
# 1. Reverse-DNS the source IP → hostname
host 66.249.66.1
# → 1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com
# 2. Confirm the suffix matches the operator
# Googlebot: googlebot.com, google.com, or googleusercontent.com
# Bingbot: *.search.msn.com
# 3. Forward-DNS that hostname → IP
host crawl-66-249-66-1.googlebot.com
# → crawl-66-249-66-1.googlebot.com has address 66.249.66.1
# 4. Confirm it matches the original IP. Match = verified.
Why the forward step is load-bearing: an attacker controls the reverse (PTR) record for their own IP, so they can point 1.2.3.4 at crawl-fake.googlebot.com. What they cannot do is forge googlebot.com's authoritative forward DNS to make that hostname resolve back to 1.2.3.4. A check that only string-matches the PTR record for "googlebot.com" (which a surprising number of tutorials suggest) is itself spoofable. The forward confirmation is the whole point. For Google specifically, with its three crawler types and their separate range files, see our guide to verifying Googlebot by IP and spotting fake Googlebots.
The accuracy ceiling to flag: PTR records are optional and frequently absent, especially for the newer AI vendors. When there's no PTR, FCrDNS returns "unverifiable", which is neither pass nor fail. That's where Method 2 comes in.
Method 2: Published CIDR allowlists, per bot
Most AI vendors skip PTR records and instead publish their crawler IP ranges as JSON. This is the most operationally practical check. Pull each operator's primary-source feed:
- OpenAI: per-bot files:
openai.com/gptbot.json,openai.com/searchbot.json,openai.com/chatgpt-user.json(OpenAI crawler docs). - Google: five CIDR files under
developers.google.com/static/crawling/ipranges/. - Anthropic: publishes one file for its bots,
claude.com/crawling/bots.json, linked from its crawler help page. The address list in its platform docs is for the API, not for crawlers. - Perplexity:
perplexitybot.jsonandperplexity-user.json. Neither Anthropic nor Perplexity documents reverse DNS, so their files are the only check; ClaudeBot and PerplexityBot IP ranges has all three files with their dates and a real lookup from each. - Apple iCloud Private Relay: an egress CSV at
mask-api.icloud.com/egress-ip-ranges.csv(Apple docs). Note: these are humans on a privacy relay, not bots. Allowlist them; don't block them.
We load 19 crawler range files from 10 operators; the verified crawler IP list has each one, with the date in it and how to build the allowlist.
Three rules that separate a correct implementation from a broken one:
- Match each bot to its operator's files, and to its own file only where the operator publishes one per bot.
GPTBotlives ingptbot.json, not "some OpenAI range", and matching the wrong OpenAI file conflates a training crawler with a user-initiated fetcher. Anthropic publishes one shared file for all three of its bots, and DuckDuckGo's two files are identical, so there the operator is the most precise match available. Our comparison of GPTBot vs OAI-SearchBot vs ChatGPT-User shows what each OpenAI bot does, what blocking it costs, and a real lookup from each file. - Match the exact published prefix, never the hosting ASN. Allowlisting "all of AWS" or "all of Google Cloud" because a bot happens to run there will wave through every attacker on the same cloud.
- Refresh on a schedule. These ranges change; treat them as a feed, not a constant.
Vendor asymmetry: which method even exists
Most guides blur the vendors together. They differ, both in which verification method exists and in the crawler tokens each ships, which carry different policy implications:
- FCrDNS: ✅ (canonical)
- Published CIDR: ✅
- Web Bot Auth: ⚠️ experimental
- Notable tokens (different policies): Googlebot vs user-triggered fetchers
OpenAI
- FCrDNS: ✕ (mostly no PTR)
- Published CIDR: ✅ per-bot
- Web Bot Auth: emerging
- Notable tokens (different policies): GPTBot (training) · ChatGPT-User (user fetch) · OAI-SearchBot
Anthropic
- FCrDNS: ✕
- Published CIDR: ✅
- Web Bot Auth: emerging
- Notable tokens (different policies): ClaudeBot · Claude-User · Claude-SearchBot
Perplexity
- FCrDNS: ✕
- Published CIDR: ✅ (disputed)
- Web Bot Auth: none
- Notable tokens (different policies): PerplexityBot · Perplexity-User

The nuance that breaks naive pipelines: user-directed fetchers behave like real visitors and may legitimately originate outside published bot ranges. When a user asks an assistant to open a specific URL, that fetch can come from general cloud infrastructure with a normal browser User-Agent. Blocking "not in the GPTBot range" can therefore block a real user's agent action. Rate-limit and block policy has to differ per token, not per vendor.
Verifying SEO and analytics crawlers: AhrefsBot and friends
The same two IP-layer methods verify the SEO and analytics crawlers people most often want to pin down in their logs: AhrefsBot, and peers like SemrushBot and DataForSeoBot. They are not AI crawlers, but the verification is identical:
- AhrefsBot is forward-confirmed reverse DNS verifiable (its addresses resolve to hostnames under Ahrefs's own domains), and Ahrefs publishes its crawler IP ranges for allowlisting. Run the same FCrDNS check from Method 1, or match against the published list from Method 2.
- The trap is the one that catches AI crawlers too: do not allowlist by ASN. "AhrefsBot" arriving from a random hosting range you never verified is exactly the spoof pattern, since anyone can send
User-Agent: AhrefsBot/7.0. Confirm the IP, not the string.
If you are searching for "AhrefsBot IP addresses" to build a firewall rule, the durable version is a forward-confirmed reverse-DNS check plus the operator's published range, refreshed on a schedule, not a hardcoded IP list that rots the first time the crawler adds capacity.
Method 3: Web Bot Auth and cryptographic verification
The structural fix to spoofing is to stop trusting the network position and verify a signature instead. Web Bot Auth has the agent sign each request with an Ed25519 key using RFC 9421 HTTP Message Signatures (a Proposed Standard, Feb 2024). The request carries:
Signature: the signature valueSignature-Input: covered components,created/expires,keyid, andtag="web-bot-auth"Signature-Agent: optional, names the signing agent
Keys are discovered as a JWKS at /.well-known/http-message-signatures-directory (built on the RFC 8615 well-known URI mechanism). Verification: fetch the directory, match the keyid thumbprint, verify the signature and the created/expires window. The advantage over IP methods is decisive: it holds up on shared, rotating, and CGNAT addresses, because identity rides the request, not the route.
Reality check: it's experimental, not a standard
Be honest with your architecture here. Web Bot Auth is specified in IETF Internet-Drafts (draft-meunier-web-bot-auth-architecture, 2 Mar 2026, Cloudflare + Google authors) that carry the boilerplate "not endorsed by the IETF and has no formal standing" and expire 3 Sep 2026. Google's own support is explicitly experimental. It is "not yet signing every request" and tells operators to keep using IP, reverse DNS, and User-Agent verification. At the infrastructure layer it's further along: Cloudflare folded Message Signatures into its Verified Bots Program (Jul 2025) and AWS WAF added support (Nov 2025). Verdict: prefer signatures where present, but do not depend on them yet.
The layered decision pipeline (copy this)
Combine the three methods into a precedence order, then enrich and classify. Once a bot is verified, decide what to do with each verified AI crawler with the three-tier policy in that guide:
1. Parse the claimed bot token from the User-Agent; treat it as a HINT only.
2. If a Web Bot Auth signature is present → verify against the JWKS.
PASS → verified-good (strongest; IP-independent). Done.
3. Else match source IP against the vendor's published CIDR file,
pinned to that specific token.
IN RANGE → verified-good.
4. Else run FCrDNS where a PTR record exists.
FORWARD-CONFIRMED → verified-good.
5. Enrich the IP (this is where an IP-intelligence API slots in):
ASN / owner, hosting/datacenter flag, VPN/proxy/Tor flags,
and an IP risk score.
6. Classify:
verified-good → allow
declared-but-unverified → rate-limit or challenge (NOT hard block)
undeclared automation → challenge / block by behavior + provenance
Step 5 is where GeoIPHub fits: when a self-identified bot's IP isn't in a published range and has no PTR, the ASN owner, datacenter/hosting flag, VPN, proxy and Tor flags, and a risk score are what let you score the inconclusive cases at login, signup, and checkout instead of guessing. If you rate-limit the declared-but-unverified bucket, key IPv6 sources on the prefix rather than the address; our guide to rate-limiting and blocking IPv6 by /64 prefix explains why.
Same Google ASN, one verified Googlebot and one not
Two addresses in the same Google network get opposite crawler verdicts, because the check uses the published prefix, not the ASN. We looked up two Google IPs on 2026-10-05 (results change as ranges move): 192.178.4.0, the first address of the first IPv4 prefix in Google's published Googlebot range file, and 8.8.8.8, Google Public DNS.
[
{
"ip": "192.178.4.0",
"asn": { "asn": 15169, "org": "GOOGLE - Google LLC", "asn_type": "hosting", "user_type": "search_engine_spider" },
"detection": { "is_hosting": true, "is_vpn": false, "is_proxy": false },
"threat": {
"is_crawler": true,
"crawler": {
"verified": true,
"name": "Googlebot",
"operator": "Google",
"category": "search_engine",
"verification_method": "ip_range_list"
}
},
"scoring": {
"fraud_score": 0,
"recommended_action": "allow",
"detection_methods": ["datacenter_ip", "no_ptr_datacenter", "verified_crawler"]
}
},
{
"ip": "8.8.8.8",
"asn": { "asn": 15169, "org": "GOOGLE - Google LLC", "asn_type": "hosting", "user_type": "hosting" },
"detection": { "is_hosting": true, "is_vpn": false, "is_proxy": false },
"threat": {
"is_crawler": false,
"crawler": { "verified": false, "name": "none", "verification_method": "none" }
},
"scoring": {
"fraud_score": 15,
"recommended_action": "allow",
"detection_methods": ["datacenter_ip", "no_ptr_datacenter"]
}
}
]
Both sit in AS15169 and both are hosting, so an ASN allowlist would treat them the same. Only the first is inside the Googlebot file, and threat.crawler.verification_method: "ip_range_list" shows the verdict came from Method 2. The verified_crawler signal pulls its fraud_score to 0, while 8.8.8.8 keeps the plain datacenter score of 15. The no_ptr_datacenter method fires on both because our record holds no PTR for them, yet a live dig -x 192.178.4.0 returns crawl-192-178-4-0.googlebot.com. That is a small, real example of why the range check is the one to automate: it does not depend on a PTR lookup succeeding.
Failure modes and false positives
This is what the ranking articles get wrong:
- Out-of-range ≠ fake. The Perplexity case (Cloudflare, Aug 2025) showed a crawler using a generic Chrome UA, IPs outside its published range, and rotating ASNs at 3-6M requests/day. In-range is a true positive; out-of-range is inconclusive, not proof of fakery.
- CGNAT geographic bias. Cloudflare research in late 2025 (via The Register) found CGNAT IPs were rate-limited ~3× more often despite lower bot signals. Hundreds of users share one IPv4, concentrated in Africa and Asia. Naive IP blocking is a fairness problem.
- iCloud Private Relay. Shared, rotating egress IPs carrying real humans; allowlist them via Apple's CSV.
- Residential proxies. They defeat ASN allowlisting entirely (Imperva's 2025 report found 21% of bot attacks used them), which is exactly why detecting anonymized traffic without blocking real customers is a behavioral problem, not a list lookup.
The rule: reserve hard blocks for behavior plus provenance, never for absence from a list alone.
Verified bots are GIVT, spoofed ones are SIVT
In the Media Rating Council's taxonomy, declared crawlers and known datacenter IPs are General Invalid Traffic, filterable by lists, while bots "masquerading as legitimate users" are Sophisticated Invalid Traffic that needs analysis and review. Verifying a crawler by IP is what moves a request from the second bucket into the first. The full category lists, Google's own nine categories and which of them an IP lookup can see are in GIVT vs SIVT: Google's invalid traffic categories.
Verify AI crawlers and agents with GeoIPHub
GeoIPHub supplies the step-5 enrichment the geo-IP incumbents leave out of their how-to content: ASN and connection-type, hosting/datacenter detection, VPN / proxy / Tor flags, verified-crawler flags from 19 official range feeds (including GPTBot, ClaudeBot, PerplexityBot and AhrefsBot), and an IP fraud-risk score. We do not detect residential proxies; those need behavioral signals. That lets you confirm a self-identified bot actually originates from vendor infrastructure, and score the inconclusive cases instead of blunt-blocking them. To check one request by hand, paste its IP and user agent into the crawler IP verifier. Pair it with the verification methods above and the related guides on account-takeover and credential-stuffing detection and how accurate IP geolocation really is.
FAQ
How do I verify that a request is really from GPTBot and not a spoof?
Don't trust the User-Agent header; it is trivially forged. Match the source IP against OpenAI's published per-bot range file (openai.com/gptbot.json), pinning the check to GPTBot specifically rather than a generic OpenAI range. If a valid Web Bot Auth signature is present, verify it against OpenAI's key directory first, since that is IP-independent. Treat an IP that is not in the published range as inconclusive, not proven fake, because user-directed agent traffic can originate from other infrastructure. Our GPTBot vs OAI-SearchBot vs ChatGPT-User comparison shows a real lookup from each OpenAI range file.
What is forward-confirmed reverse DNS (FCrDNS)?
FCrDNS is a four-step check: reverse-DNS the source IP to a hostname, confirm the hostname ends in the operator's domain, forward-DNS that hostname, then confirm it resolves back to the original IP. The forward step is what defeats spoofing: an attacker can set a fake PTR record on their own IP, but cannot forge the operator's authoritative forward DNS. A check that only inspects the PTR string is spoofable.
Why can't I just block AI crawlers by User-Agent string?
Because the User-Agent is a claim anyone can send. HUMAN Security found 5.7% of all traffic labeled as a well-known AI crawler was fake, peaking near one in six requests for the ChatGPT-User agent. Blocking by header alone lets spoofed bots through while doing nothing to stop an operator who simply stops declaring itself.
What is Web Bot Auth and is it a finished standard?
Web Bot Auth is an emerging method where a bot cryptographically signs each HTTP request with an Ed25519 key using RFC 9421 HTTP Message Signatures, and the origin verifies it against keys published at /.well-known/http-message-signatures-directory. RFC 9421 itself is a Proposed Standard (Feb 2024), but Web Bot Auth is still an IETF Internet-Draft with no formal standing as of mid-2026. Treat it as experimental: verify signatures where present, but keep IP and reverse-DNS verification as the baseline.
Where do AI vendors publish their crawler IP ranges?
OpenAI publishes per-bot JSON at openai.com/gptbot.json, searchbot.json, and chatgpt-user.json; Google publishes CIDR JSON under developers.google.com/static/crawling/ipranges/; Anthropic publishes one file for its bots at claude.com/crawling/bots.json; Perplexity publishes perplexitybot.json and perplexity-user.json. Pin each check to the specific bot's file, match exact published prefixes rather than the hosting ASN, and refresh the files on a schedule. For the three OpenAI files and their current prefix counts, see GPTBot, OAI-SearchBot and ChatGPT-User IP ranges.
If an IP is not in the vendor's published range, does that mean the request is fake?
No. An in-range IP, a passing FCrDNS, or a valid signature is a high-confidence true positive, but an out-of-range IP is inconclusive. It could be legitimate user-directed agent traffic from other infrastructure, a CGNAT-shared address, an iCloud Private Relay egress, a vendor that publishes no PTR records, or a stale range file on your side. Reserve hard blocks for behavioral and provenance signals, not for absence from a list alone.
Won't blocking crawler IP ranges affect real users on shared addresses?
Yes, which is why IP-absence should not trigger hard blocks. Cloudflare research in late 2025 found CGNAT addresses were rate-limited about three times more often than non-CGNAT ones despite showing lower bot indicators, because hundreds to thousands of users share one IPv4 address, and CGNAT is concentrated in parts of Africa and Asia, creating geographic bias. iCloud Private Relay egress IPs likewise carry real humans and should be allowlisted using Apple's published CSV.
The team behind the GeoIPHub API
Written by the GeoIPHub team, the people who build the GeoIPHub IP intelligence API and its scanner. Drafts are prepared with AI assistance and edited by the team; API examples are real lookups against the live API unless labelled illustrative.


