Verified Crawler IP List: Build a Good-Bot Allowlist From 19 Official Feeds

A firewall rule that challenges hosting-network traffic also catches the crawlers you want. Every verified crawler address we looked up on 29 October 2026 sits on a hosting network: ClaudeBot on Amazon, DuckDuckGo on Microsoft, AhrefsBot on OVH, Googlebot on Google's own. Loosen the rule by User-Agent instead, and every scraper that types "Googlebot" into its header walks in.

The operators publish the proof for free, but in 19 files on 10 different websites, with different dates and no stated update schedule. Two of the files are the same list. The traffic is real. Announcing Imperva's 2026 Bad Bot Report, Thales wrote: "Bots now dominate the internet, accounting for over 53% of all web traffic, with 40% classified as malicious."

Below is every file with the date it carried on 29 October 2026, what GeoIPHub's IP lookup API returns for real addresses from them, a script that builds the allowlist, a policy check that won't misfire on shared files, and the bots no file covers.

Good bots vs bad bots: the difference is proof, not the name

A good bot is one you have decided to let in, and whose IP you can prove belongs to its operator. A bad bot fails either test, and the most common kind is a scraper wearing a good bot's User-Agent.

Good bots include search crawlers, AI search and user-triggered fetchers, SEO tools and web archivers. Bad bots include scrapers, credential stuffers, click bots and spam bots. Many glossaries say good bots "identify themselves". Bad bots do that too: they copy the identity, which is why scrapers borrow crawler names in the first place.

So the useful question is not "is this a good bot?" but "is this request from the operator it names?" That second question has a mechanical answer, and it is the core of bot detection by IP: the operator controls its IP ranges, and nobody else does. Whether to admit a proven bot is then your policy, and it can differ per site.

The verified crawler IP list: 19 official files from 10 operators

It works as a search engine bot IP list and more: the 19 range files GeoIPHub loads come from 10 operators, from search engines and AI companies to an archive and an SEO tool. Here is each one with its prefix count and the date written inside the file, fetched on 29 October 2026 between 04:41 and 04:42 UTC. All 19 answered HTTP 200 and share one shape: a prefixes array of ipv4Prefix or ipv6Prefix entries.

GeoIPHub's scanner re-fetches every file daily. On 27 October 2026 we looked up one address from each file, and all 19 came back verified; Perplexity's two addresses came from the files it replaced on 28 October.

That is 3,626 prefixes in total, and Google's five files hold 2,181 of them. It is also only 18 distinct lists: the two DuckDuckGo files were byte-identical when we fetched them (same MD5 hash). Perplexity's two files changed between 27 and 28 October, from 4 and 8 prefixes to 10 and 13, which is a fair warning against copying anyone's table, including this one, into a config file by hand.

Four notes. Google's old path, developers.google.com/search/apis/ipranges/googlebot.json, now answers with a 301 redirect to common-crawlers.json, so older guides that link it still work if your code follows redirects. Microsoft's file is dated January 2024. Perplexity's docs still link www.perplexity.com URLs, which redirect to the www.perplexity.ai files above. And www.gstatic.com/ipranges/goog.json covers all of Google's addresses, so it is not a crawler list. For how Google's five files split across its crawler groups, see our Googlebot guide.

What a match returns: four real lookups

A match in GeoIPHub's API returns threat.crawler.verified: true, the operator, a file label and verification_method: "ip_range_list". An address outside every file returns verified: false, which means "not verified", nothing more. Here is ClaudeBot's address, trimmed from the live response (29 October 2026, 04:42:21 UTC):

dig -x 216.73.216.10 returned no PTR record on 29 October 2026, so reverse DNS can't vouch for this address. Only Anthropic's file can. Three more addresses, looked up between 04:42:28 and 04:42:44 UTC:

Now trace one request. Suppose a request claims DuckDuckBot from 104.43.54.127, an address listed as a /32 in both DuckDuckGo files. The lookup says verified: true and operator: "DuckDuckGo", but name: "DuckAssistBot" and category: "ai". A policy written as "allow search bots, rate-limit AI bots" and keyed on the name or category would throttle DuckDuckGo's real search crawler. Keyed on the claimed bot and its operator, it passes. We expected a DuckDuckBot name and got DuckAssistBot: the name field told us which file loaded last, not which bot knocked.

Meta is the other trap. We match Meta's network (AS32934) by ASN, so is_crawler is true but verified is false and verification_method is "none". Treat it as unverified. Our docs list the crawler fields and what each one means.

One file can cover several bots, so key your policy on the operator

Match the claimed User-Agent to an operator, then require verified: true and that same operator. Never use crawler.name or category as the switch, because one file can stand for several bots:

• DuckDuckGo: two files, one list. Our scanner loads duckassistbot.json second, so the last write wins and every DuckDuckGo address carries the DuckAssistBot label.
• Anthropic: one file for ClaudeBot, Claude-User and Claude-SearchBot. Anthropic's crawler help page says "If a crawler has a source IP address on this list, it indicates that the crawler is coming from Anthropic." So a match means verified as Anthropic; the User-Agent tells you which bot.
• Google: the common-crawlers file covers several Google crawlers under one label, Googlebot.

is_crawler alone isn't strong enough for an allowlist either, since it is true for Meta's unverified match. And the API never sees your request headers. It takes an IP, so the claim side of the check always runs in your code.

How to build a crawler allowlist in four steps

Build it in four steps: map each User-Agent you allow to its operator, load that operator's files, check the IP against them, and send every claim that fails to the rate limiter.

1. The claim map. List the User-Agent tokens you allow (older WAFs call this a whitelist) and the operator whose file must contain the IP.
2. The files. Either build your own merged list with the script below, or call one lookup per new IP and cache the verdict for a few hours, not weeks, because ranges change.
3. The match. Check the IP against the claimed operator's prefixes only.
4. The unverified path. Don't answer it with a 403. Rate-limit what you can't verify, so a real crawler on a new prefix slows down instead of disappearing.

Why per operator and not per file? Because some operators make per-file pinning impossible. Anthropic's three bots share one file, and DuckDuckGo's two bots share one list, so a check that demands "Claude-SearchBot's own file" has nothing to match. Operator-level matching loses a little precision on Google, OpenAI, Perplexity and Mistral, where the files are separate, and in exchange it never misfires on a shared one.

Two rules. Match exact published prefixes, never the ASN: in our lookups, Microsoft's AS8075 carries Bingbot, GPTBot, ChatGPT-User and DuckDuckGo, so allowlisting it admits every Azure customer. And never allowlist on User-Agent alone. For bots with no file, use forward-confirmed reverse DNS where the operator documents it.

The builder fetches all 19 files, groups prefixes by operator and writes allowlist.json with a fetch time per file. If a file fails to download, fails to parse, comes back empty or shrinks to less than half its last good copy, it keeps that file's previous prefixes. GeoIPHub's scanner does the same with a fixed minimum prefix count per file.

Schedule it with one cron line, such as 20 5 * * * cd /srv/bots && node build-allowlist.mjs. The policy check reads either that file or a lookup response, and both paths key on the claimed bot's operator:

We ran both paths against the 29 October 2026 files and lookups, plus the failure cases. The GPTBot case uses 45.33.32.156, Nmap's public test host, which never sent such a request. The check is per operator, not per bot: a Googlebot claim from an address in Google's fetcher file also passes as Google.

That code is the allow side. To enforce a per-bot block policy, see the nginx and Express setup in our AI crawler guide.

Can a verified bot still be a bad bot? AhrefsBot says yes

Yes. Verification proves who operates the address, not that the address is clean or welcome. AhrefsBot's 5.39.1.230 shows both at once (29 October 2026, 04:42:51 UTC):

The verified-crawler signal subtracts 28.5 points (minus 30 at 0.95 confidence), but it doesn't cancel a blocklist listing; the risk score reasons guide shows the arithmetic. We saved two earlier values for this address: 59, step_up on 19 October, when an abuse listing also fired, and 35, review since 22 October. When we investigated this address on 19 October, RIPE's registry showed the /27 registered to Ahrefs inside an OVH block.

Your allowlist decides whether AhrefsBot gets in. The score tells you its range has history. Many sites have no use for SEO crawlers and many do, so that call is yours.

Good bots that ignore robots.txt by design

"Obeys robots.txt" is not a test for a good bot, because the operators themselves say their user-triggered fetchers may not follow it:

• Google: "Because the fetch was requested by a user, these fetchers generally ignore robots.txt rules." (Google user-triggered fetchers)
• OpenAI, on ChatGPT-User: "Because these actions are initiated by a user, robots.txt rules may not apply." (OpenAI bots)
• Perplexity, on Perplexity-User: "Since a user requested the fetch, this fetcher generally ignores robots.txt rules." (Perplexity crawlers)

The reasoning is the same in all three: the fetch acts for one person, so blocking it blocks that person's request. A verified fetcher that skips your robots.txt is behaving as documented, not misbehaving.

robots.txt asks; an allowlist enforces. If you want a user-triggered fetcher out, you need a rule at the edge, not a line in a text file. For what each OpenAI bot does and what blocking each one costs, see our OpenAI comparison.

How fresh is the list? File dates, a daily fetch and a two-day lag

The list is as fresh as the operator's last edit plus your fetch. For GeoIPHub's API, that is a daily fetch and up to about two days before a change goes live.

Here are the mechanics, from GeoIPHub's scanner code and configuration, checked 29 October 2026. The crawler step fetches each file at most once a day. If a fetch fails, or returns fewer prefixes than that file's minimum, the scanner keeps the last good ranges. A prefix the operator removes stops verifying after the next good fetch and the following daily rebuild. The lookup database then rebuilds daily at 00:00 UTC and is live by about 04:00 UTC.

The practical rule: cache a verified verdict for hours, refetch the files daily, and expect a newly added prefix to fail verification for up to about two days. That is another reason the unverified path goes to a rate limiter, not a block.

Perplexity is the only operator in our checks whose docs say anything about cadence: "These addresses are updated regularly and should be the source of truth for your WAF configurations." Google's verification page states no frequency. The file dates we fetched on 29 October 2026 run from January 2024 (Microsoft) to 28 October 2026 (Google and Perplexity). A stable date is not an error; some fleets don't move.

One gap on our side: the API doesn't show which file matched or when we fetched it. If you need that audit trail, run the builder and keep its fetched_at values.

Bots with no published list, and what to do about them

Some bots you may want publish no file GeoIPHub loads, so this list can't verify them:

• OAI-AdsBot: OpenAI documents it and publishes adsbot.json, but our scanner doesn't load it yet. Add it to your own builder if you run ads on ChatGPT.
• Amazonbot: Amazon shows its ranges inside a web page, with no file to download. We don't load it.
• Meta: Meta's crawler page says "Add to your allow list either the user agent strings or the IP addresses (more secure) used by the crawler", but gives no range file. We match its network by ASN only, unverified.
• Yandex and Baidu: reverse DNS only. Yandex's guide describes the check: the reverse DNS name must end in yandex.ru, yandex.net or yandex.com, and a forward lookup of that name must return the same IP.
• Bytespider: we found no range file from ByteDance.
• Uptime monitors and payment webhooks: many publish their own lists. Stripe publishes webhook IPs as JSON, documented on its IP addresses page. Use the vendor's list; these aren't crawlers and aren't in our feeds.
• Bots that sign requests: Web Bot Auth lets a bot prove itself with a cryptographic signature instead of an IP, covered in our crawler verification guide. Cloudflare's verified bots directory accepts a bot validated either by Web Bot Auth or by a published IP list with a stable User-Agent.

What to do: use the operator's reverse DNS method where one exists, the vendor's own list for services, and the rate limiter for everything else.

Run the builder tonight, or paste an IP from your logs into the IP lookup and check whether its Crawler row reads "Verified crawler" or "Unverified crawler". The name in the header is a claim; the operator's file is the proof.

Frequently Asked Questions

What is a verified crawler IP list?

A list built from the IP range files bot operators publish for their own crawlers. GeoIPHub loads 19 such files from 10 operators, including Google, Microsoft, Apple, OpenAI and Anthropic; OpenAI publishes a 20th, adsbot.json, that we don't load yet. A request is a verified crawler when its User-Agent names a bot and its IP is inside that operator's file.

Does an IP lookup see the User-Agent?

No. A GeoIPHub lookup takes only the IP. It can tell you the address is in Anthropic's or Google's file, and your own code compares that operator with the bot the User-Agent claims.

Can a verified bot still be a bad bot?

Yes. AhrefsBot's 5.39.1.230 matched Ahrefs' published ranges and was also listed on Spamhaus ZEN, scoring 35, review, in GeoIPHub's API on 29 October 2026. Verification tells you who operates the address, not whether to let it in.

How often do crawler IP ranges change?

It depends on the operator, and most don't say. On 29 October 2026 the file dates ran from January 2024 (Microsoft) to 28 October 2026 (Google and Perplexity). GeoIPHub re-fetches every file daily, and a change can take up to about two days to reach its API.

Should I allowlist crawlers by ASN?

No. Many crawlers run on shared clouds: Microsoft's network carries Bingbot, GPTBot and DuckDuckGo's bots, so allowlisting it admits every customer on that cloud. Match the exact prefixes in each operator's file.

Which bots can't be verified by an IP list?

Bots whose operators publish no file, such as Yandex and Baidu (reverse DNS instead), Meta (no range file; GeoIPHub matches its network by ASN, unverified) and Bytespider. Amazon shows Amazonbot's ranges only inside a web page. Rate-limit what you can't verify.