AI Crawler List: User Agents of 29 Bots and Which You Can Verify by IP
An AI crawler list of 29 bots by job, with user agents, robots.txt wording and range files, and the 16 that GeoIPHub verifies by IP, checked 9 November 2026.

Scope: bots that an operator documents on a page we could open on 9 November 2026, as used for AI. Meta also documents meta-externalads, an ads crawler, which we did not count. Names that circulate on other lists without operator documentation are in a separate list, with what we checked.
Operators publish a way to prove 25 of these 29 bots. GeoIPHub's API verifies 16. The other nine are not a gap in what operators document: they are files and pages we do not load, and the list says which.
Other lists give names and tokens, but a name in a User-Agent header proves nothing, because anyone can type it. The operators' proof comes in five forms: a JSON file, a web page of addresses, reverse DNS, a signature, or nothing. This post is part of our work on bot detection by IP. Below: each bot by job, the 22 range files with their counts and dates, what GeoIPHub's API returned for a real address from 14 of the files we load, a snippet that checks a log line against the right file, and the names that never reach your logs or that we could not source.

The AI crawler list: 29 bots by job
This AI crawler list sorts 29 documented bots and their AI crawler user agents by the job their operator states: training, search, user-triggered fetching, agents, and a short group that is none of those. Four terms are used in every entry. Proof means what the operator publishes: a range file, a web page of addresses, reverse DNS, a signature, or nothing. Verified means an address from that bot's own file returned verified: true in GeoIPHub's API on the date shown. ASN match only means we match the operator's network and return verified: false. Not loaded means we do not load the operator's list.
The grouping follows the operators' words and is not clean. OpenAI says "If your site has allowed both bots, we may use the results from just one crawl for both use cases". Meta says "use cases such as training". Amazon says "may be used to train". Apple's crawler is search first and "may also be used to help train" its models. Cloudflare's bot documentation draws the same three-way split: Search ("Crawling to build search indexes or RAG databases"), Agent ("User-directed agents visiting a page on behalf of a human") and Training ("Crawling to train or fine-tune models").

Which AI crawlers train models?
Six, by their operators' own wording.
GPTBot (OpenAI)
- UA fragment: GPTBot/1.4
- Proof: file
- robots.txt, operator's words: "Disallowing GPTBot indicates a site’s content should not be used in training generative AI foundation models."
- Verified by GeoIPHub: verified
- Checked: 2026-11-09
ClaudeBot (Anthropic)
- UA fragment: ClaudeBot
- Proof: file (shared)
- robots.txt, operator's words: "Anthropic's Bots respect "do not crawl" signals by honoring industry standard directives in robots.txt."
- Verified by GeoIPHub: verified (as Anthropic)
- Checked: 2026-11-09
MistralAI-Training (Mistral)
- UA fragment: MistralAI-Training/1.0
- Proof: none
- robots.txt, operator's words: "Webmasters can disallow this user agent in their robots.txt file."
- Verified by GeoIPHub: not loaded (mistralai-training-ips.json returns 404)
- Checked: 2026-11-09
meta-externalagent (Meta)
- UA fragment: meta-externalagent/1.1
- Proof: none
- robots.txt, operator's words: "In order to block these crawlers, add a disallow for the relevant crawler to robots.txt."
- Verified by GeoIPHub: ASN match only, unverified
- Checked: 2026-11-09
Amazonbot (Amazon)
- UA fragment: Amazonbot/0.1
- Proof: web page only
- robots.txt, operator's words: "Automated crawling from these listed user agents respects the Robots Exclusion Protocol, honoring the user-agent and the allow/disallow directives."
- Verified by GeoIPHub: not loaded
- Checked: 2026-11-09
KimiBot (Moonshot)
- UA fragment: KimiBot/1.0
- Proof: file (one file, three names)
- robots.txt, operator's words: not quoted
- Verified by GeoIPHub: not loaded
- Checked: 2026-11-09
Purpose, in the operators' words: GPTBot is "used to crawl content that may be used in training our generative AI foundation models". MistralAI-Training "crawls web content to help build datasets for training Mistral generative AI models". KimiBot "crawls content potentially used to train Kimi's foundation models".
Which AI crawlers power search and answers?
Eleven, including two classic search crawlers that sit in our loaded set.
OAI-SearchBot (OpenAI)
- UA fragment: OAI-SearchBot/1.4
- Proof: file
- robots.txt, operator's words: "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers"
- Verified by GeoIPHub: verified
- Checked: 2026-11-09
Claude-SearchBot (Anthropic)
- UA fragment: Claude-SearchBot
- Proof: file (shared)
- robots.txt, operator's words: same Anthropic sentence as ClaudeBot
- Verified by GeoIPHub: verified (as Anthropic)
- Checked: 2026-11-09
PerplexityBot (Perplexity)
- UA fragment: PerplexityBot/1.0
- Proof: file
- robots.txt, operator's words: not quoted
- Verified by GeoIPHub: verified
- Checked: 2026-11-09
MistralAI-Index (Mistral)
- UA fragment: MistralAI-Index/1.0
- Proof: file
- robots.txt, operator's words: not quoted
- Verified by GeoIPHub: verified
- Checked: 2026-11-09
meta-webindexer (Meta)
- UA fragment: meta-webindexer/1.1
- Proof: none
- robots.txt, operator's words: same Meta sentence as meta-externalagent
- Verified by GeoIPHub: ASN match only, unverified
- Checked: 2026-11-09
Amzn-SearchBot (Amazon)
- UA fragment: Amzn-SearchBot/0.1
- Proof: web page only
- robots.txt, operator's words: "If robots.txt files don't mention Amzn-SearchBot but allow other search bots, Amzn-SearchBot will crawl in accordance with the robots.txt directives given to other search bots."
- Verified by GeoIPHub: not loaded
- Checked: 2026-11-09
Applebot (Apple)
- UA fragment: Applebot/0.1
- Proof: file + reverse DNS
- robots.txt, operator's words: "Applebot respects standard robots.txt directives in general search crawls that are targeted at Applebot."
- Verified by GeoIPHub: verified
- Checked: 2026-11-09
Bingbot (Microsoft)
- UA fragment: Bingbot
- Proof: file
- robots.txt, operator's words: not quoted (Microsoft's page needs JavaScript, so we could not read it)
- Verified by GeoIPHub: verified
- Checked: 2026-11-09
DuckAssistBot (DuckDuckGo)
- UA fragment: DuckAssistBot/1.2
- Proof: file
- robots.txt, operator's words: "Once you've updated the file to disallow the DuckAssistBot user agent, the change will take effect after 72 hours and DuckAssistBot will stop crawling your site."
- Verified by GeoIPHub: verified (DuckDuckBot addresses return this name too)
- Checked: 2026-11-09
Kimi-SearchBot (Moonshot)
- UA fragment: Kimi-SearchBot/1.0
- Proof: file (shared)
- robots.txt, operator's words: not quoted
- Verified by GeoIPHub: not loaded
- Checked: 2026-11-09
YouBot (You.com)
- UA fragment: YouBot/1.0
- Proof: signature + reverse DNS + web page only (one /24)
- robots.txt, operator's words: "It respects robots.txt directives and crawl rate preferences"
- Verified by GeoIPHub: not loaded
- Checked: 2026-11-09
Bingbot and Applebot are here because they are in our loaded set and Apple says its crawler may help train its models. We could not read Microsoft's crawler page, which needs JavaScript, but its 30 October post sets AI preferences through Bingbot. YouBot is the only entry with three proofs: a signature directory at you.com/.well-known/http-message-signatures-directory, reverse DNS of the form youbot-68-67-112-106, and the statement "Legitimate YouBot requests originate from 68.67.112.0/24."
Purpose, in the operators' words: PerplexityBot is "designed to surface and link websites in search results on Perplexity. It is not used to crawl content for AI foundation models." MistralAI-Index is "for automated crawling of the web for indexing purposes only". DuckAssistBot "crawls pages in real-time for our AI-assisted answers" and "This data is not used in any way to train AI models."
Which AI crawlers fetch a page because a user asked?
Seven, plus Google's user-triggered fetchers.
ChatGPT-User (OpenAI)
- UA fragment: ChatGPT-User/1.0
- Proof: file
- robots.txt, operator's words: "Because these actions are initiated by a user, robots.txt rules may not apply."
- Verified by GeoIPHub: verified
- Checked: 2026-11-09
Claude-User (Anthropic)
- UA fragment: Claude-User
- Proof: file (shared)
- robots.txt, operator's words: Anthropic's sentence for all its bots (see ClaudeBot); no user-triggered carve-out found
- Verified by GeoIPHub: verified (as Anthropic)
- Checked: 2026-11-09
Perplexity-User (Perplexity)
- UA fragment: Perplexity-User/1.0
- Proof: file
- robots.txt, operator's words: "Since a user requested the fetch, this fetcher generally ignores robots.txt rules."
- Verified by GeoIPHub: verified
- Checked: 2026-11-09
MistralAI-User (Mistral)
- UA fragment: MistralAI-User/1.0
- Proof: file
- robots.txt, operator's words: "MistralAI-User governs which sites these user requests can be made to."
- Verified by GeoIPHub: verified
- Checked: 2026-11-09
meta-externalfetcher (Meta)
- UA fragment: meta-externalfetcher/1.1
- Proof: none
- robots.txt, operator's words: "may bypass robots.txt because it performs fetches that were requested by the user"
- Verified by GeoIPHub: ASN match only, unverified
- Checked: 2026-11-09
Amzn-User (Amazon)
- UA fragment: Amzn-User/0.1
- Proof: web page only
- robots.txt, operator's words: "Because actions taken by Amzn-User can be initiated by a user, it may not follow all robots.txt directives."
- Verified by GeoIPHub: not loaded
- Checked: 2026-11-09
Kimi-User (Moonshot)
- UA fragment: Kimi-User/1.0
- Proof: file (shared)
- robots.txt, operator's words: "because actions are user-triggered, robots.txt rules may not directly apply"
- Verified by GeoIPHub: not loaded
- Checked: 2026-11-09
Google's page also names Google-GeminiNotebook, which "requests individual URLs that Gemini Notebook users have provided as sources". It does not say which of Google's user-triggered range files lists it, so we could not test it and left it out of the list.
Which AI bots are agents?
Two, both documented as agents that navigate the web for a user.
Google-Agent (Google)
- UA fragment: Google-Agent
- Proof: file (+ signature, being trialled)
- robots.txt, operator's words: user-triggered fetchers "generally ignore robots.txt rules" (page-level statement)
- Verified by GeoIPHub: verified
- Checked: 2026-11-09
Claude-Agent (Anthropic)
- UA fragment: Claude-Agent/1.0
- Proof: file (SAIL, TOML, not JSON)
- robots.txt, operator's words: "Your preferences for Anthropic's robots (crawlers and fetchers) are independent and you can manage them separately." (the agent page does not address robots.txt for the browser itself)
- Verified by GeoIPHub: not loaded
- Checked: 2026-11-09
Google-Agent is "used by agents hosted on Google infrastructure to navigate the web and perform actions upon user request". For Claude-Agent, Anthropic says to "match on the Claude-Agent token rather than the full string".
Which AI bots are none of these?
Three, each documented by its operator for a different job.
OAI-AdsBot (OpenAI)
- UA fragment: OAI-AdsBot/1.0
- Proof: file
- robots.txt, operator's words: not quoted (no statement found on OpenAI's page)
- Verified by GeoIPHub: not loaded
- Checked: 2026-11-09
CCBot (Common Crawl)
- UA fragment: CCBot/2.0
- Proof: file + reverse DNS
- robots.txt, operator's words: the page gives
User-agent: CCBotandDisallow: / - Verified by GeoIPHub: verified (category
archiver) - Checked: 2026-11-09
Google-CloudVertexBot (Google)
- UA fragment: Google-CloudVertexBot
- Proof: file (Google's common crawlers file)
- robots.txt, operator's words: not quoted
- Verified by GeoIPHub: verified (shared common-crawlers file)
- Checked: 2026-11-09
OAI-AdsBot is "used to validate the safety of web pages submitted as ads on ChatGPT". Google-CloudVertexBot "crawls requested by the site owners' for building Vertex AI Agents". Common Crawl says "we are aware of crawlers falsely identifying themselves as CCBot". Cloudflare's AI Crawl Control reference lists CCBot as an AI crawler; that link to training is a third party's, not Common Crawl's.
What blocking each job costs is covered in block AI crawlers.
User-triggered bots handle robots.txt differently, in their operators' own words
AI user-triggered bots such as ChatGPT-User, Perplexity-User and Claude-User do not share one robots.txt rule: each operator says something different, and one says nothing special at all. The fetch acts for one person, which is the reason most of them give.
- ChatGPT-User (OpenAI): "Because these actions are initiated by a user, robots.txt rules may not apply."
- Perplexity-User (Perplexity): "Since a user requested the fetch, this fetcher generally ignores robots.txt rules."
- user-triggered fetchers, including Google-Agent (Google): "Because the fetch was requested by a user, these fetchers generally ignore robots.txt rules."
- meta-externalfetcher (Meta): "may bypass robots.txt because it performs fetches that were requested by the user"
- Amzn-User (Amazon): "Because actions taken by Amzn-User can be initiated by a user, it may not follow all robots.txt directives."
- Kimi-User (Moonshot): "because actions are user-triggered, robots.txt rules may not directly apply"
- MistralAI-User (Mistral): "MistralAI-User governs which sites these user requests can be made to."
- Claude-User (Anthropic): "Anthropic's Bots respect "do not crawl" signals by honoring industry standard directives in robots.txt." and "Claude-User allows site owners to control which sites can be accessed through these user-initiated requests."
Anthropic is the outlier: its sentence has no user-triggered exception. "May not apply", "generally ignore", "may bypass" and "may not follow all" are four different strengths, so one rule for the whole group would misstate most of them.
Which AI crawlers publish IP ranges? 22 range files from 10 operators
Ten operators publish 22 range files for the AI bots we could find; GeoIPHub loads 17 of them. These AI crawler IP ranges come in four states. OpenAI, Perplexity, Mistral and Google's agent file give one JSON file per bot. Anthropic's three bots share one file, and Kimi's three files and DuckDuckGo's two files are byte-identical. Amazon's three bots have a web page only, each with its own creationTime. Meta and MistralAI-Training publish nothing, and we found no file from the operators in the last section. Claude-Agent publishes in a newer format, a TOML file at claude.com/.well-known/sail (draft-kelley-webbotauth-sail-00). ClaudeBot and PerplexityBot IP ranges traces a real lookup from each file.

GeoIPHub loads 19 official range files from 10 operators. 17 of them are in this list; the other two are duckduckbot.json, identical to the DuckDuckGo file below, and Ahrefs, which is not an AI bot. The 22 files are 20 distinct lists. Fetched on 9 November 2026 at 09:51 UTC, all with HTTP 200:
1. GPTBot
- File: gptbot.json
- IPv4 / IPv6 prefixes: 18 / 0
creationTime: 2026-09-22- Loaded: yes
2. OAI-SearchBot
- File: searchbot.json
- IPv4 / IPv6 prefixes: 39 / 0
creationTime: 2026-01-02- Loaded: yes
3. ChatGPT-User
- File: chatgpt-user.json
- IPv4 / IPv6 prefixes: 237 / 0
creationTime: 2026-11-06- Loaded: yes
4. OAI-AdsBot
- File: adsbot.json
- IPv4 / IPv6 prefixes: 2 / 0
creationTime: 2026-05-12- Loaded: no
5. ClaudeBot, Claude-User, Claude-SearchBot
- File: bots.json
- IPv4 / IPv6 prefixes: 14 / 0
creationTime: 2026-10-20- Loaded: yes
6. Claude-Agent
- File: sail (TOML)
- IPv4 / IPv6 prefixes: 9 / 0
creationTime: modified 2026-10-22- Loaded: no
7. Googlebot and common crawlers, including Google-CloudVertexBot
- File: common-crawlers.json
- IPv4 / IPv6 prefixes: 170 / 147
creationTime: 2026-11-06- Loaded: yes
8. Special-case crawlers
- File: special-crawlers.json
- IPv4 / IPv6 prefixes: 136 / 136
creationTime: 2026-11-06- Loaded: yes
9. User-triggered fetchers
- File: user-triggered-fetchers.json
- IPv4 / IPv6 prefixes: 544 / 544
creationTime: 2026-11-06- Loaded: yes
10. User-triggered fetchers (google.com)
- File: user-triggered-fetchers-google.json
- IPv4 / IPv6 prefixes: 252 / 252
creationTime: 2026-11-06- Loaded: yes
11. Google-Agent
- File: user-triggered-agents.json
- IPv4 / IPv6 prefixes: 12 / 8
creationTime: 2026-11-06- Loaded: yes
12. PerplexityBot
- File: perplexitybot.json
- IPv4 / IPv6 prefixes: 13 / 0
creationTime: 2026-10-28- Loaded: yes
13. Perplexity-User
- File: perplexity-user.json
- IPv4 / IPv6 prefixes: 10 / 0
creationTime: 2026-10-28- Loaded: yes
14. MistralAI-User
- File: mistralai-user-ips.json
- IPv4 / IPv6 prefixes: 4 / 0
creationTime: 2025-02-19- Loaded: yes
15. MistralAI-Index
- File: mistralai-index-ips.json
- IPv4 / IPv6 prefixes: 2 / 0
creationTime: 2026-04-19- Loaded: yes
16. Bingbot
- File: bingbot.json
- IPv4 / IPv6 prefixes: 29 / 0
creationTime: 2024-01-03- Loaded: yes
17. Applebot
- File: applebot.json
- IPv4 / IPv6 prefixes: 24 / 0
creationTime: 2026-09-15- Loaded: yes
18. DuckAssistBot (same list as duckduckbot.json)
- File: duckassistbot.json
- IPv4 / IPv6 prefixes: 486 / 0
creationTime: 2026-09-01- Loaded: yes
19. CCBot
- File: ccbot.json
- IPv4 / IPv6 prefixes: 4 / 1
creationTime: 2026-08-11- Loaded: yes
20. KimiBot
- File: kimibot.json
- IPv4 / IPv6 prefixes: 4 / 0
creationTime: 2026-05-18- Loaded: no
21. Kimi-User
- File: kimi-user.json
- IPv4 / IPv6 prefixes: 4 / 0
creationTime: 2026-05-18- Loaded: no
22. Kimi-SearchBot
- File: kimi-searchbot.json
- IPv4 / IPv6 prefixes: 4 / 0
creationTime: 2026-05-18- Loaded: no
A file's date does not prove the list is unchanged: Bingbot's file went from 28 to 29 prefixes while its date stayed 2024-01-03. We counted 28 in our post of 29 October and 29 on 9 November, and we have no earlier copy, so we cannot say which prefix was added. Google's special-case and user-triggered files cover non-AI fetchers too; Google lists its AI fetchers among them.
Amazon's three pages are not files. developer.amazon.com/amazonbot/ip-addresses/ (Amazonbot), .../searchbot-ip-addresses/ (Amzn-SearchBot) and .../live-ip-addresses/ (Amzn-User) carry embedded creationTime values of 2026-09-08, 2026-09-08 and 2025-11-04. We do not load them. DuckDuckGo's two files have the same MD5 hash, as do Kimi's three. Anthropic's one file covers three bots, and Anthropic says "If a crawler has a source IP address on this list, it indicates that the crawler is coming from Anthropic."
Six prefixes are identical in gptbot.json and searchbot.json on this date: 4.227.36.0/25, 74.7.175.128/25, 74.7.228.0/25, 74.7.230.0/25, 74.7.241.128/25 and 74.7.244.0/25. chatgpt-user.json shares none with them. Our verified crawler IP list shows how to build an allowlist from these files.
Which AI bots can GeoIPHub verify? 16 of 29 on 9 November
GeoIPHub's API verifies an address when it sits inside a file we load, and that proves the operator, not the bot. We looked up one real address from 14 of the 17 files we load (all but Google's special-case file and its two user-triggered-fetcher files), plus three contrast cases, keyless, about 14 seconds apart, on 9 November 2026 from 09:52 to 09:55 UTC. Every verified lookup also returned verification_method: "ip_range_list", scoring.recommended_action: "allow" and a fraud_score of 0.
OAI-SearchBot
- Address (from file): 4.227.36.1 (searchbot.json and gptbot.json)
verified: truename: OAISearchBotoperator: OpenAIcategory: ai- UTC: 09:52:11
GPTBot
- Address (from file): 132.196.86.1 (gptbot.json)
verified: truename: GPTBotoperator: OpenAIcategory: ai- UTC: 09:52:25
ChatGPT-User
- Address (from file): 104.208.184.193
verified: truename: ChatGPTUseroperator: OpenAIcategory: ai- UTC: 09:52:39
ClaudeBot, Claude-SearchBot, Claude-User
- Address (from file): 216.73.216.10 (bots.json)
verified: truename: ClaudeBotoperator: Anthropiccategory: ai- UTC: 09:52:52
PerplexityBot
- Address (from file): 107.20.236.150
verified: truename: PerplexityBotoperator: Perplexitycategory: ai- UTC: 09:53:06
Perplexity-User
- Address (from file): 18.97.43.80
verified: truename: PerplexityUseroperator: Perplexitycategory: ai- UTC: 09:53:20
MistralAI-User
- Address (from file): 20.240.160.161
verified: truename: MistralAIUseroperator: Mistralcategory: ai- UTC: 09:53:33
MistralAI-Index
- Address (from file): 135.225.57.92
verified: truename: MistralAIIndexoperator: Mistralcategory: ai- UTC: 09:53:47
Google-Agent
- Address (from file): 136.121.16.1
verified: truename: GoogleUserTriggeredAgentoperator: Googlecategory: ai- UTC: 09:54:01
Google-CloudVertexBot
- Address (from file): 192.178.4.1 (common-crawlers.json)
verified: truename: Googlebotoperator: Googlecategory: search_engine- UTC: 09:54:15
DuckAssistBot
- Address (from file): 104.43.54.127
verified: truename: DuckAssistBotoperator: DuckDuckGocategory: ai- UTC: 09:54:30
CCBot
- Address (from file): 18.97.9.169
verified: truename: CCBotoperator: CommonCrawlcategory: archiver- UTC: 09:54:46
Applebot
- Address (from file): 17.241.208.161
verified: truename: Applebotoperator: Applecategory: search_engine- UTC: 09:55:00
Bingbot
- Address (from file): 157.55.39.1
verified: truename: Bingbotoperator: Microsoftcategory: search_engine- UTC: 09:55:14
OAI-AdsBot (file not loaded)
- Address (from file): 130.131.229.129 (adsbot.json)
verified: false,is_crawler: falsename: noneoperator: nonecategory: none- UTC: 09:55:28
Amazonbot (page not loaded)
- Address (from file): 3.81.245.78 (Amazon's page)
verified: false,is_crawler: falsename: noneoperator: nonecategory: none- UTC: 09:55:42
Meta (network, no file)
- Address (from file): 31.13.64.1
verified: false,unverified: truename: Facebook-Botoperator: Metacategory: social_preview- UTC: 09:55:56
The traps are in the name field. name is the label of the file that loaded last, so Anthropic's three bots all return ClaudeBot, DuckDuckBot addresses return DuckAssistBot, and Google-CloudVertexBot returns Googlebot. category is not a job field: CCBot returns archiver, and Applebot and Bingbot return search_engine. Key on the claimed token plus operator, and write CommonCrawl without a space.
The surprise was 4.227.36.1. It sits inside 4.227.36.0/25, which both gptbot.json and searchbot.json list. A trimmed copy of the real response:
{
"threat": {
"is_crawler": true,
"crawler": {
"verified": true,
"unverified": false,
"name": "OAISearchBot",
"operator": "OpenAI",
"category": "ai",
"verification_method": "ip_range_list"
}
},
"scoring": { "fraud_score": 0, "recommended_action": "allow" },
"asn": { "asn": 8075 }
}
We are not saying a GPTBot request returns OAISearchBot. The address, which sits in both files, returns that name, and our lookup cannot tell which bot sent a request. OpenAI itself says it "may use the results from just one crawl for both use cases". The IP proves OpenAI and the User-Agent says which bot; the OpenAI post covers what each OpenAI bot does.
The not-loaded cases are plain. OAI-AdsBot's file is published and 130.131.229.129 is inside it, but our API returns is_crawler: false and a hosting answer (score 15, allow, AS8075 Microsoft). Amazonbot's own page lists 3.81.245.78 and the API returns the same kind of answer (score 16, allow, AS14618 Amazon). Nothing in our data separates a real OAI-AdsBot from any other address on that cloud. Meta's 31.13.64.1 is an address on Meta's network (AS32934), verified: false; we did not observe it crawling.
The API returns neither which file matched nor when it was fetched, and a lookup never sees your User-Agent. Field details are in the crawler fields, and you can paste an address and its user agent from your logs into the crawler IP verifier. It reads threat.crawler.verified and compares the operator with the bot your user agent names, in your browser.
Check an IP from your logs against the right file
This snippet takes a log line's IP and the User-Agent it claims, loads only the file of the bot named in the User-Agent, and reports whether the address is in it and which other files also list it. It makes no allow or block decision and writes no list. It is a sample, not a parser: a User-Agent with two bot tokens matches the longest.
import ipaddress, json, sys, urllib.request
# claude.com returns 403 to Python's default User-Agent, so send our own.
HEADERS = {"User-Agent": "ai-log-check/1.0 (+https://geoiphub.com)"}
FILES = { # claimed UA token (lowercase), mapped to that bot's own range file; add bots in the same JSON shape
"gptbot": "https://openai.com/gptbot.json",
"oai-searchbot": "https://openai.com/searchbot.json",
"chatgpt-user": "https://openai.com/chatgpt-user.json",
"oai-adsbot": "https://openai.com/adsbot.json",
"claudebot": "https://claude.com/crawling/bots.json",
"claude-searchbot": "https://claude.com/crawling/bots.json",
"claude-user": "https://claude.com/crawling/bots.json",
"perplexitybot": "https://www.perplexity.ai/perplexitybot.json",
"perplexity-user": "https://www.perplexity.ai/perplexity-user.json",
"mistralai-user": "https://mistral.ai/mistralai-user-ips.json",
"mistralai-index": "https://mistral.ai/mistralai-index-ips.json",
"duckassistbot": "https://duckduckgo.com/duckassistbot.json",
"ccbot": "https://index.commoncrawl.org/ccbot.json",
}
_cache = {}
def nets(url):
"""Prefixes of one file; None if it cannot be fetched or parsed (fail closed)."""
if url not in _cache:
try:
req = urllib.request.Request(url, headers=HEADERS)
data = json.load(urllib.request.urlopen(req, timeout=15))
_cache[url] = [ipaddress.ip_network(p.get("ipv4Prefix") or p["ipv6Prefix"])
for p in data["prefixes"]]
except Exception:
_cache[url] = None
return _cache[url]
def check(ip, ua):
ua = ua.lower()
# A UA with two bot tokens matches the longest; this is a sample, not a parser.
token = next((t for t in sorted(FILES, key=len, reverse=True) if t in ua), None)
if token is None:
return "no AI bot claimed", []
try:
addr = ipaddress.ip_address(ip)
except ValueError:
return f"not verified: {ip!r} is not an IP address", []
own = nets(FILES[token])
if own is None:
return f"not verified: {token} file unavailable", []
hits = sorted({u.rsplit("/", 1)[-1] for u in set(FILES.values())
if (n := nets(u)) and any(addr in x for x in n)})
own_name = FILES[token].rsplit("/", 1)[-1]
if own_name in hits:
return f"{token}: in {own_name}", [h for h in hits if h != own_name]
return f"not verified: {ip} is not in {own_name}", hits
if __name__ == "__main__":
for line in sys.stdin: # each line: <ip> <user agent>
ip, _, ua = line.strip().partition(" ")
verdict, also = check(ip, ua)
print(ip, "|", verdict, "| also in:", ", ".join(also) or "-")
Each log line is <ip> <User-Agent>. We ran it on 9 November 2026 at 10:01 UTC on this sample, where 45.33.32.156 is Nmap's public scan host (it sent no such request) and 203.0.113.9 and 2001:db8::1 are documentation addresses:
132.196.86.1 Mozilla/5.0 compatible; GPTBot/1.4
216.73.216.10 Mozilla/5.0 compatible; Claude-SearchBot
130.131.229.129 Mozilla/5.0 compatible; OAI-AdsBot/1.0
45.33.32.156 Mozilla/5.0 compatible; GPTBot/1.4
4.227.36.1 Mozilla/5.0 compatible; GPTBot/1.4
4.227.36.1 Mozilla/5.0 compatible; OAI-SearchBot/1.4
203.0.113.9 Mozilla/5.0 (Windows NT 10.0) Chrome/126.0
2001:db8::1 Mozilla/5.0 compatible; CCBot/2.0
132.196.86.1 | gptbot: in gptbot.json | also in: -
216.73.216.10 | claude-searchbot: in bots.json | also in: -
130.131.229.129 | oai-adsbot: in adsbot.json | also in: -
45.33.32.156 | not verified: 45.33.32.156 is not in gptbot.json | also in: -
4.227.36.1 | gptbot: in gptbot.json | also in: searchbot.json
4.227.36.1 | oai-searchbot: in searchbot.json | also in: gptbot.json
203.0.113.9 | no AI bot claimed | also in: -
2001:db8::1 | not verified: 2001:db8::1 is not in ccbot.json | also in: -
OAI-AdsBot passes here because the snippet reads the file itself, even though our API does not load it. One User-Agent token maps to one file, following Perplexity's rule: "Each bot has its own IP ranges, so match each User-Agent against its own IP ranges." An unreachable file means "not verified", never "verified": with SSL_CERT_FILE pointed at a path that does not exist, every line printed not verified: ... file unavailable. A bad IP string prints "not verified" rather than stopping the run. The snippet sets its own User-Agent because claude.com returned 403 to Python's default on 9 November 2026 while the other files returned 200.
For the bots with no file, the methods are in how to verify AI crawlers and agents: reverse DNS and signatures. Google says it "is also experimenting with the Web Bot Auth protocol", using the https://agent.bot.goog identity, and YouBot publishes a signature directory.
Names that never reach your logs, and bots we could not source
Some names on other lists are not crawlers, or have no operator page we could open. Three are control tokens that never appear as a requester, so none of them can be logged or verified by IP.
Google-Extended (Google)
- In your logs?: No
- Operator's words: "Google-Extended doesn't have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity."
- Honoured by: Google
Applebot-Extended (Apple)
- In your logs?: No
- Operator's words: "Applebot-Extended does not crawl webpages." and "Applebot-Extended is only used to determine how to use the data crawled by the Applebot user agent."
- Honoured by: Apple
__usage_ai-training (shared (IETF aipref draft))
- In your logs?: No, it is a robots.txt group name
- Operator's words: Microsoft: "Starting February 1, 2027, Microsoft will honor the new preferences as it crawls". Google: "intends to support the AIPREF shared machine-readable training preference signal (with "user-agent: __usage_ai-training") in early 2027". Anthropic: "As of October 30, 2026, Anthropic has begun recognizing a machine-readable signal"
- Honoured by: Anthropic from 2026-10-30 (its page does not name the token); Microsoft from 2027-02-01; Google "early 2027" (intent); OpenAI not verified
Anthropic's page says its signal rests on "a draft based on work done by the IETF ... in its aipref working group". We could not open OpenAI's help pages, so we make no claim about OpenAI.
On 9 November 2026 we could not find an operator-run page for the bots below. This list says what we checked. A bot missing here may still exist; we are not saying these operators run no crawler.
ByteDance: Bytespider
- What we checked on 2026-11-09: bytedance.com/robots.txt; bytedance.com/en/bytespider (404); bytespider.bytedance.com (no DNS); zhanzhang.toutiao.com home page (no crawler text)
- Result: No ByteDance page found. Third parties disagree on whether ranges exist, so we state no job or range
xAI: Grok-related tokens
- What we checked on 2026-11-09: docs.x.ai/llms.txt (no crawler or robots.txt text); docs.x.ai/docs/crawler (404); x.ai/grok-bot.json (403)
- Result: Nothing found
Cohere: cohere-ai, cohere-training-data-crawler
- What we checked on 2026-11-09: docs.cohere.com/docs/cohere-web-crawlers (opens; bot table reads "N/A"); cohere.com/crawler (404); cohere.com/cohere-ai.json (404)
- Result: The page says: "We do not use Cohere bots or user agents for the purpose of crawling or scraping web content to train generative AI foundation models at this time."
DeepSeek: DeepSeekBot
- What we checked on 2026-11-09: deepseek.com/robots.txt; api-docs.deepseek.com/llms.txt (no matches); deepseek.com/deepseekbot.json (404)
- Result: Nothing found
Some names are stale. anthropic-ai and Claude-Web are not on Anthropic's page, FacebookBot is not on Meta's page, and Gemini-Deep-Research appears on none of the four Google crawler pages we read. They are not documented on the pages we read, so we list none of them as bots.
How this list was checked, and how it changes
GeoIPHub checked this AI crawler list by opening each operator's page, fetching each file, recounting, and running the lookups above. An entry's date changes only when we re-check that entry.
Changes we saw, each with its date: Bingbot's file went from 28 to 29 prefixes under a frozen 2024-01-03 date. ChatGPT-User's file held 234 prefixes, then 235 on 29 October, then 237 in a file dated 6 November 2026. Perplexity's two files held 4 and 8 prefixes on 27 October and 10 and 13 on 28 October. Anthropic's file had 38 prefixes in our block AI crawlers post (file dated 7 October) and has 14, dated 20 October. We state no update schedule, for operators or for our API. The API does not return the fetch date, so a prefix an operator added recently may not verify yet.
The name in the header is a claim, and the proof is the operator's file when the operator publishes one.
Frequently Asked Questions
What is an AI crawler list?
A list of the automated clients that AI companies and search products run to fetch web pages, with each one's user agent, its job (training, search, user-triggered fetching or agent) and how it can be proved. On 9 November 2026 GeoIPHub's list has 29 bots that operators document, and GeoIPHub's API verifies 16 of them by IP.
What is the difference between GPTBot and ClaudeBot?
GPTBot is OpenAI's training crawler and ClaudeBot is Anthropic's. Each publishes a range file, and Anthropic's one file also covers Claude-User and Claude-SearchBot. GeoIPHub's API verifies an address from either file. What each OpenAI bot does is covered in GPTBot vs OAI-SearchBot vs ChatGPT-User.
Which AI crawlers publish IP ranges?
On 9 November 2026 ten operators publish 22 range files for AI bots we could find: OpenAI, Anthropic, Google, Perplexity, Mistral, Microsoft, Apple, DuckDuckGo, Common Crawl and Moonshot. GeoIPHub loads 17 of them. Amazon lists addresses on web pages, and Meta and MistralAI-Training publish none.
Can I verify an AI crawler by its user agent?
No. A user agent is a string any client can send. GeoIPHub's lookup takes only the IP: it can tell you the address is inside OpenAI's or Anthropic's file, and your own code compares that operator with the bot the User-Agent names.
What user agent does Meta's AI crawler use?
Meta documents meta-externalagent/1.1 for training, meta-webindexer/1.1 for Meta AI search and meta-externalfetcher/1.1 for user-requested fetches. Meta publishes no range file, so GeoIPHub matches its network by ASN and returns verified: false.
Is CCBot an AI crawler?
CCBot is Common Crawl's crawler, and Common Crawl publishes an IP file and reverse DNS for it. Other lists call it an AI crawler because the archive it builds can be used for training; that link is theirs, not Common Crawl's statement. GeoIPHub verifies its file and returns the category archiver.
The team behind the GeoIPHub API
Written by the GeoIPHub team, the people who build the GeoIPHub IP intelligence API and its scanner. Drafts are prepared with AI assistance and edited by the team; API examples are real lookups against the live API unless labelled illustrative.


