AI crawler list: user agents and robots.txt tokens
Last checked 30 October 2026 against each operator's own documentation · 27 bots · JSONAI companies send bots to read websites. Some collect pages to train future models, some index pages so an assistant can cite them, and some fetch a page only when a person asks about it. This list covers every one whose operator documents it, with the robots.txt token you would use to allow or block it.
- Training crawler Collects pages that may be used to train AI models. Blocking it is a content-licensing decision, not a visibility one.
- AI search index Indexes pages so an assistant can find and cite them. Blocking it can remove you from that assistant's answers.
- User-triggered fetcher Visits one page because a person asked about it. Several operators say these may not follow robots.txt.
- Control token Not a crawler at all: a robots.txt name that tells an existing crawler how its data may be used.
The list
"Follows robots.txt" records what the operator itself says. "Opt-out documented" means the operator explains how to block the bot in robots.txt without promising more. Tokens are case-insensitive in robots.txt.
| robots.txt token | Operator | Type | Used for training | Follows robots.txt | What it does | Source |
|---|---|---|---|---|---|---|
GPTBot | OpenAI | Training crawler | Yes | Opt-out documented | Disallowing it tells OpenAI not to use the site's content for training. | Docs IP list |
OAI-SearchBot | OpenAI | AI search index | No | Opt-out documented | Surfaces sites in ChatGPT search. Blocked sites are not shown there; changes take about a day. | Docs IP list |
ChatGPT-User | OpenAI | User-triggered fetcher | No | Not always | Visits a page when a ChatGPT user's action needs it. OpenAI says robots.txt may not apply. | Docs IP list |
OAI-AdsBot | OpenAI | Other | Not stated | Not stated | Checks the safety of pages submitted as ads on ChatGPT, and only those pages. | Docs IP list |
ClaudeBot | Anthropic | Training crawler | Yes | Yes | Blocking it keeps future content out of Anthropic's training data. Crawl-delay is supported. | Docs IP list |
Claude-SearchBot | Anthropic | AI search index | No | Yes | Indexes pages to improve the quality of Claude's search results. | Docs IP list |
Claude-User | Anthropic | User-triggered fetcher | No | Yes | Fetches pages when a Claude user asks. Anthropic's robots.txt commitment covers all its bots. | Docs IP list |
PerplexityBot | Perplexity | AI search index | No | Opt-out documented | Surfaces and links sites in Perplexity answers; Perplexity says it is not used for model training. | Docs IP list |
Perplexity-User | Perplexity | User-triggered fetcher | No | Generally ignores | Fetches a page when a user asks. Perplexity says it generally ignores robots.txt. | Docs IP list |
Googlebot | AI search index | Not stated | Yes | Google Search's crawler. Google names Googlebot's robots.txt rules as the control for AI features in Search; use nosnippet or noindex to limit AI Overviews and AI Mode. | Docs IP list | |
Google-Extended | Control token (no crawler) | Yes | n/a (token only) | No crawler of its own. Decides whether content may train future Gemini models and ground Gemini apps. Google says it does not affect inclusion in Google Search. | Docs - | |
GoogleOther | Other | Not stated | Yes | Generic crawler for Google product teams and research. Its rules do not affect a specific product. | Docs IP list | |
Google-CloudVertexBot | User-triggered fetcher | Not stated | Yes | Crawls sites a site owner asks it to, to build Vertex AI agents. No effect on Google Search. | Docs IP list | |
bingbot | Microsoft | AI search index | May be | Yes | Bing's main crawler. Microsoft's AI controls are meta tags, not a robots.txt token: NOARCHIVE keeps a page out of Bing's AI answers and training (2023 guidance, written for Bing Chat). | Docs IP list |
Applebot | Apple | AI search index | May be | Yes | Powers Spotlight, Siri and Safari search. Its data may help train Apple models unless Applebot-Extended is disallowed. Falls back to Googlebot rules if none name Applebot. | Docs IP list |
Applebot-Extended | Apple | Control token (no crawler) | Yes | n/a (token only) | Does not crawl. Disallowing it stops Apple using Applebot's data to train its models; pages stay in Apple search results. | Docs - |
meta-externalagent | Meta | Training crawler | Yes | Opt-out documented | Used for training AI models and for indexing content to improve products. One token covers both. | Docs - |
meta-webindexer | Meta | AI search index | Not stated | Opt-out documented | Improves the quality of Meta AI search results. | Docs - |
meta-externalfetcher | Meta | User-triggered fetcher | Not stated | Not always | Fetches individual links at a user's request. Meta says it may bypass robots.txt. | Docs - |
Amazonbot | Amazon | Training crawler | May be | Yes | Improves Amazon products; its data may be used to train Amazon AI models. Amazon also honours a noarchive meta tag to exclude a page from training. | Docs IP list |
Amzn-SearchBot | Amazon | AI search index | No | Yes | Improves Amazon search experiences; Amazon says it does not crawl for AI training. | Docs IP list |
Amzn-User | Amazon | User-triggered fetcher | No | Not always | Fetches live information to answer Alexa questions. Amazon says it may not follow all robots.txt rules. | Docs IP list |
CCBot | Common Crawl | Other | Not stated | Yes | Builds Common Crawl's open web archive, which anyone can reuse. Supports Crawl-delay. | Docs IP list |
MistralAI-Training | Mistral AI | Training crawler | Yes | Opt-out documented | Builds datasets for training Mistral's models. | Docs - |
MistralAI-Index | Mistral AI | AI search index | No | Opt-out documented | Indexing only; Mistral says it is not used for training. | Docs IP list |
MistralAI-User | Mistral AI | User-triggered fetcher | No | Opt-out documented | Fetches pages for user actions in Mistral's assistant. | Docs IP list |
DuckAssistBot | DuckDuckGo | User-triggered fetcher | No | Opt-out documented | Fetches pages in real time for DuckDuckGo's AI-assisted answers. A disallow takes effect after 72 hours. | Docs IP list |
Bots without official documentation
These appear in third-party bot lists, but we found no page from the operator describing them, so we do not list a token or a purpose for them.
- ByteDance (Bytespider): No official documentation found. Third-party lists disagree on its purpose and on whether it follows robots.txt.
- xAI (Grok): No official crawler documentation found. Third-party lists name several different tokens.
- DeepSeek: No official crawler documentation found.
- FacebookBot: No longer listed on Meta's crawler page, which now documents the meta-* bots above.
- Cohere: Its documentation says Cohere does not use bots to crawl or scrape web content.
robots.txt examples
Opt out of AI training, stay visible in AI answers
Blocks the documented training crawlers and control tokens, and leaves search crawlers and AI search indexes alone. Meta's token also covers product indexing, so blocking it does both.
User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: meta-externalagent Disallow: / User-agent: Amazonbot Disallow: / User-agent: MistralAI-Training Disallow: /
Block every documented AI bot
This keeps your pages out of AI search indexes too, so assistants such as ChatGPT search and Perplexity may stop citing you. It leaves Googlebot, bingbot and Applebot alone, because blocking those removes you from regular search. User-triggered fetchers may still visit, as their operators say.
User-agent: GPTBot Disallow: / User-agent: OAI-SearchBot Disallow: / User-agent: ChatGPT-User Disallow: / User-agent: OAI-AdsBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Claude-SearchBot Disallow: / User-agent: Claude-User Disallow: / User-agent: PerplexityBot Disallow: / User-agent: Perplexity-User Disallow: / User-agent: Google-Extended Disallow: / User-agent: Google-CloudVertexBot Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: meta-externalagent Disallow: / User-agent: meta-webindexer Disallow: / User-agent: meta-externalfetcher Disallow: / User-agent: Amazonbot Disallow: / User-agent: Amzn-SearchBot Disallow: / User-agent: Amzn-User Disallow: / User-agent: CCBot Disallow: / User-agent: MistralAI-Training Disallow: / User-agent: MistralAI-Index Disallow: / User-agent: MistralAI-User Disallow: / User-agent: DuckAssistBot Disallow: /
robots.txt is only one layer. A CDN or firewall rule can turn a bot away even when robots.txt allows it, so check what each crawler actually gets.
Can AI crawlers reach your site?
The free AI crawler checker reads your robots.txt for each bot on this list and requests your site as that bot, so you can see what is really allowed through.
Check your site freeHow we keep this list
Every row comes from the operator's own documentation page, linked in the Source column, and was re-read on 30 October 2026. We paraphrase rather than quote, and we leave a bot out rather than copy it from another list. Full user-agent strings change with every version, so match on the token. If an operator changes its documentation before our next check, email [email protected] and we will update it.
The data is free to reuse under CC BY 4.0. The same list is available as JSON. To cite it:
Stellarcast. "AI crawler list: user agents and robots.txt tokens." stellarcast.ai/ai-crawlers/, checked 30 October 2026.
Frequently asked questions
What is the difference between a training crawler and an AI search crawler?
A training crawler collects pages that may be used to train future AI models. An AI search crawler indexes pages so an assistant can find and cite them when it answers. Blocking a training crawler keeps your content out of future models; blocking a search crawler can keep your pages out of that assistant's answers.
Does blocking Google-Extended remove my site from AI Overviews or AI Mode?
Google says Google-Extended does not affect a site's inclusion in Google Search, and names Googlebot's robots.txt rules as the control for AI features in Search. To limit how a page appears in AI Overviews or AI Mode, Google points to nosnippet, max-snippet and noindex instead.
Do AI crawlers obey robots.txt?
It depends on the bot. Several operators say their bots follow robots.txt, but OpenAI says its user-triggered ChatGPT-User may not apply it, Perplexity says Perplexity-User generally ignores it, and Meta and Amazon say their user-triggered fetchers may bypass some rules. The robots.txt column in the table records what each operator says.
How can I check which AI crawlers can reach my site?
Run the free Stellarcast AI crawler checker. It reads your robots.txt for each crawler and requests your site as that crawler, because a firewall or CDN rule can block a bot that robots.txt allows.
How often is this list updated?
We re-check every operator's own documentation at least monthly. This version was checked on 30 October 2026. Bots that only appear in third-party lists are not added.