Registry · version 2026-10-24
The AI crawler registry
51 crawler tokens from 30 vendors, each with the purpose its operator documents. A robots.txt speaks about crawlers; a policy speaks about purposes. This is the table that connects them, and it is the one the free check, the permission API and the standard all read from.
Machine-readable
GET https://machinereadiness.com/crawlers.json returns the same table as JSON, with the purposes, under CC BY 4.0. Cache it for a day; new tokens appear a few times a year, not weekly.
Purposes
| Purpose | Meaning | Counted in the readiness score |
|---|---|---|
search | AI search index | Yes |
answers | AI answers | Yes |
user | Fetches when a user asks | Yes |
training | Training only | No |
research | Research crawl | No |
owner | Only for the site owner | No |
retired | Retired token | No |
Blocking a training-only crawler is a choice about training, not about being found, so the score does not count it. The permission API reads every purpose.
Tokens by vendor
| Vendor | Token | Purpose | Reference |
|---|---|---|---|
| OpenAI | GPTBot | Training only | What it does → |
OAI-SearchBotscored | AI search index | What it does → | |
ChatGPT-Userscored | Fetches when a user asks | What it does → | |
| Anthropic | ClaudeBot | Training only | What it does → |
Claude-Userscored | Fetches when a user asks | What it does → | |
Claude-SearchBotscored | AI search index | What it does → | |
Claude-Web | Retired token | What it does → | |
anthropic-ai | Retired token | What it does → | |
| Google AI | Google-Extendedscored | AI answers | What it does → |
GoogleOther | Research crawl | What it does → | |
Gemini-Deep-Researchscored | Fetches when a user asks | What it does → | |
Google-NotebookLMscored | Fetches when a user asks | What it does → | |
Google-CloudVertexBot | Only for the site owner | What it does → | |
GoogleOther-Image | Research crawl | What it does → | |
GoogleOther-Video | Research crawl | What it does → | |
| Perplexity | PerplexityBotscored | AI search index | What it does → |
Perplexity-Userscored | Fetches when a user asks | What it does → | |
| xAI / Grok | GrokBotscored | AI answers | What it does → |
Grok-DeepSearchscored | Fetches when a user asks | What it does → | |
xAI-Grokscored | AI answers | What it does → | |
| Microsoft Copilot | Copilotscored | AI answers | What it does → |
CopilotNativescored | AI answers | What it does → | |
| Meta AI | meta-externalagentscored | AI answers | What it does → |
meta-externalfetcherscored | Fetches when a user asks | What it does → | |
FacebookBot | Training only | What it does → | |
| Apple Intelligence | Applebot-Extended | Training only | What it does → |
| Amazon | Amazonbotscored | AI answers | What it does → |
| Huawei | PetalBotscored | AI search index | What it does → |
| Yandex AI | YandexAdditionalscored | AI answers | What it does → |
YandexAdditionalBotscored | AI answers | What it does → | |
| DeepSeek | DeepSeekBotscored | AI answers | What it does → |
| Mistral | MistralAI-Userscored | Fetches when a user asks | What it does → |
| Cohere | cohere-aiscored | AI answers | What it does → |
cohere-training-data-crawler | Training only | What it does → | |
| Common Crawl | CCBot | Training only | What it does → |
| DuckDuckGo AI | DuckAssistBotscored | AI search index | What it does → |
| Bytedance | Bytespider | Training only | What it does → |
| Diffbot | Diffbotscored | AI answers | What it does → |
| Allen AI | AI2Bot | Training only | What it does → |
AI2Bot-Dolma | Training only | What it does → | |
| Phind | PhindBotscored | AI search index | What it does → |
| You.com | YouBotscored | AI search index | What it does → |
| Kagi | Kagibotscored | AI search index | What it does → |
| Andi | AndiBotscored | AI search index | What it does → |
| Timpi | Timpibotscored | AI search index | What it does → |
| Sider | SiderAIscored | Fetches when a user asks | What it does → |
| HuggingFace | HuggingFace-Botscored | AI answers | What it does → |
| Quora / Poe | Quora-Botscored | AI answers | What it does → |
| Genspark | GensparkBotscored | AI search index | What it does → |
| Neeva | NeevaBot | Retired token | What it does → |
| Elicit | Elicit-Botscored | Fetches when a user asks | What it does → |
A token missing or misfiled? Write to support@crawlreadiness.com with the operator's documentation page. Vendor documentation is the source; community reports are marked as such on the reference pages.