Registry · version 2026-10-28

# The AI crawler registry

Every AI crawler token the Machine Readiness standard knows, by vendor and purpose: search index, AI answers, training, user-requested fetches. Human-readable here, machine-readable at /crawlers.json under CC BY 4.0.

60 crawler tokens from 30 vendors, each with its purpose: the operator's own description where one exists, and marked where only community lists name the token. A robots.txt speaks about crawlers; a policy speaks about purposes. This is the table that connects them, and it is the one the free check, the permission API and the standard all read from.

### Machine-readable

`GET https://machinereadiness.com/crawlers.json` returns the same table as JSON, with the purposes, under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/). Cache it for a day; new tokens appear a few times a year, not weekly.

## Purposes

| Purpose | Meaning | Counted in the readiness score |
| --- | --- | --- |
| `search` | AI search index | Yes |
| `answers` | AI answers | Yes |
| `user` | Fetches when a user asks | Yes |
| `training` | Training only | No |
| `research` | Research crawl | No |
| `owner` | Only for the site owner | No |
| `ads` | Advertising | No |
| `retired` | Retired token | No |

Blocking a training-only crawler is a choice about training, not about being found, so the score does not count it. The permission API reads every purpose.

## Tokens by vendor

| Vendor | Token | Purpose | Reference |
| --- | --- | --- | --- |
| OpenAI | [`GPTBot`](https://www.crawlreadiness.com/crawlers/gptbot) | Training only | [What it does →](https://www.crawlreadiness.com/crawlers/gptbot) |
|   | [`OAI-SearchBot`](https://www.crawlreadiness.com/crawlers/oai-searchbot) scored | AI search index | [What it does →](https://www.crawlreadiness.com/crawlers/oai-searchbot) |
|   | [`ChatGPT-User`](https://www.crawlreadiness.com/crawlers/chatgpt-user) scored | Fetches when a user asks | [What it does →](https://www.crawlreadiness.com/crawlers/chatgpt-user) |
|   | [`OAI-AdsBot`](https://www.crawlreadiness.com/crawlers/oai-adsbot) | Only for the site owner | [What it does →](https://www.crawlreadiness.com/crawlers/oai-adsbot) |
| Anthropic | [`ClaudeBot`](https://www.crawlreadiness.com/crawlers/claudebot) | Training only | [What it does →](https://www.crawlreadiness.com/crawlers/claudebot) |
|   | [`Claude-User`](https://www.crawlreadiness.com/crawlers/claude-user) scored | Fetches when a user asks | [What it does →](https://www.crawlreadiness.com/crawlers/claude-user) |
|   | [`Claude-SearchBot`](https://www.crawlreadiness.com/crawlers/claude-searchbot) scored | AI search index | [What it does →](https://www.crawlreadiness.com/crawlers/claude-searchbot) |
|   | [`Claude-Web`](https://www.crawlreadiness.com/crawlers/claude-web) retired | Retired token | [What it does →](https://www.crawlreadiness.com/crawlers/claude-web) |
|   | [`anthropic-ai`](https://www.crawlreadiness.com/crawlers/anthropic-ai) retired | Retired token | [What it does →](https://www.crawlreadiness.com/crawlers/anthropic-ai) |
| Google AI | [`Google-Extended`](https://www.crawlreadiness.com/crawlers/google-extended) scored | AI answers | [What it does →](https://www.crawlreadiness.com/crawlers/google-extended) |
|   | [`GoogleOther`](https://www.crawlreadiness.com/crawlers/googleother) | Research crawl | [What it does →](https://www.crawlreadiness.com/crawlers/googleother) |
|   | [`Gemini-Deep-Research`](https://www.crawlreadiness.com/crawlers/gemini-deep-research) scored community list | Fetches when a user asks | [What it does →](https://www.crawlreadiness.com/crawlers/gemini-deep-research) |
|   | [`Google-GeminiNotebook`](https://www.crawlreadiness.com/crawlers/google-gemininotebook) scored | Fetches when a user asks | [What it does →](https://www.crawlreadiness.com/crawlers/google-gemininotebook) |
|   | [`Google-Agent`](https://www.crawlreadiness.com/crawlers/google-agent) scored | Fetches when a user asks | [What it does →](https://www.crawlreadiness.com/crawlers/google-agent) |
|   | [`Google-NotebookLM`](https://www.crawlreadiness.com/crawlers/google-notebooklm) retired | Retired token | [What it does →](https://www.crawlreadiness.com/crawlers/google-notebooklm) |
|   | [`Google-CloudVertexBot`](https://www.crawlreadiness.com/crawlers/google-cloudvertexbot) | Only for the site owner | [What it does →](https://www.crawlreadiness.com/crawlers/google-cloudvertexbot) |
|   | [`GoogleOther-Image`](https://www.crawlreadiness.com/crawlers/googleother-image) | Research crawl | [What it does →](https://www.crawlreadiness.com/crawlers/googleother-image) |
|   | [`GoogleOther-Video`](https://www.crawlreadiness.com/crawlers/googleother-video) | Research crawl | [What it does →](https://www.crawlreadiness.com/crawlers/googleother-video) |
| Perplexity | [`PerplexityBot`](https://www.crawlreadiness.com/crawlers/perplexitybot) scored | AI search index | [What it does →](https://www.crawlreadiness.com/crawlers/perplexitybot) |
|   | [`Perplexity-User`](https://www.crawlreadiness.com/crawlers/perplexity-user) scored | Fetches when a user asks | [What it does →](https://www.crawlreadiness.com/crawlers/perplexity-user) |
| xAI / Grok | [`GrokBot`](https://www.crawlreadiness.com/crawlers/grokbot) scored community list | AI answers | [What it does →](https://www.crawlreadiness.com/crawlers/grokbot) |
|   | [`Grok-DeepSearch`](https://www.crawlreadiness.com/crawlers/grok-deepsearch) scored community list | Fetches when a user asks | [What it does →](https://www.crawlreadiness.com/crawlers/grok-deepsearch) |
|   | [`xAI-Grok`](https://www.crawlreadiness.com/crawlers/xai-grok) scored community list | AI answers | [What it does →](https://www.crawlreadiness.com/crawlers/xai-grok) |
| Microsoft Copilot | [`Copilot`](https://www.crawlreadiness.com/crawlers/copilot) scored community list | AI answers | [What it does →](https://www.crawlreadiness.com/crawlers/copilot) |
|   | [`CopilotNative`](https://www.crawlreadiness.com/crawlers/copilotnative) scored community list | AI answers | [What it does →](https://www.crawlreadiness.com/crawlers/copilotnative) |
| Meta AI | [`meta-externalagent`](https://www.crawlreadiness.com/crawlers/meta-externalagent) scored | AI answers | [What it does →](https://www.crawlreadiness.com/crawlers/meta-externalagent) |
|   | [`meta-webindexer`](https://www.crawlreadiness.com/crawlers/meta-webindexer) scored | AI search index | [What it does →](https://www.crawlreadiness.com/crawlers/meta-webindexer) |
|   | [`meta-externalfetcher`](https://www.crawlreadiness.com/crawlers/meta-externalfetcher) scored | Fetches when a user asks | [What it does →](https://www.crawlreadiness.com/crawlers/meta-externalfetcher) |
|   | [`meta-externalads`](https://www.crawlreadiness.com/crawlers/meta-externalads) | Advertising | [What it does →](https://www.crawlreadiness.com/crawlers/meta-externalads) |
|   | [`FacebookBot`](https://www.crawlreadiness.com/crawlers/facebookbot) retired | Retired token | [What it does →](https://www.crawlreadiness.com/crawlers/facebookbot) |
| Apple Intelligence | [`Applebot-Extended`](https://www.crawlreadiness.com/crawlers/applebot-extended) | Training only | [What it does →](https://www.crawlreadiness.com/crawlers/applebot-extended) |
| Amazon | [`Amazonbot`](https://www.crawlreadiness.com/crawlers/amazonbot) scored | AI answers | [What it does →](https://www.crawlreadiness.com/crawlers/amazonbot) |
|   | [`Amzn-SearchBot`](https://www.crawlreadiness.com/crawlers/amzn-searchbot) scored | AI search index | [What it does →](https://www.crawlreadiness.com/crawlers/amzn-searchbot) |
|   | [`Amzn-User`](https://www.crawlreadiness.com/crawlers/amzn-user) scored | Fetches when a user asks | [What it does →](https://www.crawlreadiness.com/crawlers/amzn-user) |
| Huawei | [`PetalBot`](https://www.crawlreadiness.com/crawlers/petalbot) scored | AI search index | [What it does →](https://www.crawlreadiness.com/crawlers/petalbot) |
| Yandex AI | [`YandexAdditional`](https://www.crawlreadiness.com/crawlers/yandexadditional) scored | AI answers | [What it does →](https://www.crawlreadiness.com/crawlers/yandexadditional) |
|   | [`YandexAdditionalBot`](https://www.crawlreadiness.com/crawlers/yandexadditionalbot) scored | AI answers | [What it does →](https://www.crawlreadiness.com/crawlers/yandexadditionalbot) |
| DeepSeek | [`DeepSeekBot`](https://www.crawlreadiness.com/crawlers/deepseekbot) scored community list | AI answers | [What it does →](https://www.crawlreadiness.com/crawlers/deepseekbot) |
| Mistral | [`MistralAI-User`](https://www.crawlreadiness.com/crawlers/mistralai-user) scored | Fetches when a user asks | [What it does →](https://www.crawlreadiness.com/crawlers/mistralai-user) |
|   | [`MistralAI-Index`](https://www.crawlreadiness.com/crawlers/mistralai-index) scored | AI search index | [What it does →](https://www.crawlreadiness.com/crawlers/mistralai-index) |
|   | [`MistralAI-Training`](https://www.crawlreadiness.com/crawlers/mistralai-training) | Training only | [What it does →](https://www.crawlreadiness.com/crawlers/mistralai-training) |
| Cohere | [`cohere-ai`](https://www.crawlreadiness.com/crawlers/cohere-ai) scored community list | AI answers | [What it does →](https://www.crawlreadiness.com/crawlers/cohere-ai) |
|   | [`cohere-training-data-crawler`](https://www.crawlreadiness.com/crawlers/cohere-training-data-crawler) community list | Training only | [What it does →](https://www.crawlreadiness.com/crawlers/cohere-training-data-crawler) |
| Common Crawl | [`CCBot`](https://www.crawlreadiness.com/crawlers/ccbot) | Training only | [What it does →](https://www.crawlreadiness.com/crawlers/ccbot) |
| DuckDuckGo AI | [`DuckAssistBot`](https://www.crawlreadiness.com/crawlers/duckassistbot) scored | Fetches when a user asks | [What it does →](https://www.crawlreadiness.com/crawlers/duckassistbot) |
| Bytedance | [`Bytespider`](https://www.crawlreadiness.com/crawlers/bytespider) community list | Training only | [What it does →](https://www.crawlreadiness.com/crawlers/bytespider) |
| Diffbot | [`Diffbot`](https://www.crawlreadiness.com/crawlers/diffbot) scored | AI answers | [What it does →](https://www.crawlreadiness.com/crawlers/diffbot) |
| Allen AI | [`AI2Bot`](https://www.crawlreadiness.com/crawlers/ai2bot) | Training only | [What it does →](https://www.crawlreadiness.com/crawlers/ai2bot) |
|   | [`AI2Bot-Dolma`](https://www.crawlreadiness.com/crawlers/ai2bot-dolma) community list | Training only | [What it does →](https://www.crawlreadiness.com/crawlers/ai2bot-dolma) |
| Phind | [`PhindBot`](https://www.crawlreadiness.com/crawlers/phindbot) scored community list | AI search index | [What it does →](https://www.crawlreadiness.com/crawlers/phindbot) |
| You.com | [`YouBot`](https://www.crawlreadiness.com/crawlers/youbot) scored | AI search index | [What it does →](https://www.crawlreadiness.com/crawlers/youbot) |
| Kagi | [`Kagibot`](https://www.crawlreadiness.com/crawlers/kagibot) scored | AI search index | [What it does →](https://www.crawlreadiness.com/crawlers/kagibot) |
| Andi | [`AndiBot`](https://www.crawlreadiness.com/crawlers/andibot) scored community list | AI search index | [What it does →](https://www.crawlreadiness.com/crawlers/andibot) |
| Timpi | [`Timpibot`](https://www.crawlreadiness.com/crawlers/timpibot) scored community list | AI search index | [What it does →](https://www.crawlreadiness.com/crawlers/timpibot) |
| Sider | [`SiderAI`](https://www.crawlreadiness.com/crawlers/siderai) scored community list | Fetches when a user asks | [What it does →](https://www.crawlreadiness.com/crawlers/siderai) |
| HuggingFace | [`HuggingFace-Bot`](https://www.crawlreadiness.com/crawlers/huggingface-bot) scored community list | AI answers | [What it does →](https://www.crawlreadiness.com/crawlers/huggingface-bot) |
| Quora / Poe | [`Quora-Bot`](https://www.crawlreadiness.com/crawlers/quora-bot) scored community list | AI answers | [What it does →](https://www.crawlreadiness.com/crawlers/quora-bot) |
| Genspark | [`GensparkBot`](https://www.crawlreadiness.com/crawlers/gensparkbot) scored community list | AI search index | [What it does →](https://www.crawlreadiness.com/crawlers/gensparkbot) |
| Neeva | [`NeevaBot`](https://www.crawlreadiness.com/crawlers/neevabot) retired | Retired token | [What it does →](https://www.crawlreadiness.com/crawlers/neevabot) |
| Elicit | [`Elicit-Bot`](https://www.crawlreadiness.com/crawlers/elicit-bot) scored community list | Fetches when a user asks | [What it does →](https://www.crawlreadiness.com/crawlers/elicit-bot) |

A token missing or misfiled? Write to [support@crawlreadiness.com](mailto:support@crawlreadiness.com) with the operator's documentation page. Vendor documentation is the source; community-listed tokens are marked here and on the reference pages.

---

Source: https://machinereadiness.com/crawlers
