Machine Readiness

Registry · version 2026-10-28

The AI crawler registry

60 crawler tokens from 30 vendors, each with its purpose: the operator's own description where one exists, and marked where only community lists name the token. A robots.txt speaks about crawlers; a policy speaks about purposes. This is the table that connects them, and it is the one the free check, the permission API and the standard all read from.

Machine-readable

GET https://machinereadiness.com/crawlers.json returns the same table as JSON, with the purposes, under CC BY 4.0. Cache it for a day; new tokens appear a few times a year, not weekly.

Purposes

PurposeMeaningCounted in the readiness score
searchAI search indexYes
answersAI answersYes
userFetches when a user asksYes
trainingTraining onlyNo
researchResearch crawlNo
ownerOnly for the site ownerNo
adsAdvertisingNo
retiredRetired tokenNo

Blocking a training-only crawler is a choice about training, not about being found, so the score does not count it. The permission API reads every purpose.

Tokens by vendor

VendorTokenPurposeReference
OpenAIGPTBotTraining onlyWhat it does →
OAI-SearchBotscoredAI search indexWhat it does →
ChatGPT-UserscoredFetches when a user asksWhat it does →
OAI-AdsBotOnly for the site ownerWhat it does →
AnthropicClaudeBotTraining onlyWhat it does →
Claude-UserscoredFetches when a user asksWhat it does →
Claude-SearchBotscoredAI search indexWhat it does →
Claude-WebretiredRetired tokenWhat it does →
anthropic-airetiredRetired tokenWhat it does →
Google AIGoogle-ExtendedscoredAI answersWhat it does →
GoogleOtherResearch crawlWhat it does →
Gemini-Deep-Researchscoredcommunity listFetches when a user asksWhat it does →
Google-GeminiNotebookscoredFetches when a user asksWhat it does →
Google-AgentscoredFetches when a user asksWhat it does →
Google-NotebookLMretiredRetired tokenWhat it does →
Google-CloudVertexBotOnly for the site ownerWhat it does →
GoogleOther-ImageResearch crawlWhat it does →
GoogleOther-VideoResearch crawlWhat it does →
PerplexityPerplexityBotscoredAI search indexWhat it does →
Perplexity-UserscoredFetches when a user asksWhat it does →
xAI / GrokGrokBotscoredcommunity listAI answersWhat it does →
Grok-DeepSearchscoredcommunity listFetches when a user asksWhat it does →
xAI-Grokscoredcommunity listAI answersWhat it does →
Microsoft CopilotCopilotscoredcommunity listAI answersWhat it does →
CopilotNativescoredcommunity listAI answersWhat it does →
Meta AImeta-externalagentscoredAI answersWhat it does →
meta-webindexerscoredAI search indexWhat it does →
meta-externalfetcherscoredFetches when a user asksWhat it does →
meta-externaladsAdvertisingWhat it does →
FacebookBotretiredRetired tokenWhat it does →
Apple IntelligenceApplebot-ExtendedTraining onlyWhat it does →
AmazonAmazonbotscoredAI answersWhat it does →
Amzn-SearchBotscoredAI search indexWhat it does →
Amzn-UserscoredFetches when a user asksWhat it does →
HuaweiPetalBotscoredAI search indexWhat it does →
Yandex AIYandexAdditionalscoredAI answersWhat it does →
YandexAdditionalBotscoredAI answersWhat it does →
DeepSeekDeepSeekBotscoredcommunity listAI answersWhat it does →
MistralMistralAI-UserscoredFetches when a user asksWhat it does →
MistralAI-IndexscoredAI search indexWhat it does →
MistralAI-TrainingTraining onlyWhat it does →
Coherecohere-aiscoredcommunity listAI answersWhat it does →
cohere-training-data-crawlercommunity listTraining onlyWhat it does →
Common CrawlCCBotTraining onlyWhat it does →
DuckDuckGo AIDuckAssistBotscoredFetches when a user asksWhat it does →
BytedanceBytespidercommunity listTraining onlyWhat it does →
DiffbotDiffbotscoredAI answersWhat it does →
Allen AIAI2BotTraining onlyWhat it does →
AI2Bot-Dolmacommunity listTraining onlyWhat it does →
PhindPhindBotscoredcommunity listAI search indexWhat it does →
You.comYouBotscoredAI search indexWhat it does →
KagiKagibotscoredAI search indexWhat it does →
AndiAndiBotscoredcommunity listAI search indexWhat it does →
TimpiTimpibotscoredcommunity listAI search indexWhat it does →
SiderSiderAIscoredcommunity listFetches when a user asksWhat it does →
HuggingFaceHuggingFace-Botscoredcommunity listAI answersWhat it does →
Quora / PoeQuora-Botscoredcommunity listAI answersWhat it does →
GensparkGensparkBotscoredcommunity listAI search indexWhat it does →
NeevaNeevaBotretiredRetired tokenWhat it does →
ElicitElicit-Botscoredcommunity listFetches when a user asksWhat it does →

A token missing or misfiled? Write to support@crawlreadiness.com with the operator's documentation page. Vendor documentation is the source; community-listed tokens are marked here and on the reference pages.