Machine Readiness

For RAG, agents, scrapers and dataset builders

May I use this page for this?

A site can reserve its rights in seven places that do not agree with each other. Most crawlers read robots.txt at best. This call reads all of them for one page and one purpose, says what they add up to, and hands back a dated receipt you can keep, with a hash of every file it read.

Free during the first weeks, 25 calls a day per network without a key. A free API key from the Crawl Readiness dashboard, sent as the x-api-key header, lifts that limit.

The request

curl "https://machinereadiness.com/api/permission?url=https://www.nytimes.com/&purpose=train"

GET or POST (JSON body) to /api/permission on this host or on www.crawlreadiness.com, with:

ParameterMeaning
urlThe page, with its path. The site's robots.txt, ai.txt and TDM file are read from its origin, and from the origin it redirects to.
purposesearch · answer · train · agent, as below.
agentOptional. Your own crawler's token, such as MyBot: robots.txt is then read for that crawler's rule. Naming another company's crawler (GPTBot, Googlebot) cannot loosen the answer: the verdict is then the stricter of its rule and the reading for the crawlers that do the job.
observeOptional, off by default. 1 also requests the page with a representative crawler's user-agent string; see below for what that can and cannot show.

The purposes, and the crawlers read for each

robots.txt speaks about crawlers, not purposes, so a purpose is read as the site's rules for the crawlers that do that job. A block on any one of them reserves the use.

PurposeMeaningCrawlers read
searchIndex the page and show links and excerpts in AI search results.OAI-SearchBot, Claude-SearchBot, PerplexityBot, YouBot, meta-webindexer, MistralAI-Index, Amzn-SearchBot
answerUse the page as input to a generated answer or summary, retrieval for RAG included.OAI-SearchBot, PerplexityBot, Claude-SearchBot, DuckAssistBot, Google-Extended, meta-externalagent, meta-webindexer
trainTrain or fine-tune a model on the page.GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended, Bytespider, meta-externalagent, Amazonbot, MistralAI-Training
agentFetch or act on the page on a user's behalf, live.ChatGPT-User, Claude-User, Perplexity-User, meta-externalfetcher, Google-Agent, Google-GeminiNotebook, MistralAI-User, Amzn-User

Fetchers that run when a person asks (ChatGPT-User, Perplexity-User, Google's user-triggered fetchers, Meta-ExternalFetcher, Amzn-User) are described by their own vendors as able to skip robots.txt, because a person requested the page. A verdict for the agent purpose says what the site published, not what those fetchers will do.

What it reads

  • robots.txt, for those crawlers, for a crawler it does not name through *, or for your own token. Its status is read as RFC 9309 says, with two cautious exceptions: a 404 or 410 means no file, so nothing reserved; a server error or no answer means a full disallow, as the RFC says, and so does a 429, which the RFC files with the other 4xx answers but Google reads as a server error. A 401 or 403 is reported as unknown, where RFC 9309 would let a crawler go ahead, because a refusal like that is usually a firewall, not a policy. Rules sent with another 4xx (Stack Overflow answers with 418) are read, and the answer says so.
  • Content-Signal, Cloudflare's line in robots.txt: search, ai-input, ai-train.
  • Content-Usage, the IETF AI-preferences vocabulary (train-ai, ai-use, search) as a robots.txt directive per group and path, and as an HTTP header.
  • RSL 1.0, found through a License: line in robots.txt, a Link header, a <link rel="license"> or an inline script. A License: line after the Sitemap: lines or a blank line covers the whole site; one inside a group covers that group's crawlers. Permits, prohibits and payment terms are read from the most specific <content> entry; a licence server on it is a condition, since RSL has clients obtain a licence there first. For answer, a prohibition of ai-index counts as well as ai-input.
  • The TDM reservation (W3C TDMRep), the EU text-and-data-mining opt-out: a tdm-reservation meta tag, then the header, then /.well-known/tdmrep.json, where the first matching rule applies. It decides train; for the other purposes it is reported, not counted, because whether they are text and data mining is a legal question.
  • ai.txt, about training.
  • The page's own directives in a meta tag or X-Robots-Tag, for every crawler or for one that does the job: noai (training, answers, agents), noindex (search), nosnippet and max-snippet:0 (no answers; search without excerpts), and Bing's noarchive and nocache (no Copilot answers).

When the page redirects, the host it ends on is read for its path, and the first host's robots.txt still counts for the first request. Every read is made as CrawlReadinessBot/1.0 (+https://www.crawlreadiness.com), bounded in size and time; the whole check stops reading at 20 seconds.

observe=1 also requests the page once with a representative crawler's user-agent string (GPTBot for train, PerplexityBot for answer, OAI-SearchBot for search, ChatGPT-User for agent), from our servers, and reports the status beside our own request as the control. It shows how the server treats that string, not what the real crawler gets: a site that checks a crawler's IP address or reverse DNS can tell the difference. It is skipped when robots.txt disallows that crawler, and it never changes the verdict.

What comes back

A verdict, the reason in plain words, one entry per signal with what was found and what it decided, the page's status and finalUrl, observed when asked for, and the receipt. The strongest decision among the signals sets the verdict, in this order:

VerdictMeaning
deniedA machine-readable signal reserves this use: any reservation that covers the purpose wins, a block on any one crawler that does the job included. Also when robots.txt answers a server error or does not answer, which RFC 9309 treats as a full disallow, or answers a 429, which Google reads the same way.
allowed-under-termsPermitted on conditions: an RSL licence's payment, user or region limits, or a licence server to obtain it from first (all in terms); or, for search, a nosnippet directive (no excerpts).
unknownA source that could carry a reservation could not be read: robots.txt answered 401 or 403, the page refused us or did not answer, a referenced licence could not be fetched or does not follow RSL, or, for training, tdmrep.json or ai.txt failed. Unknown outranks allowed: what could be read cannot speak for what could not.
allowedNo signal reserves the use. That is all it means: the absence of a signal is not a licence.
{
  "version": 2,
  "url": "https://www.nytimes.com/",
  "finalUrl": "https://www.nytimes.com/",
  "path": "/",
  "purpose": "train",
  "purposeLabel": "AI training",
  "agent": null,
  "verdict": "denied",
  "reason": "robots.txt blocks 7 of the 9 crawlers that do this: GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended, Bytespider, meta-externalagent. Any reservation wins; it allows Amazonbot, MistralAI-Training.",
  "terms": [],
  "signals": [
    { "source": "robots.txt", "found": true, "decision": "deny", "status": 200, "detail": "…",
      "crawlers": [ { "token": "GPTBot", "allowed": false }, … ] },
    { "source": "Content-Signal", "found": false, "decision": "silent", "detail": "No Content-Signal line." },
    …
    { "source": "Page directives", "found": false, "decision": "unknown",
      "detail": "The page answered HTTP 403, so its own directives (noai, noindex, nosnippet) could not be read." },
    …
  ],
  "note": "A reading of the signals the site publishes, not legal advice. …",
  "agentKnown": null,
  "page": { "status": 403, "fetched": false, "finalUrl": "https://www.nytimes.com/", "error": null },
  "observed": null,
  "artifacts": [ … ],
  "checkedAt": "2026-11-03T16:16:14.187Z",
  "receipt": {
    "version": 2,
    "id": "qwsqjsJi6eyCsPT5",
    "issuer": "https://machinereadiness.com",
    "checkedAt": "2026-11-03T16:16:14.187Z",
    "evidenceHash": "9aea6abd…",
    "url": "…", "finalUrl": "…", "path": "/", "purpose": "train", "agent": null, "verdict": "denied", "reason": "…",
    "page": { "status": 403, "read": false, "error": null },
    "signals": [ … ], "terms": [],
    "artifacts": [
      { "url": "https://www.nytimes.com/robots.txt", "status": 200, "bytes": 8547, "sha256": "04dee26f…", "truncated": false },
      { "url": "https://www.nytimes.com/ai.txt", "status": 404, "bytes": 0, "sha256": null, "truncated": false },
      …
    ],
    "observed": null,
    "signed": true,
    "signature": { "alg": "EdDSA", "crv": "Ed25519", "kid": "4f77bd8a74f20d99", "value": "o8YkmlLg…" }
  }
}

Errors

An address that cannot be read gets an error and no receipt.

StatuscodeWhen
400MISSING_URLNo url.
400INVALID_URLNot a page address, or a port other than 80, 443, 8080 or 8443.
400URL_TOO_LONGThe address is over 2,048 characters.
400INVALID_PURPOSEpurpose is not one of the four; the four come back in purposes.
400INVALID_AGENTagent is not a crawler token (letters, digits, . _ -, up to 64).
400PRIVATE_ADDRESSThe host is private or internal. Only public sites are read.
401INVALID_API_KEYThe x-api-key header holds a key we do not know.
405—A method other than GET, POST or OPTIONS.
422HOST_NOT_FOUNDThe host name does not resolve to a public address.
429ANON_LIMIT25 calls a day from this network without a key. Retry-After says when to try again.

The receipt

The receipt is the evidence: the address asked about and where it ended, the purpose, the verdict and the reason, the page's status, every signal as it was read with the per-crawler rows, the licence terms, and every file read, each with its status, its size and the SHA-256 of the bytes as they arrived. So it records what the site published, not only what we concluded. evidenceHash is the SHA-256 of those fields; the time, the id and the issuer sit outside it and inside the signature.

It is signed with Ed25519. The signature covers the receipt's canonical form (RFC 8785 JSON canonicalization: keys sorted at every level, no whitespace, UTF-8), without the signature and signed fields. /api/permission-key publishes the current key and every key it replaced, marked "status": "retired", so a receipt verifies later whenever it was issued: pick the key by its kid. Receipts from this host are issued as https://machinereadiness.com, from www.crawlreadiness.com as https://www.crawlreadiness.com, with the same key.

import { createHash, createPublicKey, verify } from "node:crypto";

// RFC 8785 canonical JSON: keys sorted at every level, no whitespace,
// UTF-8 with non-ASCII characters as they are.
const canonicalize = (v) =>
  Array.isArray(v) ? "[" + v.map(canonicalize).join(",") + "]"
  : v && typeof v === "object"
    ? "{" + Object.keys(v).filter((k) => v[k] !== undefined).sort()
        .map((k) => JSON.stringify(k) + ":" + canonicalize(v[k])).join(",") + "}"
    : JSON.stringify(v ?? null);

// 1. The key that signed it, current or retired, by its kid.
const { keys } = await (await fetch("https://machinereadiness.com/api/permission-key")).json();
const jwk = keys.find((k) => k.kid === receipt.signature?.kid);
if (!jwk) throw new Error("No published key has this kid");

// 2. The signature, over everything but itself.
const { signature, signed, ...rest } = receipt;
const signatureOk = verify(
  null,
  Buffer.from(canonicalize(rest)),
  createPublicKey({ key: { kty: "OKP", crv: "Ed25519", x: jwk.x }, format: "jwk" }),
  Buffer.from(signature.value, "base64url")
);

// 3. The evidence hash, recomputed from the receipt's own fields.
const { version, id, issuer, checkedAt, evidenceHash, ...evidence } = rest;
const hashOk = createHash("sha256").update(canonicalize(evidence)).digest("hex") === evidenceHash;

// 4. Who issued it.
const issuerOk = ["https://machinereadiness.com", "https://www.crawlreadiness.com"].includes(issuer);

A receipt is a dated record of what the site published and did at that moment. It is evidence, not legal advice, and not a certificate of compliance. What a use is permitted to do under copyright law in your jurisdiction is a question for a lawyer; what the site said about it is a question this answers.

Limits, keys and what is coming

  • 25 calls a day per network without a key; a free key from the Crawl Readiness dashboard during the test period. Pricing per thousand calls follows once the calls say what they are worth.
  • A check takes from a fraction of a second to 20 seconds, the most it waits for a slow site.
  • We log the purpose, the verdict and the kind of caller, not the address asked about.
  • The same call is the check_permission tool in the Crawl Readiness MCP server (npm crawl-readiness-mcp), the checkPermission endpoint on RapidAPI, and a request in the Postman collection.
  • Bulk: a list of URLs in, the share that reserves each use out. Ask at support@crawlreadiness.com.

Built by the makers of Crawl Readiness, on the same code that checks sites from the owner's side. Spawning's Data Diligence work came first for opt-outs before training; this reads the newer signals too, per purpose, and keeps a receipt of what it read.