For RAG, agents, scrapers and dataset builders

May I use this page for this?

A site can reserve its rights in half a dozen places that do not agree with each other. Most crawlers read robots.txt at best. This call reads all of them for one page and one purpose, says what they add up to, tells you what the server did when asked, and hands back a dated receipt you can keep.

Free during the first weeks, 25 calls a day per network without a key. A free API key from the Crawl Readiness dashboard lifts that limit.

The request

curl "https://machinereadiness.com/api/permission?url=https://www.nytimes.com/&purpose=train"

GET or POST (JSON body) to /api/permission on this host or on www.crawlreadiness.com, with:

What it reads

• robots.txt, for the crawlers that do the job (from the registry), for an unknown crawler through *, or for the token you name.
• Content-Signal, Cloudflare's line in robots.txt: search, ai-input, ai-train.
• Content-Usage, the IETF AI-preferences vocabulary (train-ai, ai-use, search) as a robots.txt directive per group and path, and as an HTTP header.
• RSL 1.0, found through a License: line in robots.txt, a Link header, a <link rel="license"> or an inline script: permits, prohibits, payment terms, the most specific <content> entry winning.
• The TDM reservation, as the tdm-reservation header or /.well-known/tdmrep.json with the longest matching location.
• ai.txt, and the page's own noai, noindex and nosnippet directives in a meta tag or X-Robots-Tag.

Then it requests the page once as one representative crawler for the purpose, with the same user-agent strings as the monthly study, and reports the status: served, refused, or an error.

What comes back

A verdict, the reason in plain words, one entry per signal with what was found and what it decided, the observed server response, and the receipt.

{
  "url": "https://www.nytimes.com/",
  "purpose": "train",
  "verdict": "denied",
  "reason": "robots.txt: robots.txt blocks 6 of the 6 crawlers that do this: GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended, Bytespider.",
  "signals": [
    { "source": "robots.txt", "found": true, "decision": "deny", "detail": "…", "crawlers": [ { "token": "GPTBot", "allowed": false }, … ] },
    { "source": "Content-Signal", "found": false, "decision": "silent", "detail": "No Content-Signal line." },
    { "source": "TDM reservation", "found": false, "decision": "silent", "detail": "…" },
    { "source": "RSL licence", "found": false, "decision": "silent", "detail": "…" },
    …
  ],
  "observed": { "token": "GPTBot", "status": 403, "result": "refused" },
  "checkedAt": "2026-10-24T01:12:09.511Z",
  "receipt": {
    "version": 1,
    "id": "d912C4yn4wQ9f3bb",
    "issuer": "https://machinereadiness.com",
    "checkedAt": "2026-10-24T01:12:09.511Z",
    "evidenceHash": "9c1f…",
    "url": "…", "purpose": "train", "verdict": "denied", "signals": [ … ], "observed": { … },
    "signed": true,
    "signature": { "alg": "EdDSA", "crv": "Ed25519", "kid": "4f77bd8a74f20d99", "value": "72JIG6ed…" }
  }
}

The receipt

The receipt is the evidence: the page, the purpose, the verdict, every signal as it was read, the server's response, the time, and a SHA-256 hash of all of it. It is signed with an Ed25519 key whose public half is published at /api/permission-key as a JWK set, so anyone can verify a receipt later without trusting the copy they were handed. Sign over the canonical form: keys sorted at every level, no whitespace, the signature and signed fields left out.

import { createPublicKey, verify } from "node:crypto";

const canonicalize = (v) =>
  Array.isArray(v) ? "[" + v.map(canonicalize).join(",") + "]"
  : v && typeof v === "object"
    ? "{" + Object.keys(v).filter((k) => v[k] !== undefined).sort()
        .map((k) => JSON.stringify(k) + ":" + canonicalize(v[k])).join(",") + "}"
    : JSON.stringify(v ?? null);

const { keys } = await (await fetch("https://machinereadiness.com/api/permission-key")).json();
const jwk = keys.find((k) => k.kid === receipt.signature.kid);
const { signature, signed, ...rest } = receipt;
const ok = verify(
  null,
  Buffer.from(canonicalize(rest)),
  createPublicKey({ key: { kty: "OKP", crv: "Ed25519", x: jwk.x }, format: "jwk" }),
  Buffer.from(signature.value, "base64url")
);

A receipt is a dated record of what the site published and did at that moment. It is evidence, not legal advice, and not a certificate of compliance. What a use is permitted to do under copyright law in your jurisdiction is a question for a lawyer; what the site said about it is a question this answers.

Limits, keys and what is coming

• 25 calls a day per network without a key; a free key from the Crawl Readiness dashboard during the test period. Pricing per thousand calls follows once the calls say what they are worth.
• The same call as a tool in the Crawl Readiness MCP server, as an Apify actor and on RapidAPI: next.
• Bulk: a list of URLs in, the share that reserves each use out. Ask at support@crawlreadiness.com.

Built by the makers of Crawl Readiness, on the same code that checks sites from the owner's side. Spawning's Data Diligence work came first for opt-outs before training; this reads the newer signals too, per purpose, and shows what the server actually does.