For RAG, agents, scrapers and dataset builders

# May I use this page for this?

One call: a page and a purpose in; what robots.txt, Content-Signal, Content-Usage, an RSL licence, the TDM reservation, ai.txt and the page's directives say out, with a signed, dated receipt that records every file read.

A site can reserve its rights in seven places that do not agree with each other. Most crawlers read robots.txt at best. This call reads all of them for one page and one purpose, says what they add up to, and hands back a dated receipt you can keep, with a hash of every file it read.

[The request](https://machinereadiness.com/permission#ask) [The receipt](https://machinereadiness.com/permission#receipt)

Free during the first weeks, 25 calls a day per network without a key. A free API key from the [Crawl Readiness dashboard](https://www.crawlreadiness.com/dashboard), sent as the `x-api-key` header, lifts that limit.

## The request

```
curl "https://machinereadiness.com/api/permission?url=https://www.nytimes.com/&purpose=train"
```

`GET` or `POST` (JSON body) to `/api/permission` on this host or on www.crawlreadiness.com, with:

| Parameter | Meaning |
| --- | --- |
| `url` | The page, with its path. The site's robots.txt, ai.txt and TDM file are read from its origin, and from the origin it redirects to. |
| `purpose` | `search` · `answer` · `train` · `agent`, as below. |
| `agent` | Optional. **Your own** crawler's token, such as `MyBot`: robots.txt is then read for that crawler's rule. Naming another company's crawler (`GPTBot`, `Googlebot`) cannot loosen the answer: the verdict is then the stricter of its rule and the reading for the crawlers that do the job. |
| `observe` | Optional, off by default. `1` also requests the page with a representative crawler's user-agent string; see below for what that can and cannot show. |

## The purposes, and the crawlers read for each

robots.txt speaks about crawlers, not purposes, so a purpose is read as the site's rules for the crawlers that do that job. **A block on any one of them reserves the use.**

| Purpose | Meaning | Crawlers read |
| --- | --- | --- |
| `search` | Index the page and show links and excerpts in AI search results. | OAI-SearchBot, Claude-SearchBot, PerplexityBot, YouBot, meta-webindexer, MistralAI-Index, Amzn-SearchBot |
| `answer` | Use the page as input to a generated answer or summary, retrieval for RAG included. | OAI-SearchBot, PerplexityBot, Claude-SearchBot, DuckAssistBot, Google-Extended, meta-externalagent, meta-webindexer |
| `train` | Train or fine-tune a model on the page. | GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended, Bytespider, meta-externalagent, Amazonbot, MistralAI-Training |
| `agent` | Fetch or act on the page on a user's behalf, live. | ChatGPT-User, Claude-User, Perplexity-User, meta-externalfetcher, Google-Agent, Google-GeminiNotebook, MistralAI-User, Amzn-User |

Fetchers that run when a person asks (ChatGPT-User, Perplexity-User, Google's user-triggered fetchers, Meta-ExternalFetcher, Amzn-User) are described by their own vendors as able to skip robots.txt, because a person requested the page. A verdict for the `agent` purpose says what the site published, not what those fetchers will do.

## What it reads

- **robots.txt**, for those crawlers, for a crawler it does not name through `*`, or for your own token. Its status is read as RFC 9309 says, with two cautious exceptions: a 404 or 410 means no file, so nothing reserved; a server error or no answer means a full disallow, as the RFC says, and so does a 429, which the RFC files with the other 4xx answers but Google reads as a server error. A 401 or 403 is reported as unknown, where RFC 9309 would let a crawler go ahead, because a refusal like that is usually a firewall, not a policy. Rules sent with another 4xx (Stack Overflow answers with 418) are read, and the answer says so.
- **Content-Signal**, Cloudflare's line in robots.txt: `search`, `ai-input`, `ai-train`.
- **Content-Usage**, the IETF AI-preferences vocabulary (`train-ai`, `ai-use`, `search`) as a robots.txt directive per group and path, and as an HTTP header.
- **RSL 1.0**, found through a `License:` line in robots.txt, a `Link` header, a `<link rel="license">` or an inline script. A `License:` line after the `Sitemap:` lines or a blank line covers the whole site; one inside a group covers that group's crawlers. Permits, prohibits and payment terms are read from the most specific `<content>` entry; a licence server on it is a condition, since RSL has clients obtain a licence there first. For `answer`, a prohibition of `ai-index` counts as well as `ai-input`.
- **The TDM reservation** (W3C TDMRep), the EU text-and-data-mining opt-out: a `tdm-reservation` meta tag, then the header, then `/.well-known/tdmrep.json`, where the first matching rule applies. It decides `train`; for the other purposes it is reported, not counted, because whether they are text and data mining is a legal question.
- **ai.txt**, about training.
- The page's own **directives** in a meta tag or `X-Robots-Tag`, for every crawler or for one that does the job: `noai` (training, answers, agents), `noindex` (search), `nosnippet` and `max-snippet:0` (no answers; search without excerpts), and Bing's `noarchive` and `nocache` (no Copilot answers).

When the page redirects, the host it ends on is read for its path, and the first host's robots.txt still counts for the first request. Every read is made as `CrawlReadinessBot/1.0 (+https://www.crawlreadiness.com)`, bounded in size and time; the whole check stops reading at 20 seconds.

**observe=1** also requests the page once with a representative crawler's user-agent string (GPTBot for `train`, PerplexityBot for `answer`, OAI-SearchBot for `search`, ChatGPT-User for `agent`), from our servers, and reports the status beside our own request as the control. It shows how the server treats that string, not what the real crawler gets: a site that checks a crawler's IP address or reverse DNS can tell the difference. It is skipped when robots.txt disallows that crawler, and it never changes the verdict.

## What comes back

A `verdict`, the `reason` in plain words, one entry per signal with what was found and what it decided, the page's `status` and `finalUrl`, `observed` when asked for, and the `receipt`. The strongest decision among the signals sets the verdict, in this order:

| Verdict | Meaning |
| --- | --- |
| `denied` | A machine-readable signal reserves this use: any reservation that covers the purpose wins, a block on any one crawler that does the job included. Also when robots.txt answers a server error or does not answer, which RFC 9309 treats as a full disallow, or answers a 429, which Google reads the same way. |
| `allowed-under-terms` | Permitted on conditions: an RSL licence's payment, user or region limits, or a licence server to obtain it from first (all in `terms`); or, for search, a nosnippet directive (no excerpts). |
| `unknown` | A source that could carry a reservation could not be read: robots.txt answered 401 or 403, the page refused us or did not answer, a referenced licence could not be fetched or does not follow RSL, or, for training, tdmrep.json or ai.txt failed. Unknown outranks allowed: what could be read cannot speak for what could not. |
| `allowed` | No signal reserves the use. That is all it means: the absence of a signal is not a licence. |

```
{
  "version": 2,
  "url": "https://www.nytimes.com/",
  "finalUrl": "https://www.nytimes.com/",
  "path": "/",
  "purpose": "train",
  "purposeLabel": "AI training",
  "agent": null,
  "verdict": "denied",
  "reason": "robots.txt blocks 7 of the 9 crawlers that do this: GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended, Bytespider, meta-externalagent. Any reservation wins; it allows Amazonbot, MistralAI-Training.",
  "terms": [],
  "signals": [
    { "source": "robots.txt", "found": true, "decision": "deny", "status": 200, "detail": "…",
      "crawlers": [ { "token": "GPTBot", "allowed": false }, … ] },
    { "source": "Content-Signal", "found": false, "decision": "silent", "detail": "No Content-Signal line." },
    …
    { "source": "Page directives", "found": false, "decision": "unknown",
      "detail": "The page answered HTTP 403, so its own directives (noai, noindex, nosnippet) could not be read." },
    …
  ],
  "note": "A reading of the signals the site publishes, not legal advice. …",
  "agentKnown": null,
  "page": { "status": 403, "fetched": false, "finalUrl": "https://www.nytimes.com/", "error": null },
  "observed": null,
  "artifacts": [ … ],
  "checkedAt": "2026-11-03T16:16:14.187Z",
  "receipt": {
    "version": 2,
    "id": "qwsqjsJi6eyCsPT5",
    "issuer": "https://machinereadiness.com",
    "checkedAt": "2026-11-03T16:16:14.187Z",
    "evidenceHash": "9aea6abd…",
    "url": "…", "finalUrl": "…", "path": "/", "purpose": "train", "agent": null, "verdict": "denied", "reason": "…",
    "page": { "status": 403, "read": false, "error": null },
    "signals": [ … ], "terms": [],
    "artifacts": [
      { "url": "https://www.nytimes.com/robots.txt", "status": 200, "bytes": 8547, "sha256": "04dee26f…", "truncated": false },
      { "url": "https://www.nytimes.com/ai.txt", "status": 404, "bytes": 0, "sha256": null, "truncated": false },
      …
    ],
    "observed": null,
    "signed": true,
    "signature": { "alg": "EdDSA", "crv": "Ed25519", "kid": "4f77bd8a74f20d99", "value": "o8YkmlLg…" }
  }
}
```

### Errors

An address that cannot be read gets an error and no receipt.

| Status | code | When |
| --- | --- | --- |
| 400 | `MISSING_URL` | No url. |
| 400 | `INVALID_URL` | Not a page address, or a port other than 80, 443, 8080 or 8443. |
| 400 | `URL_TOO_LONG` | The address is over 2,048 characters. |
| 400 | `INVALID_PURPOSE` | purpose is not one of the four; the four come back in purposes. |
| 400 | `INVALID_AGENT` | agent is not a crawler token (letters, digits, . _ -, up to 64). |
| 400 | `PRIVATE_ADDRESS` | The host is private or internal. Only public sites are read. |
| 401 | `INVALID_API_KEY` | The x-api-key header holds a key we do not know. |
| 405 | — | A method other than GET, POST or OPTIONS. |
| 422 | `HOST_NOT_FOUND` | The host name does not resolve to a public address. |
| 429 | `ANON_LIMIT` | 25 calls a day from this network without a key. Retry-After says when to try again. |

## The receipt

The receipt is the evidence: the address asked about and where it ended, the purpose, the verdict and the reason, the page's status, every signal as it was read with the per-crawler rows, the licence terms, and every file read, each with its status, its size and the SHA-256 of the bytes as they arrived. So it records what the site published, not only what we concluded. `evidenceHash` is the SHA-256 of those fields; the time, the id and the issuer sit outside it and inside the signature.

It is signed with Ed25519. The signature covers the receipt's canonical form (RFC 8785 JSON canonicalization: keys sorted at every level, no whitespace, UTF-8), without the `signature` and `signed` fields. `/api/permission-key` publishes the current key and every key it replaced, marked `"status": "retired"`, so a receipt verifies later whenever it was issued: pick the key by its `kid`. Receipts from this host are issued as `https://machinereadiness.com`, from www.crawlreadiness.com as `https://www.crawlreadiness.com`, with the same key.

```
import { createHash, createPublicKey, verify } from "node:crypto";

// RFC 8785 canonical JSON: keys sorted at every level, no whitespace,
// UTF-8 with non-ASCII characters as they are.
const canonicalize = (v) =>
  Array.isArray(v) ? "[" + v.map(canonicalize).join(",") + "]"
  : v && typeof v === "object"
    ? "{" + Object.keys(v).filter((k) => v[k] !== undefined).sort()
        .map((k) => JSON.stringify(k) + ":" + canonicalize(v[k])).join(",") + "}"
    : JSON.stringify(v ?? null);

// 1. The key that signed it, current or retired, by its kid.
const { keys } = await (await fetch("https://machinereadiness.com/api/permission-key")).json();
const jwk = keys.find((k) => k.kid === receipt.signature?.kid);
if (!jwk) throw new Error("No published key has this kid");

// 2. The signature, over everything but itself.
const { signature, signed, ...rest } = receipt;
const signatureOk = verify(
  null,
  Buffer.from(canonicalize(rest)),
  createPublicKey({ key: { kty: "OKP", crv: "Ed25519", x: jwk.x }, format: "jwk" }),
  Buffer.from(signature.value, "base64url")
);

// 3. The evidence hash, recomputed from the receipt's own fields.
const { version, id, issuer, checkedAt, evidenceHash, ...evidence } = rest;
const hashOk = createHash("sha256").update(canonicalize(evidence)).digest("hex") === evidenceHash;

// 4. Who issued it.
const issuerOk = ["https://machinereadiness.com", "https://www.crawlreadiness.com"].includes(issuer);
```

A receipt is a dated record of what the site published and did at that moment. It is evidence, not legal advice, and not a certificate of compliance. What a use is permitted to do under copyright law in your jurisdiction is a question for a lawyer; what the site said about it is a question this answers.

## Limits, keys and what is coming

- 25 calls a day per network without a key; a free key from the Crawl Readiness dashboard during the test period. Pricing per thousand calls follows once the calls say what they are worth.
- A check takes from a fraction of a second to 20 seconds, the most it waits for a slow site.
- We log the purpose, the verdict and the kind of caller, not the address asked about.
- The same call is the `check_permission` tool in the [Crawl Readiness MCP server](https://www.crawlreadiness.com/mcp) (npm [crawl-readiness-mcp](https://www.npmjs.com/package/crawl-readiness-mcp)), the `checkPermission` endpoint on [RapidAPI](https://rapidapi.com/crawlreadiness/api/crawl-readiness), and a request in the Postman collection.
- Bulk: a list of URLs in, the share that reserves each use out. Ask at [support@crawlreadiness.com](mailto:support@crawlreadiness.com).

Built by the makers of [Crawl Readiness](https://www.crawlreadiness.com/), on the same code that checks sites from the owner's side. Spawning's Data Diligence work came first for opt-outs before training; this reads the newer signals too, per purpose, and keeps a receipt of what it read.

---

Source: https://machinereadiness.com/permission
