Version 1.0 · October 2026
The Machine Readiness Standard
A website is machine ready when it is readable by AI crawlers, its access policy is expressed and enforced, it is usable by AI agents, and all three are verified from outside on a date. Four parts, twenty-one checks. The free check on Crawl Readiness measures the first three; the permission API reads the second from a builder's side; the verify pages will carry the fourth.
Readable
A crawler that does not run JavaScript can read what the page is about.
- R1The page returns its content in the HTML, without JavaScript. A page that is blank until a script runs is not readable.
- R2The page has a title and a meta description.
- R3The page carries structured data as JSON-LD that describes what it is.
- R4The page has Open Graph title and description tags.
- R5robots.txt names a sitemap.
- R6Each page the policy allows is served to the crawlers it allows, not refused by a firewall or a rate limit on the first request.
Access
The site says what machines may do with it, says it everywhere it can be said, and the server does what it says.
- A1robots.txt states the policy per AI crawler, using current tokens from the registry: open, AI search only, no training, or closed.
- A2A Content-Signal line in robots.txt agrees with the crawler rules.
- A3When training is reserved, a TDM reservation says so: the tdm-reservation header or /.well-known/tdmrep.json.
- A4Page directives (noai, noindex, nosnippet) agree with the policy rather than contradict it.
- A5Enforcement matches the policy both ways: nothing allowed is refused, nothing blocked is served.
- A6When use is licensed under terms, an RSL licence is published and referenced from robots.txt, a Link header or the page. Optional.
Usable
An agent can take the site in cheaply and act on it, not only read it.
- U1An llms.txt at the root that begins with a heading and describes the site.
- U2An agents.json at /.well-known that points agents at the site's resources.
- U3A markdown copy of each page, served when asked with Accept: text/markdown, about two thirds smaller than the HTML.
- U4Every form field carries a label an agent can read: a label element, aria-label, aria-labelledby or title. A placeholder alone does not count.
- U5Forms that are actions declare themselves as tools (WebMCP toolname and tooldescription). Optional while the standard is a browser trial.
Proven
Nothing above is self-declared. It is checked from outside, on a date, and checked again.
- P1The checks are run from outside the site, by a party that is not the site, the way a crawler meets it.
- P2They are dated and repeated, weekly at least, so drift is caught: a CMS update, a plugin, a CDN rule.
- P3The policy statement is published in plain words: for example, “AI search: yes · AI training: no”.
- P4Changes are recorded, so a reader can see what changed and when.
What the standard does not claim
Meeting it does not make an AI mention a site, cite it or rank it. It makes the site readable, correctly governed and usable, which is the part a site controls. Whether an answer engine then uses the site depends on far more than readiness, and no honest standard measures that as a property of the site.
A site that blocks AI on purpose can meet every part of the standard. Access is about saying what you mean and having the server do it, not about saying yes.
How it is measured
The checks are the ones the Crawl Readiness methodology describes, run from outside against the live site: robots.txt parsed per crawler with the tokens in the registry, the homepage fetched without JavaScript and again as four AI crawlers, the markdown copy requested, the forms read, the signals compared. The score Crawl Readiness shows counts the Readable checks and the crawlers that feed search and answers; the Access, Usable and Proven parts are reported beside it rather than inside it.
Versions
| Version | Date | Change |
|---|---|---|
1.0 | 24 October 2026 | First published version: Readable R1–R6, Access A1–A6, Usable U1–U5, Proven P1–P4. |
Changes are made in the open: a new token in the registry, a new signal in Access, a new convention in Usable. Suggest one at support@crawlreadiness.com.