--- title: "Steward, the auditor that fetched your site — Matt Pyle" description: "Steward is the agent-readiness auditor behind the steward-audit User-Agent: what it checks, what one audit costs your site, how to refuse it, and how to run one yourself." canonical: https://www.mattpyle.com/steward/ source: https://www.mattpyle.com/steward/ --- # Steward The auditor behind the steward-audit User-Agent, and the agent that publishes this site. Audit endpoint [www.mattpyle.com/mcp](https://www.mattpyle.com/mcp) ## Did Steward just fetch your site? Then somebody asked for a report on it. Steward reads what a site says about itself over plain HTTP, writes the report for whoever ran it, and leaves. It does not index anything and it does not come back on a schedule.
What one Steward request is
User-Agentsteward-audit/0.2.0 (+https://www.mattpyle.com/steward)
robots.txt tokensteward-audit
Cost of one auditRoughly a dozen HTTP requests over a few seconds. A deep audit adds up to three of your pages opened in a real browser, which fetch their own subresources the way any visitor's browser does.
FrequencyOnce, when a person or an agent asks. No schedule, no repeat visits, no crawl.
What it collectsYour public documents — robots.txt, the sitemap, llms.txt, agents.md, the well-known discovery documents — and the headers of a couple of pages.
**It obeys your robots.txt.** Every URL is checked against your rules before it is requested; only `/robots.txt` is fetched without asking. Anything you disallow is reported as “not checked”, never as a finding against your site. ### How to refuse it ``` User-agent: steward-audit Disallow: / ``` One audit then costs you one request — robots.txt, read once. Nothing else about how this site treats you changes. ## What Steward is Steward is this site's editorial agent. It reviews a draft — spelling, prose linting, an editorial pass, a real production build with accessibility and performance audits — waits durably for Matt to approve it, publishes by pull request, then verifies production. The agent-readiness audit started as one step of that review and turned out to be the half worth pointing at other people's sites. It checks behaviour rather than presence. A `200` from `/llms.txt` that is really the site's HTML 404 page is a failure here — the most common way a site passes a presence check while giving an agent nothing. Fast tier 13 checks over plain HTTP. Seconds, and the report comes back in the call. Deep tier Up to three pages opened in a real browser, reporting Lighthouse scores and axe-core violations. Minutes, and capped far harder. ## What the audit checks 13 checks, 3 categories, no composite score. ### Crawlability What the site says an agent may do. 4 checks - robots.txt exists and parses There is a robots.txt, it is served as text, and its groups and rules parse. medium - robots.txt lets user-triggered AI agents read the site The rules do not block the agents a person points at the site on purpose, which is a different question from whether they block training crawlers. high - Content Signals preferences are declared The site states what its content may be used for, in robots.txt or in a response header, rather than leaving it to be assumed. low - A sitemap is declared in robots.txt and fetchable robots.txt names a sitemap, and the sitemap at that URL is fetchable and parses. high ### Discovery What an agent can find without being told where to look. 6 checks - llms.txt exists and follows the spec There is an llms.txt and it is the format the convention describes, not an HTML page under that name. medium - The links in llms.txt resolve A sample of the URLs llms.txt points at are fetched, and they answer. medium - Every llms.txt list item leads with a markdown link Each list item starts with a link, so a parser can read the file as a list of URLs rather than as prose. low - agents.md exists and is markdown There is an agents.md brief, and it comes back as markdown rather than as a rendered page. medium - An MCP server is discoverable at /.well-known/mcp-server If the site has an MCP server, a JSON document names its endpoint, so an agent can find it without being told the URL. Nothing to fix if there is no server. low - An A2A agent card is published An A2A agent card is served at one of the two well-known paths, and it is JSON. low ### Content access Whether the content comes back in a form a model can read. 3 checks - The homepage serves markdown when asked for it A request for the homepage with \`Accept: text/markdown\` returns markdown, not HTML with a markdown content type. high - A content page serves markdown when asked for it The same question against a real content page, taken from the site’s own sitemap rather than guessed. high - The homepage advertises its alternates in a Link header The homepage points at its own machine-readable alternates in a \`Link\` header, where a client sees them before parsing anything. low ## Running an audit yourself Point an MCP client at `https://www.mattpyle.com/mcp` — a stateless streamable-HTTP server with three tools. No account, no key. `audit_site(url)` runs the fast checks and the report comes back in the call, with nothing to poll. Two renderings ship in one response: the canonical JSON in `structuredContent`, one entry per check with its evidence, and the same report as a markdown summary in the text content. A `GET` on [the endpoint itself](https://www.mattpyle.com/mcp) answers `405` with a worked example rather than a bare status. The unit audited is a site, not a page: any path in the URL is ignored, only `http` and `https` are accepted, and addresses on private networks are refused before a connection is opened. The site's own discovery document is at [/.well-known/mcp-server](/.well-known/mcp-server); it describes the fast tier, and `tools/list` enumerates the rest. ### The deep tier A deep audit takes longer than any MCP client holds a tool call open, so `deep_audit(url)` returns a workflow ID straight away and `get_audit(workflowId, view)` reads it back — there are no findings in the `deep_audit` response, and it says so. Poll `view: "status"` until `done` is true, then read `view: "report"` for the canonical JSON or `"summary"` for markdown. While it runs, the status view lists each unit of work with its state, and anything being retried carries its attempt number, straight from Temporal rather than from a count kept by hand. Queued means queued One worker sits behind this endpoint, so a busy moment is a real wait rather than a slow site. The status view answers queued with a position rather than leaving you to guess. Incomplete says so A deep audit that sampled pages and rendered none of them comes back marked degraded, naming which half is missing and which half can still be read. Durable either way A run survives a worker restart, and the workflow ID keeps answering. **Powered by [Temporal](https://temporal.io).** Matt works at Temporal, so that is disclosure as much as it is a credit. Fast audit ``` curl -sS https://www.mattpyle.com/mcp \ -H 'Content-Type: application/json' \ -H 'Accept: application/json, text/event-stream' \ -d '{"jsonrpc":"2.0","id":1,"method":"tools/call", "params":{"name":"audit_site","arguments":{"url":"example.com"}}}' ``` Deep audit ``` curl -sS https://www.mattpyle.com/mcp \ -H 'Content-Type: application/json' \ -H 'Accept: application/json, text/event-stream' \ -d '{"jsonrpc":"2.0","id":1,"method":"tools/call", "params":{"name":"deep_audit","arguments":{"url":"example.com"}}}' ``` Rate limits | Tier | Scope | Cap | Window | | --- | --- | --- | --- | | Fast | Per caller | 10 | One hour | | Fast | Everyone together | 500 | One UTC day | | Deep | Per caller | 4 | One UTC day | | Deep | Everyone together | 10 | One UTC day | A refusal is a JSON-RPC error with a `Retry-After` header naming which limit was hit and how long to wait. Every accepted call spends a dozen requests at somebody else's origin under this site's name; an unbounded endpoint would be an open relay with an honest User-Agent. The deep limits are two orders of magnitude smaller because a deep audit spends minutes of browser time on a machine somebody pays for. The counters hold a keyed hash of the caller's address with a lifetime no longer than the window, never the address itself. There is no key and no way to ask for more. `get_audit` is free: it reads a run you already paid for and makes no request at anybody's origin. ## Contact If Steward misbehaved on your site — ignored a rule, requested something it should not have, or reported something that is not true — open an issue. It is the same repository the auditor's code lives in, and it is read. [Open an issue](https://github.com/mattpyle/mattpyle.com-website/issues) Steward is an experiment on a personal site, not a supported service. It may change or be withdrawn without notice.