Steward
The auditor behind the steward-audit User-Agent, and the agent that publishes this site.
Did Steward just fetch your site?
Then somebody asked for a report on it. Steward reads what a site says about itself over plain HTTP, writes the report for whoever ran it, and leaves. It does not index anything and it does not come back on a schedule.
| User-Agent | steward-audit/0.2.0 (+https://www.mattpyle.com/steward) |
|---|---|
| robots.txt token | steward-audit |
| Cost of one audit | Roughly a dozen HTTP requests over a few seconds. A deep audit adds up to three of your pages opened in a real browser, which fetch their own subresources the way any visitor's browser does. |
| Frequency | Once, when a person or an agent asks. No schedule, no repeat visits, no crawl. |
| What it collects | Your public documents — robots.txt, the sitemap, llms.txt, agents.md, the well-known discovery documents — and the headers of a couple of pages. |
It obeys your robots.txt. Every URL is checked against your rules before it is requested; only /robots.txt is fetched without asking. Anything you disallow is reported as “not checked”, never as a finding against your site.
How to refuse it
User-agent: steward-audit
Disallow: /One audit then costs you one request — robots.txt, read once. Nothing else about how this site treats you changes.
What Steward is
Steward is this site's editorial agent. It reviews a draft — spelling, prose linting, an editorial pass, a real production build with accessibility and performance audits — waits durably for Matt to approve it, publishes by pull request, then verifies production. The agent-readiness audit started as one step of that review and turned out to be the half worth pointing at other people's sites.
It checks behaviour rather than presence. A 200 from /llms.txt that is really the site's HTML 404 page is a failure here — the most common way a site passes a presence check while giving an agent nothing.
What the audit checks
13 checks, 3 categories, no composite score.Crawlability
What the site says an agent may do.
4 checks
- robots.txt exists and parsesThere is a robots.txt, it is served as text, and its groups and rules parse.medium
- robots.txt lets user-triggered AI agents read the siteThe rules do not block the agents a person points at the site on purpose, which is a different question from whether they block training crawlers.high
- Content Signals preferences are declaredThe site states what its content may be used for, in robots.txt or in a response header, rather than leaving it to be assumed.low
- A sitemap is declared in robots.txt and fetchablerobots.txt names a sitemap, and the sitemap at that URL is fetchable and parses.high
Discovery
What an agent can find without being told where to look.
6 checks
- llms.txt exists and follows the specThere is an llms.txt and it is the format the convention describes, not an HTML page under that name.medium
- The links in llms.txt resolveA sample of the URLs llms.txt points at are fetched, and they answer.medium
- Every llms.txt list item leads with a markdown linkEach list item starts with a link, so a parser can read the file as a list of URLs rather than as prose.low
- agents.md exists and is markdownThere is an agents.md brief, and it comes back as markdown rather than as a rendered page.medium
- An MCP server is discoverable at /.well-known/mcp-serverIf the site has an MCP server, a JSON document names its endpoint, so an agent can find it without being told the URL. Nothing to fix if there is no server.low
- An A2A agent card is publishedAn A2A agent card is served at one of the two well-known paths, and it is JSON.low
Content access
Whether the content comes back in a form a model can read.
3 checks
- The homepage serves markdown when asked for itA request for the homepage with `Accept: text/markdown` returns markdown, not HTML with a markdown content type.high
- A content page serves markdown when asked for itThe same question against a real content page, taken from the site’s own sitemap rather than guessed.high
- The homepage advertises its alternates in a Link headerThe homepage points at its own machine-readable alternates in a `Link` header, where a client sees them before parsing anything.low
Running an audit yourself
Point an MCP client at https://www.mattpyle.com/mcp — a stateless streamable-HTTP server with three tools. No account, no key. audit_site(url) runs the fast checks and the report comes back in the call, with nothing to poll.
Two renderings ship in one response: the canonical JSON in structuredContent, one entry per check with its evidence, and the same report as a markdown summary in the text content. A GET on the endpoint itself answers 405 with a worked example rather than a bare status. The unit audited is a site, not a page: any path in the URL is ignored, only http and https are accepted, and addresses on private networks are refused before a connection is opened. The site's own discovery document is at /.well-known/mcp-server; it describes the fast tier, and tools/list enumerates the rest.
The deep tier
A deep audit takes longer than any MCP client holds a tool call open, so deep_audit(url) returns a workflow ID straight away and get_audit(workflowId, view) reads it back — there are no findings in the deep_audit response, and it says so. Poll view: "status" until done is true, then read view: "report" for the canonical JSON or "summary" for markdown. While it runs, the status view lists each unit of work with its state, and anything being retried carries its attempt number, straight from Temporal rather than from a count kept by hand.
- Queued means queued
- One worker sits behind this endpoint, so a busy moment is a real wait rather than a slow site. The status view answers queued with a position rather than leaving you to guess.
- Incomplete says so
- A deep audit that sampled pages and rendered none of them comes back marked degraded, naming which half is missing and which half can still be read.
- Durable either way
- A run survives a worker restart, and the workflow ID keeps answering.
Powered by Temporal. Matt works at Temporal, so that is disclosure as much as it is a credit.
curl -sS https://www.mattpyle.com/mcp \
-H 'Content-Type: application/json' \
-H 'Accept: application/json, text/event-stream' \
-d '{"jsonrpc":"2.0","id":1,"method":"tools/call",
"params":{"name":"audit_site","arguments":{"url":"example.com"}}}'curl -sS https://www.mattpyle.com/mcp \
-H 'Content-Type: application/json' \
-H 'Accept: application/json, text/event-stream' \
-d '{"jsonrpc":"2.0","id":1,"method":"tools/call",
"params":{"name":"deep_audit","arguments":{"url":"example.com"}}}'| Tier | Scope | Cap | Window |
|---|---|---|---|
| Fast | Per caller | 10 | One hour |
| Fast | Everyone together | 500 | One UTC day |
| Deep | Per caller | 4 | One UTC day |
| Deep | Everyone together | 10 | One UTC day |
A refusal is a JSON-RPC error with a Retry-After header naming which limit was hit and how long to wait. Every accepted call spends a dozen requests at somebody else's origin under this site's name; an unbounded endpoint would be an open relay with an honest User-Agent. The deep limits are two orders of magnitude smaller because a deep audit spends minutes of browser time on a machine somebody pays for. The counters hold a keyed hash of the caller's address with a lifetime no longer than the window, never the address itself. There is no key and no way to ask for more. get_audit is free: it reads a run you already paid for and makes no request at anybody's origin.
Contact
If Steward misbehaved on your site — ignored a rule, requested something it should not have, or reported something that is not true — open an issue. It is the same repository the auditor's code lives in, and it is read.