---
title: "Audit a site for AI agent readiness"
description: "Paste a website address and read an agent-readiness audit in the browser: robots.txt and its AI-agent rules, the sitemap, llms.txt, agents.md, the well-known discovery documents, and whether the pages serve markdown when asked for it."
canonical: https://www.mattpyle.com/audit/
source: https://www.mattpyle.com/audit/
rendered: 2026-10-30T23:21:14.620Z
---

# Audit a site

Paste an address. See how ready a site is for AI agents.

**Website address.** About a dozen requests to the site, obeying its robots.txt. What one audit does: https://www.mattpyle.com/steward/

A report for an address that has already been audited is at https://www.mattpyle.com/audit.md?url=<address>. Running a new audit is a form post, so it happens at https://www.mattpyle.com/audit/ in a browser, or through the MCP and A2A surfaces this site publishes.

## What the audit checks

### Crawlability

What the site says an agent may do.

| Check | What it asks | Weight |
| --- | --- | --- |
| robots.txt exists and parses | There is a robots.txt, it is served as text, and its groups and rules parse. | medium |
| robots.txt lets user-triggered AI agents read the site | The rules do not block the agents a person points at the site on purpose, which is a different question from whether they block training crawlers. | high |
| Content Signals preferences are declared | The site states what its content may be used for, in robots.txt or in a response header, rather than leaving it to be assumed. | low |
| A sitemap is declared in robots.txt and fetchable | robots.txt names a sitemap, and the sitemap at that URL is fetchable and parses. | high |

### Discovery

What an agent can find without being told where to look.

| Check | What it asks | Weight |
| --- | --- | --- |
| llms.txt exists and follows the spec | There is an llms.txt and it is the format the convention describes, not an HTML page under that name. | medium |
| The links in llms.txt resolve | A sample of the URLs llms.txt points at are fetched, and they answer. | medium |
| Every llms.txt list item leads with a markdown link | Each list item starts with a link, so a parser can read the file as a list of URLs rather than as prose. | low |
| agents.md exists and is markdown | There is an agents.md brief, and it comes back as markdown rather than as a rendered page. | medium |
| An MCP server is discoverable at /.well-known/mcp-server | If the site has an MCP server, a JSON document names its endpoint, so an agent can find it without being told the URL. Nothing to fix if there is no server. | low |
| An A2A agent card is published | An A2A agent card is served at one of the two well-known paths, and it is JSON. | low |

### Content access

Whether the content comes back in a form a model can read.

| Check | What it asks | Weight |
| --- | --- | --- |
| The homepage serves markdown when asked for it | A request for the homepage with `Accept: text/markdown` returns markdown, not HTML with a markdown content type. | high |
| A content page serves markdown when asked for it | The same question against a real content page, taken from the site’s own sitemap rather than guessed. | high |
| The homepage's first 8,000 characters are measured for content share | How much of the first 8,000 characters an agent reads is the page’s own content rather than navigation, as text and as markdown. A figure, not a verdict: the study it follows gives no target. | medium |
| A content page's first 8,000 characters are measured for content share | The same measurement on the sitemap-named content page. Give the audit a page URL and it measures that page too. | medium |
| The homepage advertises its alternates in a Link header | The homepage points at its own machine-readable alternates in a `Link` header, where a client sees them before parsing anything. | low |