Audit a site
Paste an address. See how ready a site is for AI agents.
About a dozen requests to the site, obeying its robots.txt. What one audit does
What the audit checks
Crawlability
What the site says an agent may do.
4 checks
- robots.txt exists and parsesThere is a robots.txt, it is served as text, and its groups and rules parse.medium
- robots.txt lets user-triggered AI agents read the siteThe rules do not block the agents a person points at the site on purpose, which is a different question from whether they block training crawlers.high
- Content Signals preferences are declaredThe site states what its content may be used for, in robots.txt or in a response header, rather than leaving it to be assumed.low
- A sitemap is declared in robots.txt and fetchablerobots.txt names a sitemap, and the sitemap at that URL is fetchable and parses.high
Discovery
What an agent can find without being told where to look.
6 checks
- llms.txt exists and follows the specThere is an llms.txt and it is the format the convention describes, not an HTML page under that name.medium
- The links in llms.txt resolveA sample of the URLs llms.txt points at are fetched, and they answer.medium
- Every llms.txt list item leads with a markdown linkEach list item starts with a link, so a parser can read the file as a list of URLs rather than as prose.low
- agents.md exists and is markdownThere is an agents.md brief, and it comes back as markdown rather than as a rendered page.medium
- An MCP server is discoverable at /.well-known/mcp-serverIf the site has an MCP server, a JSON document names its endpoint, so an agent can find it without being told the URL. Nothing to fix if there is no server.low
- An A2A agent card is publishedAn A2A agent card is served at one of the two well-known paths, and it is JSON.low
Content access
Whether the content comes back in a form a model can read.
3 checks
- The homepage serves markdown when asked for itA request for the homepage with `Accept: text/markdown` returns markdown, not HTML with a markdown content type.high
- A content page serves markdown when asked for itThe same question against a real content page, taken from the site’s own sitemap rather than guessed.high
- The homepage advertises its alternates in a Link headerThe homepage points at its own machine-readable alternates in a `Link` header, where a client sees them before parsing anything.low