Audit a site
Paste an address. See how ready a site is for AI agents.
About a dozen requests to the site, obeying its robots.txt. What one audit does
What the audit checks
Crawlability
What the site says an agent may do.
4 checks
- robots.txt exists and parsesThere is a robots.txt, it is served as text, and its groups and rules parse.medium
- robots.txt lets user-triggered AI agents read the siteThe rules do not block the agents a person points at the site on purpose, which is a different question from whether they block training crawlers.high
- Content Signals preferences are declaredThe site states what its content may be used for, in robots.txt or in a response header, rather than leaving it to be assumed.low
- A sitemap is declared in robots.txt and fetchablerobots.txt names a sitemap, and the sitemap at that URL is fetchable and parses.high
Discovery
What an agent can find without being told where to look.
6 checks
- llms.txt exists and follows the specThere is an llms.txt and it is the format the convention describes, not an HTML page under that name.medium
- The links in llms.txt resolveA sample of the URLs llms.txt points at are fetched, and they answer.medium
- Every llms.txt list item leads with a markdown linkEach list item starts with a link, so a parser can read the file as a list of URLs rather than as prose.low
- agents.md exists and is markdownThere is an agents.md brief, and it comes back as markdown rather than as a rendered page.medium
- An MCP server is discoverable at /.well-known/mcp-serverIf the site has an MCP server, a JSON document names its endpoint, so an agent can find it without being told the URL. Nothing to fix if there is no server.low
- An A2A agent card is publishedAn A2A agent card is served at one of the two well-known paths, and it is JSON.low
Content access
Whether the content comes back in a form a model can read.
5 checks
- The homepage serves markdown when asked for itA request for the homepage with
Accept: text/markdownreturns markdown, not HTML with a markdown content type.high - A content page serves markdown when asked for itThe same question against a real content page, taken from the site’s own sitemap rather than guessed.high
- The homepage's first 8,000 characters are measured for content shareHow much of the first 8,000 characters an agent reads is the page’s own content rather than navigation, as text and as markdown. A figure, not a verdict: the study it follows gives no target.medium
- A content page's first 8,000 characters are measured for content shareThe same measurement on the sitemap-named content page. Give the audit a page URL and it measures that page too.medium
- The homepage advertises its alternates in a Link headerThe homepage points at its own machine-readable alternates in a
Linkheader, where a client sees them before parsing anything.low