AI Readiness & Crawl Surface

The problem AI crawlers have with most sites

HTML pages are built for browsers. An AI agent trying to understand a site has to parse navigation, ads, sidebars, and layout scaffolding to find the actual content. A 25KB blog post can be wrapped in 80KB of HTML. There’s no standard way for an agent to discover what a site publishes, what it’s about, or which endpoints are meaningful.

This site exposes a structured discovery layer for exactly this use case.

What’s in the crawl surface

Discovery layer — how crawlers find the site and understand its rules:

• /robots.txt — standard crawl rules with a Content-Signal block declaring AI permissions (search: yes, ai-input: yes, ai-train: no, use: reference)
• /sitemap-main.xml — canonical URL map for all public pages

AI context layer — structured context for agents:

• /llms.txt — plain-text site orientation following the llmstxt.org spec. Describes who I am, what the site covers, and links to key pages in a format optimized for LLM consumption.
• /ai-catalog.json — JSON endpoint catalog listing all machine-readable endpoints with type, description, and access notes
• /entitymap.json — entity graph (see the Entity Map entry) declaring the site’s key concepts and their relationships in a structured format

Content layer (planned):

• .md variants of content pages — plain text, no HTML scaffolding, ~5x smaller payload than the equivalent HTML

The /crawl-surface/ dashboard

/crawl-surface/ is a public page that lists all machine-readable endpoints and, for each one, shows which bots are accessing it — with per-bot hit counts drawn from the crawl intelligence pipeline.

This closes the loop: I can see whether AI crawlers are actually discovering and fetching these endpoints, and which ones. /llms.txt getting hit by GPTBot is measurable evidence that the content is being consumed.

Content-Signal in robots.txt

The Content-Signal block is a proposed extension to robots.txt that declares how content may be used:

# Content-Signal: AI Usage Permissions
Content-Signal: search=yes; ai-input=yes; ai-train=no; use=reference

This lets AI systems know explicitly: use the content to answer questions (ai-input: yes), don’t use it for training data (ai-train: no), cite don’t reproduce (use: reference). It’s self-declared and not technically enforceable, but it’s a clear machine-readable signal that responsible crawlers can honor.

What’s tracked

Every access to /llms.txt, /ai-catalog.json, /entitymap.json, /robots.txt, and sitemaps is logged with full bot detail in the crawl intelligence pipeline. The snapshot:ai-paths KV key stores per-bot hit counts, first-seen, and last-seen for each endpoint. This data feeds the /crawl-surface/ dashboard.

Planned: Markdown content negotiation

The next layer is serving .md variants of content pages to AI crawlers that request them. The plan:

1. Check Accept: text/markdown header — serve .md if present
2. UA-based fallback for known AI crawlers that don’t set the header
3. Generate .md routes at build time alongside HTML pages

A typical blog post is ~85KB as HTML and ~16KB as markdown. At scale, this reduces bandwidth consumed by AI crawlers by roughly 5x and makes the content dramatically easier to process.