Crawl Surface
The machine-readable surface of this site — discovery files, AI context endpoints, and structured data — and which crawlers access each layer. Updated every 4 hours from the crawl log; owner traffic excluded.
0
Total requests
0
Distinct crawlers
0
Endpoint types
—
Last access
Discovery layer
Standard files every crawler checks first
Crawl rules + Content-Signal permissions
Not yet requested in the tracked window.
Canonical sitemap (linked from robots.txt)
Not yet requested in the tracked window.
AI context layer
Structured hints for LLMs and AI agents
LLM-readable site index (llmstxt.org spec)
Not yet requested in the tracked window.
Machine-readable endpoint catalog (JSON-LD)
Not yet requested in the tracked window.
Entity map — people, concepts, and organizations (EntityMap v1.0)
Not yet requested in the tracked window.
No machine-readable endpoint traffic in the snapshot yet. Data populates after the next snapshot run.
How this works: Every request to this site passes through a Cloudflare Worker that logs path, User-Agent, IP, and CF metadata to D1. A periodic snapshot job aggregates those rows into KV, which this page reads. Bot classification uses UA strings, CF reverse-DNS verification, and ASN fallbacks.
What each layer means: The discovery layer is what every crawler checks before indexing — robots.txt signals intent, sitemaps provide the URL map. The AI context layer is purpose-built for LLMs: llms.txt gives a plain-text orientation and page list, ai-catalog.json enumerates machine-readable endpoints, and entitymap.json declares structured entity relationships (people, concepts, organizations) following the EntityMap v1.0 spec. The raw content layer (.md requests) is agents bypassing HTML parsing entirely.
What each layer means: The discovery layer is what every crawler checks before indexing — robots.txt signals intent, sitemaps provide the URL map. The AI context layer is purpose-built for LLMs: llms.txt gives a plain-text orientation and page list, ai-catalog.json enumerates machine-readable endpoints, and entitymap.json declares structured entity relationships (people, concepts, organizations) following the EntityMap v1.0 spec. The raw content layer (.md requests) is agents bypassing HTML parsing entirely.