Crawl Surface
The machine-readable surface of this site — discovery files, AI context endpoints, and structured data — and which crawlers access each layer. Updated every 4 hours from the crawl log; owner traffic excluded.
239
Total requests
10
Distinct crawlers
6
Endpoint types
26m ago
Last access
AhrefsBotMozBotMajesticBotSemrushBotPetalBotApplebot
Discovery layer
Standard files every crawler checks first
Crawl rules + Content-Signal permissions
MozBot 33
MajesticBot 16
SemrushBot 15
AhrefsBot 12
Chrome 8
Facebook 7
OpenAI SearchBot 7
Applebot 7
+10 more crawlers
Canonical sitemap (linked from robots.txt)
AhrefsBot 68
PetalBot 5
Bingbot 5
GPTBot 4
MozBot 1
ChatGPT (User) 1
Other bot 1
AI context layer
Structured hints for LLMs and AI agents
LLM-readable site index (llmstxt.org spec)
Applebot 1
MozBot 1
Meta (Scraper) 1
OpenAI SearchBot 1
Other bot 1
Safari 1
Machine-readable endpoint catalog (JSON-LD)
Applebot 1
MozBot 1
Meta (Scraper) 1
CCBot 1
Other bot 1
Entity map — people, concepts, and organizations (EntityMap v1.0)
PetalBot 1
MozBot 1
Meta (Scraper) 1
Amazonbot 1
Other bot 1
Raw content layer
.md / .mdx page requests — AI agents preferring plain text
6 requests
PetalBot 1
SemrushBot 1
Meta (Scraper) 1
Other bot 1
Safari 1
Applebot 1
All crawlers — combined across every endpoint
AhrefsBot 80
MozBot 37
MajesticBot 16
SemrushBot 16
PetalBot 11
Applebot 10
Other bot 9
OpenAI SearchBot 8
Chrome 8
Bingbot 7
How this works: Every request to this site passes through a Cloudflare Worker that logs path, User-Agent, IP, and CF metadata to D1. A periodic snapshot job aggregates those rows into KV, which this page reads. Bot classification uses UA strings, CF reverse-DNS verification, and ASN fallbacks.
What each layer means: The discovery layer is what every crawler checks before indexing — robots.txt signals intent, sitemaps provide the URL map. The AI context layer is purpose-built for LLMs: llms.txt gives a plain-text orientation and page list, ai-catalog.json enumerates machine-readable endpoints, and entitymap.json declares structured entity relationships (people, concepts, organizations) following the EntityMap v1.0 spec. The raw content layer (.md requests) is agents bypassing HTML parsing entirely.
What each layer means: The discovery layer is what every crawler checks before indexing — robots.txt signals intent, sitemaps provide the URL map. The AI context layer is purpose-built for LLMs: llms.txt gives a plain-text orientation and page list, ai-catalog.json enumerates machine-readable endpoints, and entitymap.json declares structured entity relationships (people, concepts, organizations) following the EntityMap v1.0 spec. The raw content layer (.md requests) is agents bypassing HTML parsing entirely.