Crawl Surface
The machine-readable surface of this site — discovery files, AI context endpoints, and structured data — and which crawlers access each layer. Updated every 4 hours from the crawl log; owner traffic excluded.
3,224
Total requests
10
Distinct crawlers
11
Endpoint types
5h ago
Last access
ClaudeBotAhrefsBotChromeMajesticBotSemrushBotPetalBot
Discovery layer
Standard files every crawler checks first
Crawl rules + Content-Signal permissions
ClaudeBot 451
AhrefsBot 145
MajesticBot 132
SemrushBot 119
Chrome 113
OpenAI SearchBot 76
MozBot 69
Safari 67
+26 more crawlers
Canonical sitemap (linked from robots.txt)
AhrefsBot 546
ClaudeBot 452
Bingbot 57
GPTBot 50
PetalBot 17
Chrome 17
Other bot 8
SemrushBot 3
+12 more crawlers
AI context layer
Structured hints for LLMs and AI agents
LLM-readable site index (llmstxt.org spec)
Chrome 17
PetalBot 11
Other bot 9
AhrefsBot 8
Firefox 4
Barkrowler 4
GoogleOther 3
Google (by ASN) 3
+14 more crawlers
Machine-readable endpoint catalog (JSON-LD)
Chrome 47
AhrefsBot 8
PetalBot 7
Other bot 5
MozBot 2
Safari 2
Screaming Frog 2
Firefox 2
+9 more crawlers
Entity map — people, concepts, and organizations (EntityMap v1.0)
Chrome 48
PetalBot 9
AhrefsBot 8
Other bot 5
Safari 3
GPTBot 2
Screaming Frog 2
Firefox 2
+10 more crawlers
Raw content layer
.md / .mdx page requests — AI agents preferring plain text
66 requests
PetalBot 9
Chrome 8
AhrefsBot 8
Safari 7
Other bot 5
Firefox 4
curl 4
Barkrowler 3
+13 more crawlers
38 requests
AhrefsBot 6
curl 5
Firefox 4
Safari 3
Other bot 3
GoogleOther 2
Screaming Frog 2
PetalBot 2
+10 more crawlers
35 requests
AhrefsBot 6
curl 5
Firefox 4
Other bot 3
Screaming Frog 2
PetalBot 2
Safari 2
Chrome 2
+9 more crawlers
30 requests
AhrefsBot 6
Chrome 4
curl 3
Firefox 3
Other bot 3
PetalBot 2
MajesticBot 1
GPTBot 1
+7 more crawlers
12 requests
Other bot 2
Googlebot 2
AhrefsBot 2
Chrome 2
SemrushBot 1
GPTBot 1
PetalBot 1
Amazonbot 1
All crawlers — combined across every endpoint
ClaudeBot 910
AhrefsBot 743
Chrome 261
MajesticBot 136
SemrushBot 130
PetalBot 109
Other bot 109
Safari 89
Bingbot 83
MozBot 77
How this works: Every request to this site passes through a Cloudflare Worker that logs path, User-Agent, IP, and CF metadata to D1. A periodic snapshot job aggregates those rows into KV, which this page reads. Bot classification uses UA strings, CF reverse-DNS verification, and ASN fallbacks.
What each layer means: The discovery layer is what every crawler checks before indexing — robots.txt signals intent, sitemaps provide the URL map. The AI context layer is purpose-built for LLMs: llms.txt gives a plain-text orientation and page list, ai-catalog.json enumerates machine-readable endpoints, and entitymap.json declares structured entity relationships (people, concepts, organizations) following the EntityMap v1.0 spec. The raw content layer (.md requests) is agents bypassing HTML parsing entirely.
What each layer means: The discovery layer is what every crawler checks before indexing — robots.txt signals intent, sitemaps provide the URL map. The AI context layer is purpose-built for LLMs: llms.txt gives a plain-text orientation and page list, ai-catalog.json enumerates machine-readable endpoints, and entitymap.json declares structured entity relationships (people, concepts, organizations) following the EntityMap v1.0 spec. The raw content layer (.md requests) is agents bypassing HTML parsing entirely.