Crawl Surface
The machine-readable surface of this site — discovery files, AI context endpoints, and structured data — and which crawlers access each layer. Updated every 4 hours from the crawl log; owner traffic excluded.
5,733
Total requests
10
Distinct crawlers
24
Endpoint types
5h ago
Last access
ClaudeBotAhrefsBotChromeOther botSemrushBotPetalBot
Discovery layer
Standard files every crawler checks first
Crawl rules + Content-Signal permissions
ClaudeBot 845
AhrefsBot 285
Chrome 236
OpenAI SearchBot 201
SemrushBot 194
MajesticBot 189
Other bot 163
Safari 131
+28 more crawlers
Canonical sitemap (linked from robots.txt)
ClaudeBot 847
AhrefsBot 607
Bingbot 111
GPTBot 95
Chrome 33
PetalBot 24
Other bot 15
Safari 7
+15 more crawlers
AI context layer
Structured hints for LLMs and AI agents
LLM-readable site index (llmstxt.org spec)
Chrome 37
Other bot 24
PetalBot 19
AhrefsBot 17
Firefox 11
Amazonbot 10
Barkrowler 10
Safari 7
+16 more crawlers
Machine-readable endpoint catalog (JSON-LD)
Chrome 58
AhrefsBot 17
PetalBot 15
Other bot 7
Safari 4
SemrushBot 3
MozBot 2
Screaming Frog 2
+11 more crawlers
Entity map — people, concepts, and organizations (EntityMap v1.0)
Chrome 59
AhrefsBot 18
PetalBot 16
Other bot 8
Safari 8
GPTBot 3
SemrushBot 3
MozBot 2
+12 more crawlers
Raw content layer
.md / .mdx page requests — AI agents preferring plain text
117 requests
Chrome 20
PetalBot 18
AhrefsBot 18
Barkrowler 10
Other bot 9
Safari 9
curl 5
SemrushBot 4
+16 more crawlers
71 requests
Chrome 28
AhrefsBot 14
Other bot 7
curl 4
Firefox 3
PetalBot 2
GPTBot 2
Safari 2
+9 more crawlers
69 requests
Chrome 15
AhrefsBot 14
Other bot 6
curl 5
Safari 5
Firefox 4
GPTBot 3
GoogleOther 2
+12 more crawlers
69 requests
AhrefsBot 14
Chrome 14
Other bot 7
curl 6
Safari 6
Firefox 4
GPTBot 3
Screaming Frog 2
+12 more crawlers
47 requests
Chrome 15
AhrefsBot 10
Other bot 5
Googlebot 5
GPTBot 2
PetalBot 2
SemrushBot 1
Amazonbot 1
+6 more crawlers
32 requests
Chrome 10
AhrefsBot 8
Safari 3
PetalBot 2
Other bot 2
GoogleOther 1
Amazonbot 1
MajesticBot 1
+4 more crawlers
31 requests
Chrome 9
AhrefsBot 8
Safari 3
PetalBot 2
Other bot 2
Firefox 1
Amazonbot 1
MajesticBot 1
+4 more crawlers
31 requests
Chrome 10
AhrefsBot 8
Safari 3
PetalBot 2
Other bot 2
Amazonbot 1
MajesticBot 1
GPTBot 1
+3 more crawlers
31 requests
Chrome 11
AhrefsBot 8
PetalBot 2
Other bot 2
Safari 2
Amazonbot 1
MajesticBot 1
GPTBot 1
+3 more crawlers
29 requests
Chrome 10
AhrefsBot 8
PetalBot 2
Firefox 1
Amazonbot 1
MajesticBot 1
GPTBot 1
SemrushBot 1
+4 more crawlers
29 requests
Chrome 10
AhrefsBot 8
PetalBot 2
Other bot 2
Safari 1
Amazonbot 1
MajesticBot 1
GPTBot 1
+3 more crawlers
22 requests
Scanner / probe 7
AhrefsBot 6
PetalBot 2
Barkrowler 2
Amazonbot 1
Googlebot 1
GoogleOther 1
GPTBot 1
+1 more crawlers
14 requests
Scanner / probe 7
PetalBot 5
GPTBot 2
3 requests
Scanner / probe 2
MajesticBot 1
1 requests
curl 1
All crawlers — combined across every endpoint
ClaudeBot 1,700
AhrefsBot 1,068
Chrome 576
Other bot 262
SemrushBot 219
PetalBot 211
MajesticBot 203
OpenAI SearchBot 202
Safari 193
Bingbot 166
How this works: Every request to this site passes through a Cloudflare Worker that logs path, User-Agent, IP, and CF metadata to D1. A periodic snapshot job aggregates those rows into KV, which this page reads. Bot classification uses UA strings, CF reverse-DNS verification, and ASN fallbacks.
What each layer means: The discovery layer is what every crawler checks before indexing — robots.txt signals intent, sitemaps provide the URL map. The AI context layer is purpose-built for LLMs: llms.txt gives a plain-text orientation and page list, ai-catalog.json enumerates machine-readable endpoints, and entitymap.json declares structured entity relationships (people, concepts, organizations) following the EntityMap v1.0 spec. The raw content layer (.md requests) is agents bypassing HTML parsing entirely.
What each layer means: The discovery layer is what every crawler checks before indexing — robots.txt signals intent, sitemaps provide the URL map. The AI context layer is purpose-built for LLMs: llms.txt gives a plain-text orientation and page list, ai-catalog.json enumerates machine-readable endpoints, and entitymap.json declares structured entity relationships (people, concepts, organizations) following the EntityMap v1.0 spec. The raw content layer (.md requests) is agents bypassing HTML parsing entirely.