Crawl Intelligence Pipeline

What it does

Every HTTP request to leedriggers.com is intercepted by Cloudflare Worker middleware before the page handler runs. The middleware extracts the user-agent string, request path, country code, ASN, and Cloudflare’s cf-verified-bot flag, then writes a row to a D1 table (crawl_log).

A scheduled snapshot job (/api/snapshot) runs incrementally — it reads only rows newer than the last run, classifies each request into a named bot and group, and merges the results into persistent KV aggregates. Public dashboards read from KV only. No D1 on page load.

The incremental cursor pattern

The key to scalability is snapshot:meta.lastRun. Each run records its own timestamp to KV. The next run queries D1 for rows with ts > lastRun. This means:

• The first ever run backfills all history (lastRun defaults to 0)
• Subsequent runs process only new rows — query time stays constant as the table grows
• Resetting lastRun to 0 triggers a full historical recompute

No row cap, no pruning of crawl_log. The table grows indefinitely.

What gets tracked

All-time aggregates (indefinite retention):

• snapshot:totals — hit count, first/last seen, country breakdown per bot
• snapshot:daily — day × bot hit counts, kept forever (no pruning)
• snapshot:ua-registry — every UA string seen per bot, with first-seen timestamp

Rolling windows:

• snapshot:perpath-daily — path × date × bot, 90-day rolling, capped at 300 paths
• snapshot:ai-paths-daily — same for machine-readable endpoints only, 90-day window

Derived keys:

• snapshot:ua-new — UA strings first seen in last 30 days (surfaced on dashboards for new crawler detection)
• snapshot:unknown-uas — UA strings that didn’t match any known classifier
• snapshot:ai-paths — per-bot hit counts, first/last seen for machine-readable endpoints

What I can see now

• Which search engines, AI crawlers, and autonomous agents hit the site, and how often
• Per-page bot breakdown: which bots crawl /about/ vs /llms.txt/ vs blog posts
• New UA strings as they appear — early warning system for new crawlers entering the market
• Which machine-readable endpoints (/llms.txt, /entitymap.json, /ai-catalog.json) are being fetched and by whom

KV size constraint

KV values max out at 25MB. Per-path daily data grows the fastest — hence the 300-path cap and 90-day rolling window. All-time totals and daily counts are much smaller in practice. The site has never approached the limit, but the cap is a real constraint to watch as traffic scales.