# How This Site Was Built — leedriggers.com

> A topic-by-topic record of every major decision, system, and iteration that went into building leedriggers.com — stack choices, crawl intelligence, performance monitoring, and AI readiness.
> First-person, specific, honest about tradeoffs.

> Source: https://leedriggers.com/notes/changelog/
> Generated: 2026-07-17

---

## Table of Contents

- **Foundation** — Core stack choices — runtime, framework, hosting.
  - [Edge Architecture — Astro 5 on Cloudflare Workers](#edge-architecture) · 2026-06-25
- **Identity & Design** — Visual identity, theme, typography, favicon.
  - [Site Identity & Design](#site-identity-design) · 2026-06-29
- **Crawl Intelligence** — Bot monitoring, classification, and pipeline architecture.
  - [Crawl Intelligence Pipeline](#crawl-intelligence-pipeline) · 2026-06-26
  - [Bot Classification System](#bot-classification) · 2026-06-30
- **Performance** — Real user monitoring and Core Web Vitals.
  - [Real User Monitoring — Core Web Vitals](#real-user-monitoring) · 2026-06-27
- **Content & Discovery** — RSS, search, blogroll, and personal pages.
  - [Building the Changelog — Docs From Git Archaeology](#building-the-changelog) · 2026-07-08
  - [Content & Discovery Layer — RSS, Search, Blogroll](#content-discovery-layer) · 2026-07-07
  - [/stack/, /uses/, /blogroll/ — Personal Infrastructure Pages](#stack-uses-pages) · 2026-07-07
- **AI Readiness** — Machine-readable endpoints, entity map, and crawl surface.
  - [AI Readiness & Crawl Surface](#ai-readiness-crawl-surface) · 2026-07-06
  - [Entity Map](#entity-map) · 2026-07-07

---

## Foundation

*Core stack choices — runtime, framework, hosting.*

### Edge Architecture — Astro 5 on Cloudflare Workers

**Date:** 2026-06-25  
**Category:** foundation  
**URL:** https://leedriggers.com/notes/changelog/edge-architecture/

> Chose Astro 5 SSR with the Cloudflare Workers adapter as the runtime — no origin server, every request processed at the edge, static pages prerendered at deploy and dynamic routes served by Workers.

### Why Astro 5

I evaluated Next.js, SvelteKit, Remix, and static-only generators before settling on Astro. The deciding factor was the islands architecture: zero JavaScript ships by default, and you opt into client-side JS only where it's genuinely needed. For a personal site where most pages are read-only content, this matters. The runtime footprint starts at nothing rather than starting heavy and hoping tree-shaking does its job.

Astro 5 added content layer improvements and a cleaner SSR story. The Cloudflare adapter (@astrojs/cloudflare) is first-party and well-maintained.

### Why Cloudflare Workers (not Vercel, not Netlify)

The real reason is the data layer. I wanted to run bot classification middleware before the page handler runs — no round-trip to an origin, no cold start waiting on a Lambda. Cloudflare Workers run in V8 isolates with sub-millisecond startup time. KV reads are co-located with the edge node, typically under 1ms. D1 (SQLite at the edge) adds ~10ms for writes.

That combination means: every HTTP request can be logged to D1, classified, and have its data written to KV for dashboard reads — all without adding perceptible latency to the page response.

What I gave up: The Cloudflare adapter is stricter than Node. Some npm packages that depend on Node built-ins (fs, crypto, stream) don't run at the edge. This forced me to use edge-compatible alternatives or vendor small utilities. The constraint is real but manageable.

### Request flow

1. Browser sends request to leedriggers.com
2. DNS resolves to Cloudflare's network → nearest edge node
3. Worker middleware intercepts: extracts UA, path, country, ASN, cf-verified-bot header
4. Log entry written to D1 (crawl_log table) — async, doesn't block response
5. Request routed to Astro handler:
   - Static pages: served from prerendered HTML in Workers Assets
   - Dynamic routes (/crawl-stats/, /api/*): Astro SSR handler runs
6. Response returned from edge — no origin server involved

### Tradeoffs

| Decision | Benefit | Cost |
|---|---|---|
| Astro islands | Zero JS by default | Extra config when you do need client JS |
| Cloudflare Workers | Sub-ms KV reads, co-located D1 | No Node built-ins; adapter constraints |
| SSR at edge | Dynamic routes, real-time data | Build pipeline more complex than pure static |
| Prerender static pages | Fast TTFB for content | Must be deterministic at build time |

---

## Identity & Design

*Visual identity, theme, typography, favicon.*

### Site Identity & Design

**Date:** 2026-06-29  
**Category:** identity  
**URL:** https://leedriggers.com/notes/changelog/site-identity-design/

> Built the site's visual identity: LD monogram SVG favicon with light/dark inversion, no-flash dark mode via localStorage + inline script, system font stack with zero external requests, and a CrawlWidget in every page footer.

### Favicon: LD monogram

The default Astro favicon is a placeholder. I built an SVG favicon with the LD initials as a monogram: dark background, white letterforms. The SVG inverts for light mode via @media (prefers-color-scheme: dark).

Why SVG over PNG/ICO: scales to any size without blur, adapts to light/dark without multiple image files, under 1KB, supported by all modern browsers with a fallback 1×1 transparent ICO.

### Dark mode: no flash

The hardest part of dark mode is preventing the flash of wrong theme — when the page loads in light mode for a fraction of a second before JavaScript reads the stored preference and switches.

The solution is an inline <script> in the <head> that runs synchronously before the browser paints anything. CSS custom properties on :root[data-theme="dark"] handle all color switching. The toggle button updates localStorage and flips data-theme. No JavaScript frameworks, no CSS-in-JS.

### Typography: system fonts

No Google Fonts. No Typekit. No external font requests at all.

Font stack: system-ui, -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Arial, sans-serif.

On macOS: San Francisco. On Windows: Segoe UI. On Android: Roboto. Everywhere else: Arial.

What this eliminates: a DNS lookup to fonts.googleapis.com, a TCP/TLS connection, a render-blocking resource request, and the font file request itself. In practice, system fonts look correct and familiar on every platform.

Monospace: ui-monospace, 'Cascadia Code', 'Fira Code', monospace.

### CrawlWidget

Every page footer includes the CrawlWidget: a small component that shows a 14-day bot activity sparkline and a live breakdown of bot groups hitting the site. It reads from KV on each SSR page load — sub-millisecond — and renders with no client-side JavaScript.

---

## Crawl Intelligence

*Bot monitoring, classification, and pipeline architecture.*

### Crawl Intelligence Pipeline

**Date:** 2026-06-26  
**Category:** crawl  
**URL:** https://leedriggers.com/notes/changelog/crawl-intelligence-pipeline/

> Built a D1 + KV bot monitoring pipeline that logs every HTTP request, classifies crawlers by UA and ASN, and surfaces aggregated data on public dashboards — with zero D1 reads on page load.

### What it does

Every HTTP request to leedriggers.com is intercepted by Cloudflare Worker middleware before the page handler runs. The middleware extracts the user-agent string, request path, country code, ASN, and Cloudflare's cf-verified-bot flag, then writes a row to a D1 table (crawl_log).

A scheduled snapshot job (/api/snapshot) runs incrementally — it reads only rows newer than the last run, classifies each request into a named bot and group, and merges the results into persistent KV aggregates. Public dashboards read from KV only. No D1 on page load.

### The incremental cursor pattern

The key to scalability is snapshot:meta.lastRun. Each run records its own timestamp to KV. The next run queries D1 for rows with ts > lastRun. This means:

- The first ever run backfills all history (lastRun defaults to 0)
- Subsequent runs process only new rows — query time stays constant as the table grows
- Resetting lastRun to 0 triggers a full historical recompute

No row cap, no pruning of crawl_log. The table grows indefinitely.

### What gets tracked

All-time aggregates (indefinite retention):
- snapshot:totals — hit count, first/last seen, country breakdown per bot
- snapshot:daily — day × bot hit counts, kept forever (no pruning)
- snapshot:ua-registry — every UA string seen per bot, with first-seen timestamp

Rolling windows:
- snapshot:perpath-daily — path × date × bot, 90-day rolling, capped at 300 paths
- snapshot:ai-paths-daily — same for machine-readable endpoints only, 90-day window

Derived keys:
- snapshot:ua-new — UA strings first seen in last 30 days
- snapshot:unknown-uas — UA strings that didn't match any known classifier
- snapshot:ai-paths — per-bot hit counts, first/last seen for machine-readable endpoints

### KV size constraint

KV values max out at 25MB. Per-path daily data grows the fastest — hence the 300-path cap and 90-day rolling window. All-time totals and daily counts are much smaller in practice.

---

### Bot Classification System

**Date:** 2026-06-30  
**Category:** crawl  
**URL:** https://leedriggers.com/notes/changelog/bot-classification/

> Built botClassify.js and botGroups.js — a UA-string-first classifier mapping 60+ known crawlers to named bots and groups, with ASN-based fallback for bots that spoof or omit user-agent strings.

### The problem with generic classification

Early versions of the crawl log just recorded raw UA strings. Useful for raw analysis, but useless for aggregation or trend visualization. You can't answer "are AI crawlers increasing?" if every GPTBot version is a separate row.

The classifier solves this by mapping UA strings to canonical bot names (GPTBot, Googlebot, AhrefsBot) and then into groups that stay stable as new versions appear.

### Two modules, one source of truth

botClassify.js — the classifier. Takes a UA string, AS org name, path, and verified-bot flag. Returns a canonical bot name string. Checks UA patterns in priority order: verified bots first, known UA substrings next, ASN fallback last. Falls through to "Other bot" if nothing matches.

botGroups.js — the group registry. Maps every canonical bot name to a group. Used consistently across snapshot.js, CrawlWidget, /crawl-stats/, and /dashboard/. Changing a group assignment here changes it everywhere.

### The bot groups

| Group | Examples |
|---|---|
| Search | Googlebot, Bingbot, Applebot, DuckDuckBot |
| AI Crawlers | GPTBot, ClaudeBot, PerplexityBot, Gemini |
| SEO Tools | AhrefsBot, SemrushBot, Majestic |
| Social | Twitterbot, LinkedInBot, Slackbot |
| Scanners | Shodan, Censys, Nuclei |
| Headless | Playwright, Puppeteer, Selenium |
| Dev Tools | curl, wget, Python HTTP libraries |
| ASN-identified | Cloud ranges with generic UAs |
| Unknown | Anything that slips through |

### ASN fallback

Some bots send completely generic UAs (Mozilla/5.0 with nothing else) or no UA at all. The ASN fallback matches on known datacenter IP ranges — AWS, GCP, Azure, DigitalOcean — to classify them as machine traffic even without a useful UA string.

Cloudflare's cf-verified-bot header is used as a confidence signal on top of UA matching, not as the primary classifier.

### Handling new crawlers

New AI crawlers appear regularly. The snapshot:ua-new KV key surfaces UA strings first seen in the last 30 days. When a new string appears in Unknown, I add a pattern to botClassify.js and the next snapshot run re-classifies historical rows for that UA.

---

## Performance

*Real user monitoring and Core Web Vitals.*

### Real User Monitoring — Core Web Vitals

**Date:** 2026-06-27  
**Category:** performance  
**URL:** https://leedriggers.com/notes/changelog/real-user-monitoring/

> Integrated the web-vitals library to capture LCP, INP, CLS, FCP, and TTFB from real browser sessions, beaconing to /api/cwv → D1 → KV snapshot pipeline with p50/p75/p95 percentiles and indefinite retention.

### Why RUM instead of synthetic testing

Lighthouse and PageSpeed Insights give you a single synthetic measurement in a controlled environment. Real User Monitoring captures what actual visitors experience — on their devices, their connections, their browser load states. For Core Web Vitals, Google uses field data (CrUX), not lab data, to determine pass/fail thresholds. RUM matches what Google actually measures.

The web-vitals library (Google's own) reports the same metrics Chrome uses for CrUX, so my data is directly comparable to what affects search rankings.

### Data flow

1. web-vitals library initializes on page load (client-side, ~2KB gzipped)
2. Metrics fire as they become available: FCP and LCP on paint, CLS on layout shift, INP on interaction, TTFB immediately
3. Each metric beaconed via sendBeacon to /api/cwv — non-blocking, fires even on page unload
4. Worker handler writes to D1 cwv_log: metric name, value, page path, timestamp
5. cwv-snapshot job runs on schedule: reads new rows incrementally, computes p50/p75/p95, writes to KV
6. /page-speed/ dashboard reads from KV only — no D1 on page load

### Why sendBeacon

fetch from an unload handler is unreliable — browsers cancel in-flight requests when navigating away. sendBeacon is designed exactly for this: it queues a small payload to be delivered asynchronously, survives page unload, and doesn't block the next navigation.

### Percentile choice: p75

Google uses p75 as the threshold for Core Web Vitals pass/fail: your 75th percentile user needs to have a Good experience. I track p50, p75, and p95 but use p75 as the primary metric to match Google's standard.

### Reservoir sampling

Raw CWV values are capped at 500 samples per metric. New values replace random existing values once the cap is hit. This gives statistically valid percentile estimates with bounded storage.

### Privacy

No IP address. No cookies. No session ID. No user identifier of any kind. Only the page URL and metric value are collected.

### CWV thresholds

| Metric | Good | Needs Improvement | Poor |
|---|---|---|---|
| LCP | ≤ 2.5s | 2.5s – 4.0s | > 4.0s |
| INP | ≤ 200ms | 200ms – 500ms | > 500ms |
| CLS | ≤ 0.1 | 0.1 – 0.25 | > 0.25 |
| FCP | ≤ 1.8s | 1.8s – 3.0s | > 3.0s |
| TTFB | ≤ 800ms | 800ms – 1800ms | > 1800ms |

---

## Content & Discovery

*RSS, search, blogroll, and personal pages.*

### Building the Changelog — Docs From Git Archaeology

**Date:** 2026-07-08  
**Category:** content  
**URL:** https://leedriggers.com/notes/changelog/building-the-changelog/

> Built this changelog itself: a Keystatic-managed content collection with topic-based entries, a /changelog.md export for AI agents, a dedicated changelog.xml sitemap — with entry dates verified against the actual git history.

### The idea

Most changelogs are chronological commit noise. This one is organized by topic — each entry covers one system with the decisions, tradeoffs, and context that a commit log can't carry.

### How it's built

- Astro content collection (title, date, category, summary, draft) managed in Keystatic
/notes/changelog/ index with color-coded category badges; /changelog/[slug]/ entry pages
- /changelog.md — the whole changelog as one markdown document for AI agents
- /changelog.xml — dedicated sitemap submitted to Google Search Console so changelog indexation can be monitored in isolation

### Git archaeology — the dates were wrong

The first draft carried estimated dates spread across 2025. Overlaying with git log showed the truth: the entire site was built in 13 days — June 25 to July 7, 2026 — across 102 commits and 58 pull requests. Every entry date was corrected to the day its work actually landed.

What the overlay surfaced: day 1 took six attempts to make the Cloudflare adapter deploy; CWV capture landed five days before its dashboard (instrument first, visualize later); the blogroll's fetch-per-request mistake was fixed the same day it shipped; owner-traffic tagging took until day 10.

### Backlog as drafts

Future work is drafted as unpublished changelog entries (draft: true), so the roadmap and the history share one system. When a backlog item ships, its draft flips to published with the real date.

---

### Content & Discovery Layer — RSS, Search, Blogroll

**Date:** 2026-07-07  
**Category:** content  
**URL:** https://leedriggers.com/notes/changelog/content-discovery-layer/

> Built the site's content discovery infrastructure: dual RSS feeds (blog + notes), Pagefind client-side search with post-build indexing, and a static blogroll with live descriptions fetched at build time.

### RSS feeds

Two feeds, intentionally separate:

/rss.xml — blog posts only. The primary feed — durable content, longer-form writing. Listed in RSS autodiscovery <link> in the page <head> so any RSS reader can find it automatically.

/notes-rss.xml — notes only. Shorter observations, links, quick thoughts. Separate feed lets people subscribe to just that signal.

The reasoning for the split: blog posts and notes have different publishing frequencies and different expectations. Mixing them into one feed means blog readers get spammed with observations and note-subscribers miss longer posts.

### Pagefind — client-side search without the weight

Pagefind generates a search index at build time as a CLI step after astro build. The index is sharded into small binary files — only the shards relevant to a query are loaded, not the full index on page load.

| | Pagefind | Fuse.js |
|---|---|---|
| Index size on load | Sharded, load on query | Full JSON on page load |
| Build step required | Yes (CLI after build) | No |
| Accuracy | Good, weighted | Good, fuzzy |
| CDN dependency | None | None if self-hosted |
| Offline support | Yes | Yes |

Setup: data-pagefind-body on <main> scopes the index to page content, excluding nav, footer, and the CrawlWidget. Individual pages can opt out entirely with a noindex Layout prop.

### Blogroll with live descriptions

The blogroll at /blogroll/ lists sites I read regularly. Each entry shows the site's meta description, fetched from its homepage at build time using fetch() in Astro's component script. The fetch happens once per build, the result is baked into the HTML — no runtime requests, no latency on page load.

### Feed icon in nav and footer

The RSS/feed icon appears in the nav alongside search, and in the footer social cluster alongside LinkedIn, GitHub, and X. The icon is an inline SVG — no external image request.

---

### /stack/, /uses/, /blogroll/ — Personal Infrastructure Pages

**Date:** 2026-07-07  
**Category:** content  
**URL:** https://leedriggers.com/notes/changelog/stack-uses-pages/

> Built /stack/ with a three-tier data pipeline diagram, /uses/ documenting hardware and software across machines, networking, finance, and fitness, and /blogroll/ with build-time-fetched meta descriptions.

### /stack/ — the data pipeline diagram

The /stack/ page documents the full technical stack with a three-tier pipeline diagram showing how data flows:

Author tier (blue): Keystatic CMS → GitHub PR → astro build → Cloudflare Pages deploy

Serve tier (green): Browser → Cloudflare Edge → Worker middleware → Astro SSR or Workers Assets

Observe tier (amber):
- Worker logs to D1 → snapshot.js → KV → Dashboards
- web-vitals → sendBeacon → /api/cwv → D1 → cwv-snapshot.js → KV → /page-speed/

The diagram is pure CSS and HTML — no image, no canvas, no JS.

### /uses/ — gear and software

Machines: Asus Vivobook S (daily driver), Asus Chromebook Plus 514 (lightweight travel)
Peripherals: Das Keyboard mechanical, Logitech MX Master 3, Dell 24" monitor, Uplift standing desk, Topo ergo mat
Networking: Ubiquiti UniFi UDR7 router, U7 Lite AP, Starlink Mini (travel / backup)
Power & Home: EcoFlow Delta 2, Home Assistant Green
Software: Cursor IDE, Obsidian, Proton Mail, LastPass
Finance: Monarch Money, Betterment + Wealthfront, Robinhood + Fidelity
This site: Porkbun, Cloudflare Workers/D1/KV/R2, Astro 5, GitHub, Keystatic CMS
Fitness: Coros Apex 2 Pro GPS watch, Topo Athletic trail shoes

Listed on uses.tech.

### /blogroll/ — who I read

A curated list of sites and writers I follow. Each entry shows the site's meta description, fetched from its homepage at build time. No runtime requests. Managed in Keystatic as a singleton.

---

## AI Readiness

*Machine-readable endpoints, entity map, and crawl surface.*

### AI Readiness & Crawl Surface

**Date:** 2026-07-06  
**Category:** ai  
**URL:** https://leedriggers.com/notes/changelog/ai-readiness-crawl-surface/

> Built the machine-readable surface that AI crawlers and agents need to navigate and understand the site: llms.txt, ai-catalog.json, entitymap.json, and a public /crawl-surface/ dashboard showing what's being accessed and by whom.

### The problem AI crawlers have with most sites

HTML pages are built for browsers. An AI agent trying to understand a site has to parse navigation, ads, sidebars, and layout scaffolding to find the actual content. A 25KB blog post can be wrapped in 80KB of HTML. There's no standard way for an agent to discover what a site publishes, what it's about, or which endpoints are meaningful.

### What's in the crawl surface

Discovery layer — how crawlers find the site and understand its rules:
- /robots.txt — standard crawl rules with a Content-Signal block declaring AI permissions (search: yes, ai-input: yes, ai-train: no, use: reference)
- /sitemap-main.xml — canonical URL map for all public pages

AI context layer — structured context for agents:
- /llms.txt — plain-text site orientation (llmstxt.org spec)
- /ai-catalog.json — JSON endpoint catalog
- /entitymap.json — entity graph (EntityMap v1.0)

Content layer (planned):
- .md variants of content pages — plain text, ~5x smaller payload than HTML

### The /crawl-surface/ dashboard

/crawl-surface/ lists all machine-readable endpoints and, for each one, shows which bots are accessing it — with per-bot hit counts drawn from the crawl intelligence pipeline.

### Content-Signal in robots.txt

Content-Signal: search=yes; ai-input=yes; ai-train=no; use=reference

Lets AI systems know explicitly: use the content to answer questions, don't use it for training data, cite don't reproduce.

### Planned: Markdown content negotiation

1. Check Accept: text/markdown header — serve .md if present
2. UA-based fallback for known AI crawlers that don't set the header
3. Generate .md routes at build time alongside HTML pages

A typical blog post is ~85KB as HTML and ~16KB as markdown. At scale, this reduces bandwidth consumed by AI crawlers by roughly 5x.

---

### Entity Map

**Date:** 2026-07-07  
**Category:** ai  
**URL:** https://leedriggers.com/notes/changelog/entity-map/

> Implemented /entitymap.json (EntityMap v1.0) with 8 entities — Person, Organization, and 6 Concepts — with structured relations, hasChunks text extracts, and a human-readable /entitymap/ companion page.

### Why an entity map

Knowledge graphs are how Google understands entities and their relationships. For a personal site, declaring these relationships explicitly helps search engines and AI systems build an accurate picture without having to infer everything from prose.

The EntityMap v1.0 spec provides a structured format for self-declaring entities: their type, description, sameAs links to authoritative sources, relations to other entities, and hasChunks — actual text extracts that serve as grounding data for LLMs.

### The 8 entities

People:
- e_001 Lee Driggers (Person) — SEO professional, trail runner. sameAs LinkedIn.

Organizations:
- e_002 leedriggers.com (Organization) — this site. sameAs the canonical URL.

Concepts:
- e_003 Crawl Intelligence — the D1 + KV bot monitoring pipeline
- e_004 Edge SEO Architecture — running SEO logic at the CDN edge
- e_005 Real User Monitoring — CWV measurement pipeline. sameAs Wikidata.
- e_006 Bot Classification — the UA-string classifier
- e_007 Technical SEO — the broader discipline. sameAs Wikipedia.
- e_008 Ultra Running — trail and mountain running. sameAs Wikipedia.

### Relations

Relations use explicit predicates: AFFILIATED_WITH, AUTHORED_BY, INCLUDES, DEPENDS_ON, RELATES_TO, PART_OF, PRODUCED_BY.

Key connections:
- Lee Driggers AFFILIATED_WITH leedriggers.com, AUTHORED_BY Crawl Intelligence + Edge SEO Architecture
- Crawl Intelligence INCLUDES Edge SEO Architecture + Bot Classification
- Technical SEO INCLUDES Crawl Intelligence + Real User Monitoring
- Ultra Running AFFILIATED_WITH Lee Driggers

### hasChunks — grounding data for LLMs

Each entity includes hasChunks: short text extracts structured like retrieved documents: chunkId, text, sourceUrl, pageTitle, relevanceScore, contentType (definition, evidence, example).

### Two surfaces

/entitymap.json — machine-readable, cached for 24 hours, tracked in the crawl pipeline.
/entitymap/ — human-readable page with entity cards, type badges, and relation chips.

---

*End of changelog — leedriggers.com*