Bot Classification System
Built botClassify.js and botGroups.js — a UA-string-first classifier mapping 60+ known crawlers to named bots and groups, with ASN-based fallback for bots that spoof or omit user-agent strings.
The problem with generic classification
Early versions of the crawl log just recorded raw UA strings. Useful for raw analysis, but useless for aggregation or trend visualization. You can’t answer “are AI crawlers increasing?” if every GPTBot version is a separate row.
The classifier solves this by mapping UA strings to canonical bot names (GPTBot, Googlebot, AhrefsBot) and then into groups that stay stable as new versions appear.
Two modules, one source of truth
botClassify.js — the classifier. Takes a UA string, AS org name, path, and verified-bot flag. Returns a canonical bot name string. Checks UA patterns in priority order: verified bots first, known UA substrings next, ASN fallback last. Falls through to "Other bot" if nothing matches.
botGroups.js — the group registry. Maps every canonical bot name to a group. Used consistently across snapshot.js, CrawlWidget, /crawl-stats/, and /dashboard/. Changing a group assignment here changes it everywhere.
Sharing the group module prevents the common failure mode where the dashboard shows “AI Crawlers” but the snapshot job wrote the data under “AI” — silent mismatch that corrupts aggregates.
The bot groups
| Group | Examples |
|---|---|
| Search | Googlebot, Bingbot, Applebot, DuckDuckBot |
| AI Crawlers | GPTBot, ClaudeBot, PerplexityBot, Gemini |
| SEO Tools | AhrefsBot, SemrushBot, Majestic |
| Social | Twitterbot, LinkedInBot, Slackbot |
| Scanners | Shodan, Censys, Nuclei |
| Headless | Playwright, Puppeteer, Selenium |
| Dev Tools | curl, wget, Python HTTP libraries |
| ASN-identified | Cloud ranges with generic UAs |
| Unknown | Anything that slips through |
ASN fallback
Some bots send completely generic UAs (Mozilla/5.0 with nothing else) or no UA at all. The ASN fallback matches on known datacenter IP ranges — AWS, GCP, Azure, DigitalOcean — to at least classify them as machine traffic even without a useful UA string.
Cloudflare provides the cf-verified-bot header for crawlers that have verified their identity with Cloudflare’s bot management. This is used as a confidence signal on top of UA matching, not as the primary classifier — not all legitimate bots are verified.
Handling new crawlers
New AI crawlers appear regularly. The snapshot:ua-new KV key surfaces UA strings first seen in the last 30 days. When a new string appears in Unknown, I add a pattern to botClassify.js and the next snapshot run re-classifies historical rows for that UA.
The snapshot:unknown-uas key tracks unknown UAs with hit counts — high-volume unknowns get prioritized for classification.