Bot Detection Techniques That Work at Scale

A survey of 300 enterprise leaders across IT security, engineering, infrastructure, and SRE found that 79 percent are confident they can detect bots, yet only 23 percent have a proactive strategy. Respondents estimated that AI bots generate about 17 percent of their own traffic. That figure is not a measurement of all automated traffic, and it should not be subtracted from Imperva’s 53 percent as a single 36-point gap. Only 33 percent said a WAF or bot-detection product blocked more than half of AI bot traffic in the prior 12 months.

Detection that still works at majority-bot volume is not a tighter User-Agent rule. On Cloudflare it starts by enabling Bot Management and acting on Definitely Automated, Likely Automated, and Verified bots. Search, Agent, and Training presets are the later AI-crawler policy layer. On Akamai it starts in monitor mode. AWS WAF Bot Control starts at the Common inspection level of the AWSManagedRulesBotControlRuleSet rule group, with rule actions overridden to Count. Fastly starts by choosing pre-cache or post-cache inspection and enabling Bot Management on the service. Identity and sequence fields on every request row, kept long enough to classify automated requests from LLM crawlers, are what make those decisions auditable.

Signature and User-Agent Checks Fail First

Robots.txt is a voluntary protocol. A crawler built to ignore it will ignore it. User-Agent strings are self-declared. A short script can claim to be Chrome or Googlebot and pass any filter that trusts the header. At majority-bot volume, that check is a first cut, not a detector.

The failure is worse for AI crawlers and agents than for classic scrapers. User-agent blocking misses most AI agents because many now run through real browsers. A UC Davis preprint from May 2026 (FP-Agent, arXiv) tested seven AI browsing agents on an instrumented site. Cloudflare’s free AI Crawl Control and Block AI Bots path stopped Manus’s browsing agent, which self-identified in the verified-bot directory. A classifier trained on typing, scroll, and mouse features on that site flagged all seven.

Those mouse, scroll, and typing features are not present in CDN or WAF request logs. Production client-side detections are. Cloudflare Precursor, which supersedes JavaScript Detections, contributes to bot score as JavaScript Fingerprinting. Session state stays on the cf_clearance cookie. Akamai Bot Manager Premier injects a script for behavioral detection. AWS WAF Bot Control Targeted expects a JS or Mobile SDK. Fastly Bot Management offers optional client-side detections that operators embed as a script tag in the page head, and they work with both pre-cache and post-cache inspection. If-then WAF rules that assume linear request sequences should be expected to miss agentic traffic that plans and retries.

Cloudflare’s policy split is by what a bot does on the site. Its controls, live since 1 July 2026, expose three AI use cases as presets for every customer including the Free tier: Search, Agent, and Training. Agent covers chat fetchers such as ChatGPT-User and browser-use agents (Gemini or Claude driving Chrome). Search is the class Cloudflare keeps allowed by default, and Cloudflare Radar’s robots.txt data shows Training crawlers such as GPTBot, CCBot, and anthropic-ai are the most widely disallowed.

Those three are the configurable presets, not the whole taxonomy. Cloudflare’s BotBase classifies each bot across eleven behaviors, from Transact and Data Collection to Ads Verification and Feed Fetching, and a bot can carry more than one. A second axis is on the way: Cloudflare has said it is building controls for content use — immediate, reference, and full — meant to combine with those classes, so a rule can allow Search up to reference and no further. It is also testing a use signal in robots.txt, extending Content Signals, that carries the same three values.

Two separate changes land on 15 September 2026. New Cloudflare zones default to blocking Training and Agent on pages that display ads, with single-purpose Search still allowed; those defaults apply only to domains onboarding after that date. Separately, mixed-purpose Search and Training crawlers such as Googlebot, Applebot, and BingBot begin following the most restrictive rule, so a Training block also blocks those crawlers — and that change reaches every customer already blocking Training, including through the legacy Block AI bots preset, not only new zones. Existing zones can set Search, Agent, and Training presets now. Operators who do not want mixed-purpose crawlers swept into an existing Training block can mark in Security settings, any time before 15 September, that they want no change to Training crawlers that also crawl for Search.

For properties where bot traffic is not yet a measurable cost line, the vendor’s first bot-management control plus an edge rate limiter still covers most exposure. The log techniques below are for teams where a misclassified bot has a measurable cost in origin egress or generative engine optimization (GEO) visibility.

Verify Identity Instead of Trusting a Header

Identifying, classifying, and controlling automated traffic starts with the CDN’s verified-bot catalog and managed categories. On Cloudflare that catalog is BotBase, live since 1 July 2026 for Enterprise Bot Management: a searchable directory of every tracked bot, the behaviors it is classified under, and a detection ID an operator can copy straight into a Security rule. Cloudflare has said it will expand BotBase from visibility into a control centre later this year. DNS, CIDR, and signatures are how an operator tests a catalog entry against the request in front of it, and how the log warehouse reconstructs an edge decision. Without that split, allow-lists fail when a spoofed “good” crawler gets through, and block-lists fail when a partner tool that looks unusual gets challenged.

Reverse DNS and Published IP Ranges

Major search engines still support a reverse-then-forward DNS loop. A reverse lookup on the client IP should land on a publisher-owned domain. A forward lookup on that domain should return the original IP. If the loop fails, treat the request as unverified even when the User-Agent names a trusted crawler.

Many operators also publish CIDR ranges. Google, OpenAI, and Common Crawl do this. Coverage is uneven. Anthropic published a public IP list for its crawlers in 2026, so older runbooks that assume no list exists are already stale. Google documents both the DNS loop and a match against published IP lists as supported methods. CIDR matching is the usual path at volume. Neither method covers residential-proxy farms. Those IPs sit on clean residential ASNs. Farms pair those residential exits with CAPTCHA-solving services and human solver farms, so a captcha pass is a poor farm detector. Those clients also often match a real browser’s JA4. Catching that class depends on JavaScript session signals plus request history. JA4 and ASN reputation are not decisive there.

Cryptographic Request Signatures

RFC 9421 (2024) defines HTTP Message Signatures as a generic primitive. Web Bot Auth is an IETF draft profile on that primitive: a participating bot signs requests and the receiving edge verifies them. Cloudflare and Akamai verify Web Bot Auth in production. AWS WAF Bot Control treats a verified Web Bot Auth label as an exception inside that rule group, on CloudFront since Version_4.0 and on Regional resources as of Version_6.0. That exception is not a web-ACL-wide allow. Fastly classifies bots with its own category fields, not Web Bot Auth.

OpenAI signs requests from ChatGPT Work’s Cloud browser. GPTBot, OAI-SearchBot, and ChatGPT-User identify with User-Agent strings and published IP lists. The IETF webbotauth working group is standardizing cryptographic authentication for automated clients. Until most crawlers sign, unverified traffic that claims a famous User-Agent should stay unclassified rather than inherit the trusted crawler’s policy.

TLS Fingerprints and Behavior Catch What Headers Miss

TLS fingerprinting in bot detection looks at cipher suites and extensions in the handshake, not at the User-Agent string. JA4 is a structured a_b_c identifier. Part b is a truncated SHA-256 of sorted, GREASE-filtered cipher suites. Part c is a truncated SHA-256 of sorted extensions with SNI and ALPN removed, then signature algorithms in wire order. That structure is why JA4 survives the extension-order randomization that made JA3 unreliable. A request that claims to be Chrome on iOS but presents a Python or curl handshake is unverified regardless of the header.

Headless Chrome that uses the same browser engine and OS as a headed browser often matches that browser’s TLS fingerprint. Analysts usually catch that traffic through missing JavaScript artifacts or mechanical timing, not JA4 alone. Reverse DNS verification and JA4 fingerprints belong in the same request row so analysts can stack signals instead of treating any one of them as decisive.

Lab classifiers remain useful when identity cannot be proved. FP-Agent’s mouse and scroll traces are not in CDN rows. Production client-side results are, as bot-score source fields such as JS Fingerprinting. Mechanical request intervals still mark unsophisticated scrapers, not a headless browser that fetches CSS and JavaScript the way a person would.

Classification at Scale Needs Months of Full Request History

A bot manager at the edge answers allow, challenge, or block on this request. That decision is necessary. It is not a classification over time. Retention on enforcement-product dashboards runs from three hours on AWS WAF’s sampled-requests view to 90 days on Akamai Security Center, with Cloudflare Bot Analytics at 30 days — and Cloudflare’s view is sampled at 1 to 10 percent on top of being short. Windows that short cannot show whether a rule change last quarter reduced scraping, or whether it started blocking a crawler that sends referrals.

Purpose-based classification needs history. IAB Tech Lab’s May 2026 Bot and Crawler Management Guidance, written by the CoMP working group, evaluates crawlers across five value dimensions. Public comment on that guidance closed on 25 June 2026. Traffic Return (referral) is one of those dimensions, not the whole test. A crawler may look like an ally for one quarter and drift the next. Teams making that call need unsampled CDN and WAF rows that outlast the enforcement dashboard’s window, with each vendor’s detection field on the same row as identity.

Bot scores do not share a scale, and not every CDN exposes one. Cloudflare uses 1 for bot and 99 for human; 0 means the request was not evaluated and does not mean safe. Akamai’s Bot Manager score uses 0 for human and 100 for bot, and Client Reputation is a separate field on a 1 to 10 scale. A shared threshold such as bot_score below 30 inverts the two vendors and also catches Cloudflare’s unevaluated 0. Fastly classifies with a category string, fastly.bot.category, plus a set of is_* boolean flags. AWS WAF Bot Control writes labels in WAF logs, not a numeric score.

Sampling is a cost choice, not a universal default. On Cloudflare, HTTP Logpush jobs default to every event (sample_rate 1). BotScore, JA4, and JSDetectionPassed are Bot Management fields that must be selected on the http_requests job; VerifiedBotCategory must be selected too but carries no Bot Management entitlement. An unsampled access-log job can still leave those columns empty. Akamai DataStream takes a sampling percentage from 1 to 100, and for streams configured through Property Manager the DataStream behavior there always overrides the stream’s own sampling rate. CloudFront real-time logs require a SamplingRate from 1 to 100; 100 keeps every event. CloudFront standard access logs are unsampled. AWS asks operators to treat both CloudFront log types as best-effort rather than a complete accounting of every request.

A 1 percent or 10 percent keep rate is fine for uptime charts and a poor fit for distributed probes. A sleeper botnet that sends one request every few minutes from thousands of IPs looks like noise in a sampled set. When teams keep CDN and WAF logs for only 7 to 30 days, the probing phase is often gone before an investigation starts. Security and performance teams already share the same CDN log stream. Bot detection uses that stream too. Analysts still need to retain WAF logs long enough to tune rules and reconstruct campaigns that never tripped a rate limit.

What a Pilot Should Prove

A useful pilot does not prove that block counts went up. It proves that every request is kept and that rows from each CDN share a schema. Cloudflare and Akamai expose numeric bot scores with opposite polarity. Fastly classifies with a category string and boolean flags. AWS WAF Bot Control writes labels, not a 1 to 99 score. Identity fields (verified DNS, published CIDR, signature status, JA4) should sit next to those detection fields, plus WAF action, cache hit, and origin bytes. Queries should return on a representative volume in seconds, not after a restore from cold storage. Leadership can then defend which crawlers cost origin money and which ones send referrals.

FOX ran comparable log volumes at Super Bowl scale: a 17.4 GB/second peak ingest rate, about 200 TB ingested, 55 billion records, and a p50 query response time of 0.481 seconds on summary-table dashboards querying thirty-minute windows. Navy Federal Credit Union, on Akamai TrafficPeak, reports shortening time to resolution from hours or days to seconds and minutes. Those figures show the ingest scale the platform has handled. The same tests apply on a smaller property: ingest that matches each CDN’s export defaults, hot data that outlasts the enforcement window, and a path from an edge decision to the request rows that justified it.

Hydrolix is built for that data layer. Fifteen months of unsampled bot data across Akamai, Fastly, Cloudflare, CloudFront, and Tencent is the default retention for bot classification, and full-fidelity security telemetry stays queryable from existing SIEM workflows, including directly from Splunk via SPL. Edge enforcement still happens in the bot manager. Analysts use the retained rows to decide whether that enforcement was the right call.

Book a Walkthrough

Hydrolix can walk through these detection questions on a representative CDN volume, including which crawlers cost origin money and which ones send referrals. Book a demo.