Automated traffic is no longer an edge case; it is the baseline reality of the open internet. According to Imperva’s Bad Bot Report, automated traffic has accounted for more than half of all web traffic in recent years, but treating this volume as a monolith of “human vs. machine” is a strategic error. The binary distinction between a “good bot” (like Googlebot) and a “bad bot” (like a credential stuffer) isn’t easy to make for many bots. AI agents, scrapers, and partner crawlers now force engineering teams to evaluate traffic based on business intent, not simply “benign” versus “malicious.”
TL;DR
- Differentiating modern bots goes beyond identifying the actor. It’s also about the intent of their behavior (malicious versus benign) and the impact on the business. Even “benign” bot traffic can drive up costs, impact bandwidth, and have other negative effects on the bottom line. User-Agent strings are trivial to spoof, making validation via reverse DNS, IP ranges, or cryptographic signatures necessary for trusted traffic. However, User-Agent strings remain a valuable source of information for benign bots, since many bots make no attempt to hide their origin. AI crawlers and other bots have introduced a “gray” category of traffic that honors protocols but drives up origin costs and imposes latency penalties. Sampling log data makes it difficult for security teams to detect low-and-slow attacks, making full-fidelity data retention a requirement for accurate classification.
Moving Beyond Binary Classification
The traditional security model relies on a simple allowlist and blocklist approach. You allow known search engines because they bring SEO value, and you block known malicious signatures to prevent fraud. Such a model effectively collapses in an environment where AI training crawlers, competitive price scrapers, and copyright enforcement bots all exhibit different behaviors and offer different value propositions.
Differentiating good and bad bots is now an exercise in policy and economics as much as security. A crawler that scrapes your pricing might be “bad” if it belongs to a competitor, but “good” if it belongs to an aggregator ensuring your products are listed in a marketplace. Similarly, an AI bot might be desirable if it drives traffic through search citations (like OpenAI’s OAI-SearchBot) but undesirable if it merely scrapes content to train a model without attribution (like GPTBot).
Effective differentiation requires assessing traffic against three criteria:
- Identity: Can you cryptographically or structurally prove who the bot is?
- Behavior: Does the bot respect rate limits, robots.txt directives, and standard traversal patterns?
- Value: Does the cost of serving this bot (bandwidth, compute, cache processing) yield a return on investment?
Verifying Identity: Where Classification Starts
A common mistake in bot management is trusting the User-Agent header. The header is a self-reported identifier, which makes it one of the simplest signals for a malicious actor to spoof. A basic Python script can declare itself to be Googlebot/2.1 and bypass naive filters.
To accurately classify traffic, you must validate that the bot is who it claims to be.
Reverse and Forward DNS Lookups
Major search engines and service providers support verification through DNS. When a request claims to be from a major crawler, you can perform a reverse DNS lookup on the IP address to verify the domain (e.g., googlebot.com). You then perform a forward DNS lookup on that domain to ensure it resolves back to the original IP address. If the loop fails, the request should be treated as likely spoofed.
Publishing IP Ranges
Many legitimate bot operators, including Google, OpenAI, and Common Crawl, publish lists of the IP ranges (CIDR blocks) their crawlers use. Ingesting these lists allows for rapid verification at the network edge. However, this is not a universal standard, and coverage varies in maturity — Anthropic, for example, only began publishing a public IP list for its crawlers in 2026, so teams working from older guidance may still assume no such list exists and fall back on weaker signals like user agents or behavioral heuristics.
Cryptographic Signatures
The industry has largely settled on a model where bots carry a cryptographic passport. HTTP Message Signatures — finalized as RFC 9421 in 2024 — let a bot operator sign its requests with a private key, and the receiving server or WAF validates that signature against a public key directory. Cloudflare, Akamai, and OpenAI have already built support for this “Web Bot Auth” approach into production, and an IETF working group formed in 2026 is standardizing the discovery layer around it. This shifts the burden of proof from the defender, who previously had to guess whether an IP was malicious, to the bot operator, who must now cryptographically prove its identity.
Not every legitimate bot signs its requests yet, so solutions like Bot Insights still play a role: they give teams visibility into traffic patterns and volumes by bot category, which helps flag anomalies even before cryptographic verification is universal. As signing adoption grows, organizations will be able to build policies that automatically challenge or drop unauthenticated traffic claiming to be a major crawler, substantially reducing — though not fully eliminating — the “imposter bot” vector.
Analyzing Behavioral Signals
When identity cannot be verified (or when a verified bot behaves aggressively) engineers must rely on behavioral signals. Malicious bots often reveal themselves through technical anomalies in their request patterns that differ from standard browser behavior.
TLS Fingerprinting
Practitioners use TLS fingerprinting — primarily JA4 hashes today, since modern browsers now randomize the order of cipher suites and extensions in a way that made the older JA3 method unreliable — to identify the client application starting the connection. A standard web browser (Chrome, Firefox) negotiates its handshake using its own cipher suite and extension set; a script built on a library like Python Requests or curl will produce a distinctly different signature. (Headless browsers that run a real browser engine, such as current versions of Headless Chrome, typically produce the same TLS fingerprint as the full browser — they’re usually caught by other signals, like missing JavaScript execution artifacts, rather than by TLS fingerprinting.)
By analyzing these TLS fingerprints, you can identify traffic that claims to be a human user on an iPhone but starts a handshake that matches a Linux-based scraping tool, and flag the request for further scrutiny regardless of how legitimate the User-Agent string looks.
Request Ordering and Velocity
Humans browse the web with a level of chaotic timing. We pause to read, scroll at irregular speeds, and click clusters of links. Bots tend to be mechanically precise. They may request pages at exact one-second intervals or traverse a directory structure alphabetically (e.g., /product/a…, /product/b…) rather than following logical navigation paths.
High-velocity requests during non-peak hours often signal a scraper trying to clone a site or an inventory hoarder looking for limited-stock items. Analyzing the sequence of HTTP requests — such as a request for an HTML page that is not immediately followed by requests for the associated CSS and JavaScript assets — is a useful signal for catching unsophisticated bots, though it won’t catch headless-browser-based scrapers that render a page’s full assets the same way a real browser does.
The Risk of False Positives in Bot Classification
Aggressive differentiation strategies often introduce a secondary risk: misclassifying legitimate tools as malicious actors. Because modern verification relies heavily on IP reputation and behavioral signatures, legitimate services that behave atypically can trigger false positives. For example, Bing Site Scan is a tool used by webmasters to audit their own sites, but because its requests do not originate from the standard Bingbot IP ranges, strict allowlists often block it as a “fake Bingbot.”
These misclassifications can disrupt critical business workflows. If a security policy strictly blocks any requester that starts a high volume of requests without a verified search engine signature, you might inadvertently block a partner’s inventory synchronization tool or an accessibility audit service. Effective differentiation requires exception workflows and “graylisting” capabilities (temporarily challenging suspicious traffic with a cryptographic challenge rather than a hard block) to ensure that edge cases do not result in business interruptions.
The Gray Area of AI Crawlers
AI has complicated bot differentiation by introducing high-volume, well-behaved crawlers that may not provide direct business value. In one case Hydrolix has publicly described, a customer reported a six-figure overcharge from their ISP after AI-driven bot traffic drove a spike in bandwidth usage. An AI crawler might fetch millions of pages from your origin server, respecting your robots.txt but causing a massive spike in egress costs and compute usage.
Autonomous Crawling vs User-Initiated Retrieval
It is critical to distinguish between autonomous crawling and user-initiated retrieval. OpenAI, for example, separates its traffic into distinct categories: GPTBot performs autonomous crawling for model training, while ChatGPT-User identifies traffic triggered by a human asking ChatGPT to browse a live URL.
This distinction is vital for policy enforcement. Blocking GPTBot via robots.txt prevents that crawler from fetching your content going forward — though it doesn’t retroactively remove content already crawled, and it doesn’t address other paths (licensing deals, third-party datasets) by which content can end up in a training set. It also may not prevent ChatGPT-User from accessing your site to answer a specific query, as some user-initiated agents act as a proxy for a human and may not strictly adhere to crawler directives. Differentiating these signals allows you to allow traffic that serves a user right now while blocking traffic that merely harvests your data for later use.
Segmenting AI Traffic by Category
Teams must now segment automation into granular categories rather than a single “bot” bucket.
- Search crawlers (SEO): Important for visibility. Typically allowlisted.
- AI training bots: Harvest data for models. Often provide little to no attribution or click-through traffic. Whether to block them is a policy decision — save bandwidth now, or allow them for potential future relevance.
- AI search bots: Retrieval agents (like Perplexity’s PerplexityBot or OpenAI’s OAI-SearchBot) that scan content to answer user queries. These may drive some traffic but also answer queries directly on the results page, reducing click-throughs.
- Autonomous AI agents: Browser-driving agents (like ChatGPT Atlas, Claude in Chrome, or Gemini’s Agent Mode) that navigate and take actions on a user’s behalf in real time. This is the fastest-growing category of automated traffic and generally warrants the same “user-initiated” treatment as ChatGPT-User rather than being blocked outright.
- Commercial scrapers: Marketing tools that aggregate data. These are often technically “safe” (low security risk) but “bad” for business competitiveness.
With solutions like Bot Insights, organizations can analyze traffic volume and cache-hit patterns by segment, which helps estimate the cost impact of each one. If an AI crawler is generating, say, a 40% cache miss rate and pulling heavy assets from the origin, the operational cost might outweigh any theoretical visibility benefit.
Why Sampling Destroys Detection Accuracy
A critical failure mode in differentiating good and bad bots is reliance on sampled logs. Many observability platforms sample trace or log data — sometimes as aggressively as 1%, sometimes closer to 10% — to reduce costs, though defaults vary by vendor and some platforms now use adaptive sampling instead of a fixed rate. While this is acceptable for tracking general uptime, it renders security analysis far less effective.
Adversarial bots often operate “low and slow” to evade rate limits. A bot might make one request every few minutes from thousands of different IP addresses. In a sampled dataset, these requests appear as disconnected, random noise. You cannot correlate the pattern because the vast majority — 90% or more, depending on the sampling rate — of the evidence has been discarded.
Full-Fidelity Logging and Long-Term Retention
Differentiating sophisticated bots requires full-fidelity logging. You need to see every request, every header, and every error code to fingerprint a distributed attack. You also need to retain this data for months, not days. A “sleeper” botnet might probe a network for weeks before executing an attack. If your logs cycle out after a matter of days to a few weeks — a common default for many CDN and WAF products — you lose the forensic history needed to identify the initial probing phase.
Visibility into essential CDN metrics (including cache status, origin response time, and request/error patterns) provides the baseline data needed to distinguish a misconfigured partner bot from a malicious probe.
Building a Data-Driven Bot Strategy
The era of simplistic bot blocking is over. Security and engineering teams now face a complex ecosystem where trusted partners, aggressive AI agents, and malicious actors all blend into the same stream of HTTP requests. Differentiating good and bad bots is no longer just about checking a User-Agent string; it involves establishing a governance model based on verified identity, granular behavior analysis, and business value. For many organizations, the bottleneck is data, as security teams often struggle to differentiate bots without a unified, granular history of their traffic. With Hydrolix, teams can ingest and retain fully unsampled log data for 15 months or longer at a lower cost than many traditional observability tools. This retention capability lets you perform long-term behavioral analysis, accurately measure the cost of AI crawlers, and build a strategic bot management policy aligned with your business goals. By separating signal from noise at the log level, you can stop reacting to bot traffic and start managing it.
FAQs About Differentiating Good and Bad Bots
How do I verify a Googlebot request is actually from Google?
You should use a combination of reverse and forward DNS lookups. Run a reverse DNS lookup on the accessing IP address to see if it resolves to a googlebot.com domain, then run a forward DNS lookup on that domain to verify it matches the original IP. Google also publishes a list of IP ranges that can be used for allowlisting.
Can robots.txt successfully block bad bots?
No, the robots.txt file is a governance mechanism, not a security tool. It relies on the voluntary cooperation of the crawler to respect your directives. While legitimate bots (like Google or Bing) will honor it, malicious scrapers and vulnerability scanners will ignore it or use it as a map to the paths you are trying to hide.
Why is AI traffic considered different from standard search crawlers?
AI traffic is often categorized differently because the intent and value exchange differ from traditional SEO. While search crawlers index content to drive user traffic to your site, AI training bots often ingest content to train proprietary models without guaranteeing attribution or referral traffic, changing the ROI calculation for hosting that traffic.
What is TLS fingerprinting in bot detection?
TLS fingerprinting, primarily JA4 today, since it’s resilient to the extension-order randomization that modern browsers use and that made the older JA3 method less reliable, analyzes the specific parameters of the TLS handshake, including cipher suites and extensions, to identify the client application. This helps identify bots that claim to be a standard web browser (like Chrome) but technically behave like a script (like Python or curl) during the secure connection establishment.
Why shouldn’t I use sampled logs for bot management?
Sampled logs frequently miss low-volume or distributed bot attacks that are designed to stay below detection thresholds. By analyzing only a fraction of your traffic, you lose the ability to correlate subtle patterns across different IP addresses and miss the outlier events that often signal a sophisticated account takeover or scraping campaign.
