Bot detection is the process of identifying whether traffic to a website, app, or API comes from automated software or a human. It also separates harmful bots from legitimate ones, such as search engine crawlers. Detection systems score signals from the network, the client, and user behavior. The result decides whether a request is allowed, challenged or blocked.

The Layers of Bot Detection

  • Network-level signals: IP reputation, ASN filtering
  • Client-level signals: TLS and browser fingerprints, header consistency
  • Behavioral signals: behavioral telemetry, session-aware rate limits

A request that passes one layer can still be flagged by another.

Some signals can be checked against each other. OWASP notes that Client Hints headers are declared by the client but can be verified against its TLS fingerprint.

Verifying Good Bots

Not all automated traffic is unwanted. Search engine crawlers, for example, fetch pages so they can appear in search results. Because any bot can claim to be a crawler in its User-Agent, sites verify the IP address instead. Google documents two ways to verify Googlebot:

  • DNS check: Run a reverse DNS lookup on the IP. Confirm the hostname ends in googlebot.com, google.com, or googleusercontent.com. Then run a forward lookup and confirm it returns the original IP.
  • IP ranges: Match the IP against the lists of crawler IP ranges that Google publishes in JSON.

Threats Bot Detection Targets

The OWASP Automated Threats to Web Applications project names 21 automated threats. Examples include:

  • Credential stuffing (OAT-008): Mass login attempts that test stolen username and password pairs.
  • Scraping (OAT-011): Collecting application content or other data for use elsewhere.
  • Scalping (OAT-005): Getting limited-availability or preferred goods and services by unfair methods.
  • Account creation (OAT-019): Creating many accounts for later misuse.