A web crawler is a program that systematically browses and indexes web pages by following links from page to page, typically used to discover and catalog content across a site or the broader web. Search engines use crawlers to build their indexes, and AI companies increasingly use crawlers to gather training and retrieval data.

Crawlers differ from targeted scrapers in scope: a scraper usually extracts specific data from known pages, while a crawler's job is discovery, finding pages it doesn't already know about by following links outward. Both can use proxies for the same reasons: avoiding rate limits and IP-based blocking when making a high volume of requests to the same or related domains.

A crawler typically keeps a queue of URLs it has discovered but not yet visited, along with a record of what it's already crawled, to avoid revisiting the same pages endlessly on a large site.

A well-behaved crawler checks a site's robots.txt file before crawling and skips any paths it disallows, though nothing forces it to, which is why sites also rely on rate limits and detection rather than robots.txt alone.