Rate limiting is a control that caps how many requests a client, identified by IP address or user account, can make within a set time window, then blocks or delays requests over the limit. Servers signal it with HTTP 429 Too Many Requests, which may include a Retry-After header saying how long to wait before trying again.
Levels Rate Limits Get Enforced At
- Per IP
- Per subnet
- Per API key
- Per account
When a client exceeds them, the server can temporarily stop fulfilling its requests and respond with HTTP 429 "Too Many Requests". Distributing requests across a large, rotating proxy pool spreads them across many per-IP counters, helping keep each IP under per-IP rate limits at scale.
Common Rate-Limiting Algorithms
- Fixed window: Counts requests and resets the counter at the start of each period, so traffic spikes can slip through around each reset.
- Sliding window: Weights the previous period's count by how much of it overlaps the current window, which smooths out those spikes.
- Token bucket: Tokens refill at a fixed rate up to a maximum. Each request spends one, so bursts are allowed up to the bucket's size.
- Leaky bucket: Requests drain at a constant rate. When the bucket is full, new requests are rejected.
Why Sites Use Rate Limiting
Scraping is only one target. Rate limits also help:
- Slow brute-force login attempts, which Cloudflare recommends tracking by both IP address and username.
- Blunt DoS and DDoS traffic floods.
- Keep API usage within per-user quotas, such as a set number of calls per hour or day, even when the overuse isn't malicious.