**Rate limiting and retry logic** Respecting rate limits and building smart retry logic into your tools can prevent bans and maximize data collection. Handling errors like 429 responses gracefully helps maintain scraping efficiency without overwhelming target servers.
## **What is rate limiting?** **Rate limiting** is a server-side mechanism to restrict the number of requests accepted per: - IP address - Session or cookie - API token or user-agent - Time window (e.g., X req/minute or second) Common signals of rate limiting: - HTTP 429 Too Many Requests - Unexpected timeouts or connection resets - CAPTCHAs or JS challenges appearing mid-task - Empty or shortened response bodies
## **General principles** - Never hardcode fixed delays — use adaptive or randomized wait times. - Always monitor response status codes — don't blindly retry. - Isolate proxies and sessions per domain — avoid cross-contamination. - Throttle proactively — don’t wait for bans to start slowing down.
## **Designing a retry system** A robust retry system includes: - **Detection** – interpret signals like 429, 403, or content anomalies. - **Delay** – exponential backoff, jitter, or retry buckets. - **Recovery** – re-queue failed tasks, optionally with session reset. - **Fallback** – switch proxy, user-agent, or scrape strategy.
## **Exponential backoff with jitter** **Basic backoff** ```python for i in range(max_retries): delay = 2 ** i time.sleep(delay) ``` **Backoff with jitter** ```python import time, random for attempt in range(max_retries): base_delay = 2 ** attempt jitter = random.uniform(0, 1.5) time.sleep(base_delay + jitter) ```
## **Status-aware retry handling** | Status Code | Meaning | Retry Action | |-------------|----------------------------------|--------------------------| | 429 | Too Many Requests | Backoff + rotate proxy | | 403 | Forbidden (possibly banned) | Rotate proxy + UA | | 503 | Service Unavailable | Retry with backoff | | Timeout | Network congestion/slowdown | Retry immediately | Always log these events and tie them to: - Proxy ID or pool - Target domain - Session/cookie/user-agent fingerprint
## **Proxy-aware rate limiting** With Squid Proxies, manage rate limits per proxy, not globally: | Technique | Description | |------------------------|-----------------------------------------------------| | Token buckets | Allow X requests per proxy per minute | | Time-based throttling | Enforce minimum delay between requests per proxy | | Load-balancing by proxy| Assign tasks based on recent load or error rate | **Python threaded example** ```python from threading import Semaphore import time limit_per_minute = 10 semaphore = Semaphore(limit_per_minute) for proxy in proxy_list: semaphore.acquire() do_request(proxy) time.sleep(60 / limit_per_minute) semaphore.release() ```
## **Session persistence vs. rotation** Sessions reduce friction, but overusing one session increases detection risk. **Strategies**: - Rotate sessions every N requests or T minutes - Keep a map of session → proxy → UA → cookies - When blocked, discard and regenerate session with new proxy/headers
## **Handling soft rate limits (behavioral)** Some websites won’t return a 429 but will: - Inject delays (slow HTTP responses) - Drop cookies silently - Trigger JavaScript-based puzzles **Detection strategies**: - Log response time spikes - Compare content length or structure hashes - Trigger re-validation if expected keys are missing
## **Retry budgeting and circuit breakers** Avoid infinite retry loops. Use a retry budget per task, and implement a circuit breaker that: - Marks a domain or IP as temporarily down - Logs failure reason for future review - Escalates when retry volume exceeds threshold
## **Rate limiting and retry best practices** | Challenge | Solution | |---------------------------|---------------------------------------| | 429 or 403 status codes | Rotate proxy + backoff | | Delayed or partial response| Log + retry with delay | | High fail rate | Circuit break the domain or proxy pool| | Intermittent bans | Increase proxy pool size | | API rate-limiting | Follow Retry-After headers |
## **Tools to consider** - **Tenacity (Python)** – configurable retry framework - **axios-retry (Node.js)** – auto-retry with exponential backoff - **Custom Redis-based rate limiters** for distributed workers
## **Final recommendations for rate limiting and retries** - Implement retry logic by status code and task type - Distribute load across rotating proxies with enforced delay windows - Maintain a global error log with retry metadata - Use adaptive throttling to simulate real-user traffic behavior - Monitor your retry rate and response latency over time
---
**More Guides:** - [IP Hygiene & DNS Leak Protection](https://www.squidproxies.com/developer-docs/dev-advanced-guides/ip-hygiene-amp-dns-leak-protection) - [Proxy Pool Management and Failover](https://www.squidproxies.com/developer-docs/dev-advanced-guides/proxy-pool-management-and-failover)