**Scrapy** is a robust Python framework for building scalable web crawlers and data extraction tools. When targeting sites with anti-bot defenses or rate limits, relying on a single IP can lead to quick blocks. Integrating datacenter or residential proxies into your Scrapy spiders helps maintain anonymity, bypass geo-restrictions, and ensure consistent access—especially during large-scale scraping operations.
## **Prerequisites** Before you begin, make sure you have the following: - Python 3.7+ installed - Scrapy installed: ```bash pip install scrapy ``` - A working proxy list from your Squid Proxies dashboard - Your IP whitelisted in your Squid Proxies dashboard (for datacenter or residential proxies)
## **Understanding proxy types and formats** | Proxy Type | Format Example | Auth Type | |--------------|--------------------------------------|------------------------------| | Datacenter | `http://123.45.67.89:8000` | IP Whitelist & User/Pass Authorization | | Residential | `http://p102.squidproxies.com:8907` | IP Whitelist & User/Pass Authorization |
## **Basic static proxy configuration** For simple cases, define a single proxy in your `settings.py` file: ```python HTTP_PROXY = 'http://123.45.67.89:8000' DOWNLOADER_MIDDLEWARES = { 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 110, 'scrapy.downloadermiddlewares.retry.RetryMiddleware': 90, } ``` Or override per request inside your spider: ```python def start_requests(self): yield scrapy.Request( url='https://httpbin.org/ip', meta={'proxy': 'http://123.45.67.89:8000'}, callback=self.parse ) ```
## **Rotating proxies with middleware** To avoid detection and rate-limiting, rotate proxies using middleware. ### **Step 1: Add your proxy list** Create a file called `proxies.txt`: ``` http://123.45.67.89:8000 http://123.45.67.90:8000 http://123.45.67.91:8000 ``` ### **Step 2: Create middleware (`middlewares.py`)** ```python import random class ProxyMiddleware: def __init__(self): with open('proxies.txt') as f: self.proxies = [line.strip() for line in f] def process_request(self, request, spider): proxy = random.choice(self.proxies) request.meta['proxy'] = proxy ``` ### **Step 3: Register middleware in `settings.py`** ```python DOWNLOADER_MIDDLEWARES = { 'your_project_name.middlewares.ProxyMiddleware': 350, 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 400, } ``` > 💡 You can also rotate proxies per domain or spider.
## **Proxy authentication (if required)** Squid Proxies uses IP whitelisting. If you need basic auth (for other providers): ```python from scrapy.utils.httpobj import basic_auth_header request.meta['proxy'] = "http://proxy-ip:port" request.headers['Proxy-Authorization'] = basic_auth_header("USERNAME", "PASSWORD") ``` > Skip this step when using Squid Proxies (no login needed if IP is whitelisted).
## **Optional: Rotating user-agents** To reduce detection, rotate user agents. ### **Add to `middlewares.py`:** ```python class RotateUserAgentMiddleware: def __init__(self): self.user_agents = [ 'Mozilla/5.0 (Windows NT 10.0; Win64; x64)...', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)...', # Add more user-agents here ] def process_request(self, request, spider): request.headers['User-Agent'] = random.choice(self.user_agents) ``` ### **Update `settings.py`:** ```python DOWNLOADER_MIDDLEWARES.update({ 'your_project_name.middlewares.RotateUserAgentMiddleware': 543, }) ```
## **Testing your proxy configuration** **Use this URL inside your spider:** ```python url = 'https://httpbin.org/ip' ``` **Or test via terminal:** ```bash curl -x http://123.45.67.89:8000 https://httpbin.org/ip ``` **You can also check your IP on:** - https://ipinfo.io - https://httpbin.org/ip
## **Debugging proxy issues** | Problem | Solution | |----------------|-----------------------------------------------------| | 403/429 Errors | Rotate IPs and user-agents more aggressively | | 407 Errors | Check that your IP is correctly whitelisted | | Timeouts | Increase `DOWNLOAD_TIMEOUT` in `settings.py` | | CAPTCHAs | Use services like 2Captcha or rotate IP more often |
## **Summary and best practices** | Task | Recommendation | |----------------------|-----------------------------------------------| | Basic Crawling | Use 1–2 datacenter proxies via `settings.py` | | Large-Scale Scraping | Rotate IPs with middleware + proxy pool | | Geo-specific Targets | Use Squid Proxies’ residential proxies | | Avoid Detection | Rotate user-agents and respect delays | | Test Proxies | Use `httpbin.org/ip` or `curl -x` |
---
**More Guides:** - [Python Proxy Setup](https://www.squidproxies.com/developer-docs/dev-getting-setup/python) - [Avoiding Blocks at Scale](https://www.squidproxies.com/developer-docs/dev-advanced-guides/avoiding-blocks-at-scale)