**BeautifulSoup** and **Requests** are a powerful Python combo for lightweight web scraping. Requests handles HTTP communication, while BeautifulSoup parses HTML to extract structured data—making them ideal for quick projects and simple crawlers without the overhead of full frameworks.
## **Prerequisites** - Python 3.7 or higher installed - Squid Proxies account with active proxy list - Your IP whitelisted in the Squid Proxies dashboard (for both residential and datacenter proxies) - Libraries installed: ```````dark pip install requests beautifulsoup4 ```````
## **Basic web scraping with BeautifulSoup and Requests (no proxy)** ```````dark import requests from bs4 import BeautifulSoup url = 'https://example.com' response = requests.get(url) soup = BeautifulSoup(response.text, 'html.parser') print(soup.title.text) ``````` This is how standard scraping works. Next, we'll add a proxy.
## **Adding a proxy to Python Requests** **Format reminder:** - Datacenter: `http://123.45.67.89:8000` - Residential: `http://p102.squidproxies.com:8907` **Example with a static proxy:** ```````dark proxies = { "http": "http://123.45.67.89:8000", "https": "http://123.45.67.89:8000" } response = requests.get('https://example.com', proxies=proxies) soup = BeautifulSoup(response.text, 'html.parser') print(soup.h1.text) ``````` 💡 Note: No username/password is needed if you're using IP whitelisting with Squid Proxies.
## **Beginner-friendly proxy rotation with a proxy pool** ```````dark import random proxy_list = [ "http://123.45.67.89:8000", "http://123.45.67.90:8000", "http://123.45.67.91:8000" ] proxy = random.choice(proxy_list) proxies = {"http": proxy, "https": proxy} response = requests.get('https://example.com', proxies=proxies) soup = BeautifulSoup(response.text, 'html.parser') ```````
## **Advanced scraping techniques** **1. Custom headers** ```````dark headers = { "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)...", "Accept-Language": "en-US,en;q=0.9", "Referer": "https://google.com" } requests.get('https://example.com', proxies=proxies, headers=headers) ``````` **2. Timeout & retry handling** ```````dark try: response = requests.get('https://example.com', proxies=proxies, timeout=10) if response.status_code == 200: soup = BeautifulSoup(response.text, 'html.parser') except requests.exceptions.RequestException as e: print(f"Request failed: {e}") ``````` **3. Handling blocks (403, 429, CAPTCHA)** - Rotate headers and proxies on 403/429 responses - Slow down scraping with `time.sleep()` or randomized delays - Use CAPTCHA-solving if you encounter reCAPTCHA or hCaptcha (see Advanced Proxy Guide)
## **Optional: Sticky IP for consistent sessions** ```````dark sticky_proxy = "http://p102.squidproxies.com:8907?session=abc123" proxies = {"http": sticky_proxy, "https": sticky_proxy} ``````` This keeps the same IP across multiple requests for login/session-based scraping.
## **Testing your proxy setup** ```````dark requests.get("https://ipinfo.io", proxies=proxies) ``````` **Or test directly in terminal:** ```````dark curl -x http://123.45.67.89:8000 https://httpbin.org/ip ```````
## **Best practices summary** | Practice | Why it matters | |----------|----------------| | Rotate proxies | Avoid IP bans and blocks | | Rotate headers | Appear like different browsers | | Add delays | Mimic human behavior | | Handle retries | Prevent script crashes on timeouts | | Use sticky IP (if needed) | Maintain sessions for login flows | | Test proxies first | Ensure they’re working before scraping |
With this setup, you can confidently scrape with BeautifulSoup + Requests using Squid Proxies at scale while staying under the radar.
---
**More Guides:** - [CAPTCHA Handling with 2Captcha](https://www.squidproxies.com/developer-docs/dev-advanced-guides/captcha-handling-with-2captcha) - [Avoiding Blocks at Scale](https://www.squidproxies.com/developer-docs/dev-advanced-guides/avoiding-blocks-at-scale)