When crawling large, sensitive, or geo-targeted websites, your IP address may be blocked, rate-limited, or otherwise restricted. That’s where Squid Proxies come in. By configuring datacenter or residential proxies, you can: - Bypass IP-based bans and firewall filters - Simulate user behavior from specific regions or countries - Scale audits across multiple domains without losing access or speed ## **Requirements: Proxy details** Before you start, collect from your Squid Proxies dashboard: - Proxy IP (e.g., 123.456.78.90) - Port (commonly 8800) - Username & Password (_if using user/pass authentication_) **OR** Ensure your device IP is whitelisted (_for IP-auth plans_)
## **Netpeak Spider setup guide** ### **Step 1: Install and Open Netpeak Spider** 1. Download Netpeak Spider at https://netpeaksoftware.com/spider. 2. Install the application on your Windows system 3. Log in using your Netpeak Software credentials 4. Open the dashboard where you can begin configuring crawl tasks

### **Step 2: Configure your Squid Proxy in Netpeak Spider**

1. Click Settings > Network Settings 2. In the Proxy Server section: - Check the box to enable proxy use - Select HTTP or HTTPS 3. Enter your proxy details: - IP Address: Your Squid Proxy IP - Port: (e.g., `8800`) - Login / Password: Only if you're using user/pass authentication 4. You can also enable proxy use for DNS resolution (optional) 5. Click Test Proxy to ensure everything is working correctly

✅ A successful test means Netpeak Spider is now routing all crawl traffic through your Squid Proxy.
### **Step 3: Run a Proxy-Powered Crawl** 1. Enter the URL of the site you want to audit in the “Enter URL” bar. 2. Hit Start to begin crawling. 3. All outgoing requests will now route through your Squid Proxy—allowing for: - Block-free access to protected sites - Access to country-specific content (when using geo-targeted residential proxies) - Higher crawl concurrency with fewer IP restrictions

## **Using rotating residential proxies** If you're using Squid Proxies' residential service: - Use your rotating residential endpoint (e.g., `http://res.squidproxies.com:PORT`) - This setup will change your IP automatically behind the scenes - It’s ideal for large-scale or repeated crawls where websites might block static IPs
## **Netpeak Spider troubleshooting** - **Crawl not starting?** - Recheck proxy IP, port, and authentication method - Ensure you’re not blocked by firewall or antivirus software - **403 Forbidden or blank responses?** - The target site may be blocking datacenter IPs—try switching to residential proxies - **Wrong IP still showing?** - Make sure proxy is enabled in Network Settings and you clicked Apply or Save - **Connection timeouts?** - Lower the crawl speed in **Settings > Speed** - Choose proxies closest to your target server location

## **Best practices for proxies in Netpeak Spider** | **Use Case**| **Recommended Proxy Type** | |----------|----------| | Large-scale internal site audits | Datacenter proxies | | Geo-targeted SEO audits | Residential proxies with location targeting | | Crawling client competitor websites | Residential proxies for stealth | | Auditing duplicate content or site structure | Either, based on crawl depth and site protection | ✔️ Use one proxy per project for consistency ✔️ Match proxy geo-location to your audit market ✔️ Use rotating proxies for frequent scans ✔️ Always test on small crawls before scaling
---
**More Guides:** - [Testing Your Proxies](https://www.squidproxies.com/proxy-tutorials/tutorials-getting-started/testing-your-proxies) - [Common Proxy Errors](https://www.squidproxies.com/proxy-tutorials/troubleshooting/common-proxy-errors)