Concurrent connections, often called threads, are the number of requests you can have open through a proxy at the same moment. A limit of 50 means up to 50 requests can run in parallel. When one finishes, another can start. More concurrency finishes large jobs faster, but every open connection also uses resources on the proxy and the target server.

Thread limits are usually set per proxy or per account, depending on the provider, and running more threads than your plan allows typically results in rejected requests (often HTTP 429 errors) rather than an outright ban.

Threads, Connections, and Requests

  • Thread: A unit of work in your own program. In a simple scraper, each thread sends one request and waits for its response.
  • HTTP/1.x: One connection handles requests largely one at a time, so HTTP/1.0 and HTTP/1.1 clients open several connections to make concurrent requests.
  • HTTP/2: Many requests, called streams, can be interleaved on a single connection. A server sets its limit with SETTINGS_MAX_CONCURRENT_STREAMS.
  • Why it matters: Your thread count, open connections, and in-flight requests are not always the same number, so check which one a limit refers to.

Choosing a Safe Concurrency Level

  • Be conservative: The HTTP/1.1 specification (RFC 9112) says a client ought to limit the number of simultaneous open connections to a given server. It sets no fixed maximum.
  • Why: Each connection consumes server resources, and many parallel connections can cause congestion that fewer connections would avoid.
  • Respect Retry-After: A 429 Too Many Requests response may include a Retry-After header that says how long to wait before trying again.
  • Ramp up gradually: Start with a low thread count and raise it while watching error rates and response times.