Scrapy Proxy Middleware: Rotating Proxies Without the Bans

Scrapy has shipped proxy support since version 0.8, but the docs never tell you how to rotate, retry and survive bans at scale. Here is the middleware anatomy, the code, and the settings that decide whether your crawl finishes.

Scrapy has shipped a proxy middleware since version 0.8, so 'does Scrapy support proxies' was settled fifteen years ago. What the docs never quite assemble is the production picture: how the built-in Scrapy proxy middleware actually processes credentials, when to set proxies per request versus globally, and how to rotate and recover when a target starts banning exits mid-crawl. This guide walks the whole chain — the built-in HttpProxyMiddleware, per-request meta, a custom rotation middleware with ban handling, and the settings that decide whether a 100k-page crawl finishes or dies at 3am.

How the built-in Scrapy proxy middleware works

scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware is enabled by default at priority 750 in DOWNLOADER_MIDDLEWARES_BASE. Reading its source (about 100 lines in Scrapy 2.17) tells you everything you need for debugging:

• On startup it calls urllib.request.getproxies(), so http_proxy / https_proxy environment variables configure a spider with zero code. no_proxy is honoured for http and https schemes only.
• Per request, request.meta['proxy'] always wins over the environment. Set it to None to force a direct connection for that request.
• Credentials embedded in the proxy URL (http://user:pass@host:port) are stripped, Base64-encoded and sent as a Proxy-Authorization: Basic header. Encoding defaults to latin-1 and is configurable via HTTPPROXY_AUTH_ENCODING.
• The middleware unquotes credentials before encoding them, so URL-encode special characters like @ in passwords — they pass through correctly.
• If a retry changes the proxy, the middleware drops the stale auth header instead of leaking it to the new proxy — a credential-hygiene fix in modern Scrapy, and a good reason not to pin ancient versions.

Practical consequence: you almost never need a third-party proxy package. Anything that puts a valid proxy URL into request.meta['proxy'] before priority 750 gets authentication and header handling for free.

Per-request proxies with meta

The lightest integration is setting meta directly in the spider — useful when only some requests need a proxy, or different targets need different exit countries:

import scrapy


class PricesSpider(scrapy.Spider):
    name = "prices"

    def start_requests(self):
        yield scrapy.Request(
            "https://example.com/product/1",
            meta={"proxy": "http://USER:PASS@gate.quantumproxies.io:PORT"},
        )

Per-request meta shines when the proxy choice itself is data-driven: route German product pages through a Germany-targeted username, send image downloads through cheap datacenter exits while HTML goes residential, or escalate a stubborn URL to a stickier session on its second attempt. Because meta survives retries and redirects, the decision you make when building the request follows it through the whole download cycle. Setting meta on thousands of requests by hand does not scale, though — that is what downloader middlewares are for.

The simplest production setup: a rotating gateway

With rotating proxies behind a single gateway endpoint, rotation happens server-side: every request through the same URL exits from a different IP in a 90M+ residential pool spanning 200+ countries. Your middleware shrinks to three lines, and there is no list to health-check, prune or refresh:

# middlewares.py
class RotatingGatewayMiddleware:
    PROXY = "http://USER:PASS@gate.quantumproxies.io:PORT"

    def process_request(self, request, spider):
        request.meta["proxy"] = self.PROXY


# settings.py
DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.RotatingGatewayMiddleware": 350,
}

Priority 350 matters: your middleware must run before the built-in at 750 so HttpProxyMiddleware can still convert the credentials into the auth header. This pairing — gateway rotation plus Scrapy's stock retry machinery — is the highest reliability per line of code, because every retry automatically travels through a fresh exit IP.

A custom rotating proxy middleware with ban handling

If you do run your own proxy list — static ISP addresses, or a mixed pool — the classic pattern (popularised by the scrapy-proxies package, which recommended RETRY_TIMES = 10 because free-list proxies fail so often) is: choose randomly per request, evict on ban signals, re-queue the request:

import random


class ProxyListMiddleware:
    BAN_CODES = {403, 429}

    def __init__(self, proxies):
        self.proxies = list(proxies)

    @classmethod
    def from_crawler(cls, crawler):
        return cls(crawler.settings.getlist("PROXY_LIST"))

    def process_request(self, request, spider):
        if self.proxies and "proxy" not in request.meta:
            request.meta["proxy"] = random.choice(self.proxies)

    def process_response(self, request, response, spider):
        if response.status in self.BAN_CODES:
            bad = request.meta.get("proxy")
            if bad in self.proxies and len(self.proxies) > 1:
                self.proxies.remove(bad)
                spider.logger.warning("Evicted %s (%d left)", bad, len(self.proxies))
            return request.replace(dont_filter=True)  # re-queue on a new proxy
        return response

Note everything this middleware must do that a gateway does for free: track pool health, evict burned IPs, re-queue requests. It also shrinks under fire — a pool of 20 static IPs can evaporate in minutes on an aggressive target. The wider anti-ban playbook (pacing, headers, session discipline) is in our IP ban avoidance checklist.

Scrapy settings that decide the outcome

Middleware places the proxy; settings decide whether the crawl behaves. These matter most:

# settings.py — a sane baseline for proxied crawls
RETRY_TIMES = 5
RETRY_HTTP_CODES = [429, 403, 500, 502, 503, 504, 408]

CONCURRENT_REQUESTS = 32
CONCURRENT_REQUESTS_PER_DOMAIN = 8
DOWNLOAD_TIMEOUT = 30

AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_TARGET_CONCURRENCY = 4.0

• RETRY_HTTP_CODES — add 429 and 403 so soft bans trigger a retry (and, with rotation, a new IP) instead of a failed item.
• RETRY_TIMES — 3-5 is right for a quality pool; needing 10 is a sign the pool itself is unhealthy.
• CONCURRENT_REQUESTS_PER_DOMAIN — the real politeness knob. Rotation spreads load across IPs, but the target still sees total volume.
• DOWNLOAD_TIMEOUT — the default 180 seconds lets one slow exit occupy a concurrency slot for three minutes. 30 is plenty.
• AUTOTHROTTLE — adapts pacing to observed latency; pairs well with rotation on rate-limited targets. Still drowning in 429s? Our rate-limit guide covers concurrency budgets in depth.

Ban detection beyond status codes

Sophisticated targets do not send honest 403s. They serve 200s containing a CAPTCHA page, an empty product grid, or a stripped template. Robust spiders verify content, not just status — check for an element that only exists on the real page, and re-queue when it is missing. If pages come back structurally empty even through good proxies, the content is probably rendered client-side; see why scrapers get empty pages. And when one domain defeats plain HTTP no matter the IP, hand that domain to a Scraper API that renders JavaScript and returns clean HTML or markdown — Scrapy consumes it like any other response, and you keep your pipeline.

def parse(self, response):
    if not response.css("div.product-grid"):
        # 200 OK but the real content is missing: soft ban or JS wall
        yield response.request.replace(dont_filter=True)
        return
    for product in response.css("div.product-grid article"):
        yield {"name": product.css("h2::text").get()}

Frequently asked questions

Does Scrapy support proxies out of the box?

Yes. HttpProxyMiddleware ships enabled at priority 750: it reads http_proxy / https_proxy environment variables and honours request.meta['proxy'] per request, including user:pass@ credentials, which it converts to a Proxy-Authorization header. You only write code to decide which proxy each request gets.

How do I set a proxy for a single request in Scrapy?

Pass it in the request's meta: scrapy.Request(url, meta={'proxy': 'http://user:pass@host:port'}). Meta always overrides environment-level proxies, and setting the value to None forces that one request to go direct — useful for mixing proxied and unproxied traffic in one spider.

How do I rotate proxies in Scrapy?

Either write a downloader middleware that picks a proxy per request from a list and evicts banned ones, or point every request at a rotating gateway endpoint that assigns a fresh exit IP server-side. The gateway approach needs a three-line middleware, no health checks, and turns every Scrapy retry into a rotation for free.

Why does my Scrapy proxy return 407?

The proxy rejected authentication. Check that credentials sit inside the proxy URL in meta, that special characters are URL-encoded, and that your server IP is whitelisted if your plan uses IP auth. If credentials contain non-ASCII characters, set HTTPPROXY_AUTH_ENCODING to match your provider.

The pattern to remember: set meta['proxy'] early, let the built-in middleware at 750 do auth, add 429 and 403 to your retry codes, and prefer server-side rotation over list babysitting. If parts of your crawl need JavaScript, weigh the cost of headless browsers versus HTTP requests before reaching for one — most Scrapy projects need better IPs, not a browser.

Plug rotating proxies into your Scrapy project