Large-Scale Web Scraping Architecture: A Field Guide

A few hundred pages is a script. Millions is a distributed system. Here's the architecture that gets you there — queues, async workers, proxy tiering, retries, dedup and data-quality checks, with working code.

Scraping a few hundred pages is a script. Scraping millions is a distributed system. Once your target count crosses from "runs overnight on my laptop" to "needs to finish this week without melting," the hard part stops being the parsing and becomes everything around it — how you queue work, fan it out, avoid blocks, retry failures, and store and check the result. This is large-scale web scraping as an architecture, not a snippet, with working code at each stage.

Scale is concurrency, not a bigger loop

One figure makes it concrete. Take a category with 20,000 listing pages, 20 items each — 400,000 pages to fetch. At a realistic 2.5 seconds per page, a strictly sequential run is about 1,000,000 seconds, roughly 11.5 days of waiting on page loads before you parse a single field. Drive 200 pages in parallel and those 11.5 days collapse toward an hour of wall-clock time. Time is the constraint at scale, and concurrency is how you buy it back. Everything else in the architecture exists to make that concurrency survivable.

Stats diagram showing 400,000 pages taking 11.5 days sequentially versus about one hour at 200 in parallel, with 10,000 failures per million at 1 percent — Concurrency turns a 11.5-day run into an hour — but at a million requests, even a 1% failure rate is 10,000 dead pages.

The architecture at a glance

A scraper that survives millions of pages is a small distributed system with a handful of named parts, each solving a problem that only shows up at volume:

A queue holds the URLs still to fetch and decouples discovery from work.
Async or distributed workers pull from the queue and fetch concurrently — this is where the wall-clock savings live.
A proxy and anti-bot layer rotates IPs and presents real-browser traffic so no single address trips a rate limit.
Rendering, only when needed, because a headless browser is the most expensive thing in the pipeline.
Retries with backoff, deduplication, storage, and monitoring plus data-quality checks round it out.

Queue first: decouple discovery from fetching

The single most important structural decision is putting a queue between "what to scrape" and "doing the scrape." A producer enumerates URLs; a pool of workers drains them. Neither side knows how fast the other runs, and you add workers without touching the producer. In Python this is Celery or RQ over Redis; in Node, BullMQ; at larger scale, RabbitMQ or Kafka. The pattern in one file:

import asyncio, aiohttp

CONCURRENCY = 50
queue = asyncio.Queue()

async def worker(session):
    while True:
        url = await queue.get()
        try:
            async with session.get(url, timeout=20) as resp:
                await handle(url, await resp.text(), resp.status)
        except Exception as err:
            await on_failure(url, err)
        finally:
            queue.task_done()

async def run(urls):
    for u in urls:
        queue.put_nowait(u)
    async with aiohttp.ClientSession() as session:
        tasks = [asyncio.create_task(worker(session)) for _ in range(CONCURRENCY)]
        await queue.join()
        for t in tasks:
            t.cancel()

The knob that matters is CONCURRENCY. Too low wastes the parallelism that makes scale possible; too high overwhelms both the target and your own egress. You find the right value by watching error rates climb — which is exactly why monitoring is a first-class part of the system, not an afterthought.

Flow diagram of a large-scale scraping pipeline: queue, async workers, proxy tier, parse and quality checks, storage — Eight named parts. The proxy, anti-bot and rendering layers are the hardest to keep healthy — and the natural place to buy.

The proxy tier breaks first

At low volume you barely notice anti-bot defenses; at scale they break the run first. Send a few hundred thousand requests from one IP and you get rate-limited, then challenged, then blocked. The fix is rotation across many addresses. Tier it: cheap datacenter IPs for lenient targets and APIs, rotating residential IPs for hard commercial targets that expect real-user traffic. But rotation alone isn't enough — modern defenses also read TLS fingerprints and header order, so the traffic has to look like a browser, not just come from a fresh IP.

Retries: failure is the steady state

At a million requests, a 1% transient failure rate is 10,000 failed pages. Failure isn't an edge case at this volume — it's routine, and the pipeline has to treat a failed fetch as normal rather than fatal. Retry with exponential backoff and a cap, then move the URL to a dead-letter queue instead of blocking the run. Read why it failed: a timeout or 503 is worth retrying, a hard 404 is not.

import asyncio, random

async def fetch_with_retry(session, url, tries=4):
    for attempt in range(tries):
        try:
            async with session.get(url, timeout=20) as r:
                if r.status == 404:
                    return None                # don't retry a hard 404
                if r.status < 400:
                    return await r.text()
        except Exception:
            pass
        await asyncio.sleep(2 ** attempt + random.random())  # backoff + jitter
    await dead_letter(url)                     # give up after the cap
    return None

Deduplication: don't crawl the same page twice

Discovery at scale produces duplicates constantly — the same product reachable from three paths, tracking parameters that make one page look like ten. Normalize URLs before they enter the queue, then keep a seen-set (a Redis set, or a Bloom filter once the set reaches hundreds of millions):

from urllib.parse import urlsplit, urlunsplit, parse_qsl, urlencode

seen = set()

def normalize(url):
    s = urlsplit(url.lower())
    q = [(k, v) for k, v in parse_qsl(s.query) if not k.startswith("utm_")]
    return urlunsplit((s.scheme, s.netloc, s.path.rstrip("/"), urlencode(sorted(q)), ""))

def enqueue(url):
    key = normalize(url)
    if key not in seen:
        seen.add(key)
        queue.put_nowait(key)

Storage and data quality

Two scale-specific habits: write in batches so storage isn't your bottleneck, and separate raw from parsed so you can re-parse without re-crawling when selectors change. Then add the half of monitoring most teams skip — data-quality checks. A run can report 100% HTTP success and still produce garbage if the layout drifted and your selectors now match nothing. Assert required fields are non-empty and values are sane:

def validate(row):
    assert row.get("title"), "empty title — selector may have drifted"
    price = row.get("price")
    assert isinstance(price, (int, float)) and 0 < price < 1_000_000, "bad price"
    return row

# fail loud on page 5,000, not silently after 5,000,000 empty rows

Offload proxies, anti-bot and rendering to one API

Render only when you must

A headless browser is the most expensive operation in the pipeline — CPU, memory, and seconds per page, which dominate everything at a million pages. Many sites still ship data in the initial HTML or a JSON endpoint; a plain fetch plus a parser is an order of magnitude cheaper. Try the cheap path first, confirm the fields are present, and escalate to rendering only for the pages that need it. Our breakdown of render cost versus HTTP puts real numbers on that gap.

Build vs buy the hard layers

Everything above is buildable, so the honest question is which parts earn your engineering time. The data model, parsing logic, quality checks, and storage schema are specific to your project — only you can build them well. The proxy pool, anti-bot handling, headless render fleet, and retry-and-delivery queue are generic infrastructure that's expensive to build and a grind to keep healthy as targets evolve. That's the line a managed Scraper API sits on: rent the parts that are the same for everyone. Our build-versus-buy breakdown works the maintenance tax in detail, and running scrapers as production software covers keeping the whole thing observable.

Frequently asked questions

How do you scrape millions of pages?

With concurrency, not a bigger loop. Put a queue between URL discovery and fetching, drain it with a pool of async or distributed workers, rotate IPs to avoid blocks, retry transient failures with backoff, deduplicate URLs, and batch writes to storage. A sequential million-page run takes days; the same job across a concurrent worker pool finishes in hours.

What is the best architecture for large-scale web scraping?

A queue-and-worker pipeline: a producer enumerates URLs onto a queue (Redis, RabbitMQ, or Kafka), workers fetch concurrently through a rotating proxy layer, render only the pages that need JavaScript, retry failures into a dead-letter queue, dedupe with a seen-set, and store raw and parsed data separately. Wrap it in monitoring with data-quality checks so drift surfaces early.

How many requests can you run in parallel?

It depends on the target and your egress, not a fixed number. Scraping is IO-bound, so a modest box can hold many hundreds of in-flight requests. Start with a concurrency of around 50, watch the error rate, and raise it until failures climb — that's your ceiling. Beyond one machine's limits, add distributed workers rather than pushing a single node harder.

How do you handle failures at scale?

Assume failure — at a million requests, even a 1% error rate is 10,000 dead pages. Retry transient errors (timeouts, 503s) with exponential backoff and jitter, cap the attempts, and move persistent failures to a dead-letter queue instead of blocking the run. Don't retry hard 404s. Randomizing collection order also spreads failures so you don't fail on the same pages every run.

Scale is mostly the parts that aren't fun to build: rotating IPs, anti-bot, headless rendering, queues, and retries. Own the pieces specific to your data — the model, the parsers, the quality checks — and rent the generic infrastructure that's the same grind for everyone. Get the queue and concurrency right first; everything else is making that concurrency survive contact with a million real pages.

Power the fetch layer with rotating residential IPs