Polite Scraping That Still Scales: A Rate-Limiting Playbook
Hammering a site gets you banned; crawling one request at a time gets you nowhere. The scalable middle is adaptive pacing per domain, spread across clean IPs — polite to the site, fast for you.
Nearly half of all internet traffic is now automated, and websites know it. Rate limiting is their first and most basic defence — a speed limit that decides how much you can take and how fast. The temptation is to treat it as an obstacle to smash through with brute force, but that gets you banned. The opposite mistake, crawling one careful request at a time, gets you nowhere at scale. The scalable answer is polite scraping: pace each target the way a heavy-but-legitimate user would, spread the load across clean IPs, and let the site's own responses tune your speed. This is how you stay fast without becoming the traffic that gets everyone blocked.
Know which limit you are hitting
Servers count your requests against an identifier — usually your IP, sometimes an API key or account — over a time window, and act when you cross a threshold. The three common models behave differently: a fixed window counts requests per calendar minute; a token bucket hands you a fixed number of tokens (say 100 a minute) and spends one per request, forcing a wait when the bucket empties; a sliding window counts over the last rolling 60 seconds at any instant. The practical upshot is that a burst is more dangerous than a steady stream — ten requests in one second can trip a limit that a hundred spread over a minute would not.
Limits also come in flavours. Soft limits are gentle: the server slows you, or returns 429 Too Many Requests with a Retry-After header telling you exactly how long to wait (Cloudflare's Error 1015 is this). Some sites tolerate small bursts — allowing 100 a minute but only throttling at 120 — while others throttle early, say at 15 a minute, and only hard-block at 30. Hard limits are strict ceilings: an API that allows 1,000 requests an hour locks you out completely once you exceed it. Push a soft limit repeatedly and it escalates: 429 becomes 403, then a temporary ban of minutes to hours whose window grows each time you trip it, and finally a permanent blacklist of your IP or entire subnet.
Read the response, do not guess
The single biggest upgrade to a scraper is to react to what the server tells you instead of blasting a fixed rate. Learn the vocabulary: 429 means slow down and honour Retry-After; 403 means this identity is flagged, so rotate rather than retry; 503 is often a challenge or temporary rejection; and a 200 that returns a CAPTCHA page is a soft block, not a success. Treat each one differently. The wrong move — retrying the same burned IP on a 403, or ignoring Retry-After and pounding through a 429 — is what turns a soft warning into a permanent ban. Our deep dives on fixing 429s cover the response-handling in detail.
import time, requests
def polite_get(session, url, max_tries=4):
for attempt in range(max_tries):
r = session.get(url, timeout=20)
if r.status_code == 200 and "captcha" not in r.text.lower():
return r
if r.status_code == 429: # obey the server
wait = int(r.headers.get("Retry-After", 2 ** attempt))
time.sleep(wait)
continue
if r.status_code in (403, 503): # this exit is burned
rotate_ip(session) # fresh IP, then retry
time.sleep(2 ** attempt) # exponential backoff
continue
return r
return None
Budget requests per domain
A crawler that touches many sites should never apply one global rate to all of them. A small blog and a hardened marketplace tolerate wildly different loads, so give each domain its own budget. A per-host token bucket is the clean pattern: assign a conservative rate per domain, refill it over time, and let requests to different hosts run in parallel while requests to the same host stay within its limit. Start slow on a new target and let the response codes tell you whether you can speed up.
import time
from collections import defaultdict
class DomainLimiter:
def __init__(self, per_min=30):
self.gap = 60.0 / per_min # min seconds between hits per host
self.last = defaultdict(float)
def wait(self, host):
now = time.time()
delay = self.gap - (now - self.last[host])
if delay > 0:
time.sleep(delay)
self.last[host] = time.time()
# 30 req/min to any single host; different hosts proceed independently
limiter = DomainLimiter(per_min=30)
limiter.wait("example.com")Spread the load so each IP stays polite
Here is the move that reconciles "polite" with "scales": politeness is measured per IP, but your total throughput is the sum across IPs. If a target tolerates 30 requests a minute per address, one IP caps you at 30 — but ten clean IPs, each doing 30, give you 300 a minute while every individual exit stays courteous. A rotating residential gateway does this automatically, handing a fresh IP per request across 90M+ addresses so no single exit ever looks aggressive. This is not a trick to hammer harder; it is distributing genuine load so no one server bears a suspicious spike. The IP-rotation fundamentals are in what IP rotation is and why it matters.
Spread load across clean rotating IPs
Take less, cache more, pick your hours
The politest request is the one you never send. Three habits cut load without costing you data. First, cache aggressively and use conditional requests — send If-Modified-Since or If-None-Match so an unchanged page returns a tiny 304 instead of the full body, which spares the server and your bandwidth. Second, budget concurrency deliberately: a semaphore capping in-flight requests per domain keeps you from accidentally bursting. Third, schedule heavy jobs for the target's off-peak hours, when your traffic is a smaller share of theirs and less likely to trip a threshold. Combined, these can halve the requests a job needs — the discipline behind our wider anti-ban checklist.
One more thing worth saying plainly: check robots.txt and honour a site's stated crawl expectations. Politeness is not only self-preservation — being a good citizen keeps the open web scrapeable for everyone. Our guide on robots.txt in practice covers what it does and does not bind.
Frequently asked questions
How do I avoid rate limiting when scraping?
Pace each domain with its own request budget, honour Retry-After on 429s, back off exponentially, and spread load across a rotating pool of clean IPs so no single address looks aggressive. Add caching and conditional requests to send fewer requests overall, and schedule heavy jobs for the target's off-peak hours.
What does HTTP 429 mean and how should I handle it?
429 Too Many Requests is a soft rate limit — the server is asking you to slow down, not banning you. Read the Retry-After header and wait exactly that long before retrying; if it is absent, back off exponentially. Never ignore it and keep hammering, because repeated 429s escalate to 403s and then to timed or permanent bans.
How many requests per minute is safe?
There is no universal number — it depends entirely on the target. A small site may tolerate only a few requests a minute; a large one, far more. Start conservatively (say 20–30 a minute per IP), watch for 429s, and adjust from the responses. Scale total throughput by adding IPs, not by raising the rate on a single one.
Do rotating proxies count as impolite?
Not when used to distribute genuine load. Rotation keeps each individual IP within a courteous rate while your aggregate throughput grows — the server never sees a suspicious spike from any one address. It becomes impolite only if you use it to exceed what the site can reasonably handle in total; pace the aggregate, not just the per-IP rate.
Polite and scalable are not opposites. Read the signals, honour Retry-After, budget per domain, cache what you can, and spread the rest across clean rotating IPs. You end up faster than the reckless scraper — because you are the one who never gets banned.