# Scrape Cloudflare Protected Sites by API | QuantumProxies.io

Scrape Cloudflare protected sites with one Web Unlocker call: a residential exit, a browser-grade request and a real browser when the challenge is served.

## Scrape Cloudflare protected sites with one Web Unlocker call

**Web Unlocker · Cloudflare**

To scrape a Cloudflare protected site you need a residential or mobile IP, a request that looks like one real browser, and a browser that can run the challenge script when it is served. The Web Unlocker does the three for each URL you send and returns the page with its status, or tells you it stayed blocked.

9 of 9 calls to Cloudflare-protected pages came back as the real page in our canary of 11 November 2026. Across the whole run, 27 protected pages called 3 times each, it was 72 of 81, with a median of 4.1 s. [Method and every run: Web Unlocker status](https://quantumproxies.io/web-unlocker/status)

## How to tell that Cloudflare is what stopped your request

Cloudflare sits in front of the site and answers in its own voice. Five places in the response tell you that it was Cloudflare, and which of its features said no.

| Where to look | What a blocked request shows |
| --- | --- |
| Status code | 403 with the challenge page. 429 or error 1015 when the site rate limits you. 503 on older JavaScript challenges. |
| Response headers | cf-mitigated: challenge on a challenge. server: cloudflare and a cf-ray id are on every answer, blocked or not, so they prove that Cloudflare is there and nothing more. |
| Page text | "Just a moment..." with "Verifying you are human", and a script loaded from /cdn-cgi/challenge-platform/. A Turnstile box says "Verify you are human". |
| Error pages | Error 1020 Access denied (a rule of the site owner), 1015 (rate limited), 1010 (the browser signature was refused), 1009 (your country is blocked), or "Sorry, you have been blocked" with a Ray ID. |
| Cookies | cf_clearance after a passed challenge; it belongs to the visitor that earned it. __cf_bm belongs to bot scoring. |

![What Cloudflare answers a script and what one QuantumProxies Web Unlocker call returns: on the left the 403 with the words Just a moment and the cf-mitigated: challenge header, in the middle the challenge page that came back instead of the site, on the right the answer one call later, with status 200, blocked false, tier_served and the vendor named on a try that was blocked.](https://quantumproxies.io/images/web-unlocker/systems/cloudflare.webp?v=20261112)

A Cloudflare challenge as a script receives it, and the answer of one Web Unlocker call for the same URL.

## What it takes to scrape a Cloudflare protected site

Cloudflare scores a request before the site sees it. The score has three inputs that a script usually gets wrong: where the request comes from, what the client looks like on the wire, and whether it can run JavaScript. Hosting IPs score worse than home and mobile connections. A Python or Go client that names Chrome in its User-Agent does not connect like Chrome, and the mismatch shows in the TLS and HTTP/2 handshake. The challenge page expects a browser to run its script.

Fix one of the three and the other two still stop you. That is why a residential proxy alone often changes nothing on a protected page, and why a headless browser on a server IP is challenged again and again. The three have to be right in the same request, every time, and they have to stay right when browsers and Cloudflare change.

## What the Web Unlocker does with a Cloudflare URL

You send the URL. With mode auto, the default, the first attempt is a browser-grade request from a residential exit in the country you chose. When the site answers that with the page, the call ends there and tier_served says tls.

If the answer is a challenge, the call does not come back yet. It is retried from a fresh exit, and when the site insists on its JavaScript check a real browser opens the page, waits for the check to clear and hands back what it sees, with tier_served browser. You get one answer for the whole sequence, and attempt_log lists each step.

- Residential exits by default, 4G/5G carrier exits with tier mobile
- Country, state and city targeting on every call
- GET, HEAD and OPTIONS are retried; a POST is sent exactly once
- After a passed challenge the session is reused for your next calls to that site

## The call: one POST from Python

Send the URL and a country to the REST endpoint with your API key. There is nothing to install and no certificate. The same engine also answers as a forward proxy on unlock.quantumproxies.io:9000, for a scraper that already speaks proxy.

The answer is JSON. The page is in body; the fields below tell you what happened on the way.

```
import requests

r = requests.post(
    "https://app.quantumproxies.io/api/v1/scraper/unlock",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    json={"url": "https://example.com/", "country": "us"},
    timeout=130,
)
page = r.json()["payload"]
print(page["status"], page["blocked"], page["latency_ms"], page["cost"])
print(page["body"][:500])
```

[Every field of the request and the answer, in the API docs](https://quantumproxies.io/docs/#tag/Web-Unlocker)

### What comes back for a Cloudflare URL

| Field | What it says |
| --- | --- |
| status | The status of the page that was served: 200 when the real page arrived. |
| blocked | false for the real page. true when every attempt ended on a Cloudflare page; body then holds that page, so you can look at it. |
| blockClass | On Cloudflare usually js_challenge for "Just a moment...", captcha when the box needs a person, and access_denied, rate_limited or geo for a setting of the site. |
| vendor | cloudflare, on a blocked answer and on each blocked entry of attempt_log. |
| tier_served | tls when the browser-grade request was enough, browser when the challenge had to be run. |
| attempts | How many attempts the call made. attempt_log has one entry per attempt, with its tier, exit country, outcome and time. |
| cost | The bytes the call moved and what they cost, so you can add up a job site by site. |

## When a Cloudflare call still comes back blocked

A blocked answer is not an HTTP error. It is a 200 from our API with blocked true, so your code decides what happens next. If you would rather get an error than a flagged answer, send fail_on_block true: a page that stays blocked then answers 424 target_blocked with the same diagnostics.

On Cloudflare the block class points to the next step.

- js_challenge: leave mode on auto. With mode tls no browser is opened, and a challenged page is returned as it is
- captcha: the box asked for a person. Retry later; the next call leaves from another exit
- geo, or error 1009 in the body: set country to one the site serves
- rate_limited, or error 1015: the exit went over the site's limit. Spread the calls and do not pin one session
- access_denied, or error 1020: a rule of the site owner. Try another country, then the mobile tier

## Many pages from one Cloudflare site

The first call to a challenged site is the slow one, because the check itself takes a few seconds in any browser. Keep calling from the same account: once a challenge is passed the session is kept, and the clearance field of the answer says minted on the call that passed it and hit on the calls that reuse it.

For a list of URLs, the batch endpoint takes up to 100 at a time and can notify a webhook when they are done. For a page you ask for often, cache_ttl answers a repeated GET from a stored copy for as long as you set.

## Web Unlocker or a plain proxy on Cloudflare sites

Not every site on Cloudflare challenges its visitors. If your client already gets the page from a home IP, residential proxies are the simpler buy: the same exits, without the retries and the browser. Use the Web Unlocker for the URLs that answer 403, "Just a moment..." or a 1xxx error. A quick way to sort a list is to send it once and count blockClass and tier_served per site.

## What it does not do

It fetches pages anyone can open in a browser without signing in: prices, listings, articles, search results. It does not log in, and it does not create accounts. A Turnstile box that needs a click, or any other captcha that asks for a person, is not solved: the call reports blockClass captcha.

Cloudflare in front of a site says nothing about whether you may collect its pages. The terms of the site and the law that covers the data, personal data above all, apply exactly as they would without it.

## Frequently Asked Questions

### Can the Web Unlocker scrape a Cloudflare protected site?

Yes, when the page is public. You send the URL; the first attempt is a browser-grade request from a residential IP, and if Cloudflare answers with its JavaScript challenge a real browser opens the page and returns what it sees. The answer carries the status, the tier that served the page and the number of attempts.

### Do I need to run my own headless browser for Cloudflare pages?

No. In mode auto the Web Unlocker opens a real browser only when the site asks for one, and tier_served tells you whether it did. If you know that a site always challenges, send mode browser to render straight away. Mode tls never opens the browser and returns a challenged page as it is.

### What happens when Cloudflare shows a Turnstile box that needs a click?

Turnstile often clears by itself in a real browser, and then the page comes back. When it does not clear, the answer says so: blocked true with its block class, never the challenge page passed off as content. A retry leaves from a different exit and may not be asked at all.

### Which country should a Cloudflare request leave from?

A country the site serves. Error 1009 is a country rule set by the site owner, and no change to the client lifts it. Set country on the call, and state or city when the site shows different content by region. On the proxy, a request without a country leaves from the site's own country when its domain names one.

### Can I use it from Scrapy, Playwright or n8n?

Yes. Anything that can send a POST can call the REST endpoint, an n8n HTTP Request node included. A scraper that already speaks proxy, such as Scrapy or Playwright, can point at unlock.quantumproxies.io:9000 with an unlocker user and password instead, and read the verdict in the x-qp-unlocker-blocked and x-qp-block-class headers.

## The Web Unlocker on other protection systems

- [Web Unlocker for DataDome](https://quantumproxies.io/web-unlocker/datadome)
- [Web Unlocker for Akamai Bot Manager](https://quantumproxies.io/web-unlocker/akamai)
- [Web Unlocker for PerimeterX (HUMAN)](https://quantumproxies.io/web-unlocker/perimeterx)
- [Web Unlocker for Kasada](https://quantumproxies.io/web-unlocker/kasada)
- [Web Unlocker for Imperva (Incapsula)](https://quantumproxies.io/web-unlocker/imperva)
- [Web Unlocker for F5 BIG-IP and Shape](https://quantumproxies.io/web-unlocker/f5)
- [Web Unlocker for AWS WAF](https://quantumproxies.io/web-unlocker/aws-waf)

## Sources

- [Cloudflare docs: interstitial challenge pages](https://developers.cloudflare.com/cloudflare-challenges/challenge-types/challenge-pages/)
- [Cloudflare docs: detecting a challenge response (the cf-mitigated header)](https://developers.cloudflare.com/cloudflare-challenges/challenge-types/challenge-pages/detect-response/)
- [Cloudflare docs: clearance and the cf_clearance cookie](https://developers.cloudflare.com/cloudflare-challenges/concepts/clearance/)
- [Cloudflare docs: 1xxx errors (1009, 1010, 1015, 1020)](https://developers.cloudflare.com/support/troubleshooting/http-status-codes/cloudflare-1xxx-errors/)
- [QuantumProxies.io API docs: Web Unlocker request, answer and blocked pages](https://quantumproxies.io/docs/#tag/Web-Unlocker)
- [Web Unlocker status: the canary list, the method and every run](https://quantumproxies.io/web-unlocker/status)

## Related

- [Residential Web Unlocker: the default tier](https://quantumproxies.io/web-unlocker/residential)
- [Mobile Web Unlocker on 4G/5G carrier IPs](https://quantumproxies.io/web-unlocker/mobile)
- [Web Unlocker status and the latest canary run](https://quantumproxies.io/web-unlocker/status)
- [Cloudflare error 1020 Access Denied, explained](https://quantumproxies.io/blog/cloudflare-error-1020-access-denied)
- [Residential proxies for sites that do not challenge](https://quantumproxies.io/residential-proxies)
- [All Web Unlocker](https://quantumproxies.io/web-unlocker)