Imperva Incapsula web scraping without storing block pages

Web Unlocker · Imperva (Incapsula)

Imperva Incapsula web scraping has a trap: the check page often arrives with status 200, so a scraper saves it as data. The Web Unlocker sends your URL from a residential or mobile IP, lets a real browser clear the check when it is served, and marks every answer that is still a block page as blocked.

Imperva-protected pages are among the 27 on our canary list. In the run of 11 November 2026, with 3 calls per page, 72 of 81 calls came back as the real page (88.9%); the median call took 4.1 s. Method, list and runs: Web Unlocker status

Imperva's two block pages, and why the status misleads

Imperva, which many sites still show as Incapsula, has two layers that stop scrapers, and each has its own page. Neither can be recognised by the status code alone.

Where to lookWhat Imperva shows
Status code403 on a refusal. The check page can arrive with 200, so the status does not settle it.
Bot check page"Pardon Our Interruption. As you were browsing, something about your browser made us think you were a bot."
Incident page"Request unsuccessful. Incapsula incident ID:" followed by a number with a dash in it.
FrameAn iframe whose source starts with /_Incapsula_Resource?. On a blocked answer it is the only content of the page.
Response headersx-iinfo on every answer, and x-cdn: Imperva or x-cdn: Incapsula.
Cookiesvisid_incap_ and incap_ses_ followed by the id of the site, nlbi_, and reese84 on sites that run Advanced Bot Protection.
What Imperva, formerly Incapsula, answers a script and what one QuantumProxies Web Unlocker call returns: on the left a check page titled Pardon Our Interruption that arrives with status 200, or the Incapsula incident ID page, with the _Incapsula_Resource mark, in the middle that page as it came back instead of the site, on the right the answer one call later, with status 200, blocked false, the tier that served the page and the vendor named on a try that was blocked.
An Imperva check page as a script receives it, and the answer of one Web Unlocker call for the same URL.

Two layers, two refusals

Imperva is a cloud firewall in front of the site. Its firewall layer classifies the client on the first request from the IP reputation, the handshake and the headers; a refusal there is the incident page with its ID. Its bot layer, Advanced Bot Protection, loads a script that inspects the browser and writes the reese84 cookie; a client that never runs it gets "Pardon Our Interruption".

The session cookies visid_incap_ and incap_ses_ tie later requests to that first verdict, and a request without them starts from zero. For Imperva Incapsula web scraping that means two separate things to get right, and a status code that will not tell you which one went wrong.

How the Web Unlocker answers each layer

The first attempt is a browser-grade request from a residential exit, which is what the firewall layer judges. When the bot layer serves its check, a real browser opens the page, lets the script finish and returns the page behind it. A refused attempt is retried from a fresh exit.

Then comes the part a homemade scraper tends to skip: every page that comes back is read before it is handed to you. If it is the check or the incident page, the answer says blocked true, whatever status the site used. That is how a page of nothing stays out of your database.

The call, with the check on blocked

The Python example below is complete: one POST, then the fields you should log. On Imperva sites, branch on blocked and not on status.

If you prefer an error to a flag, add fail_on_block true to the request. A page that is still blocked then answers 424 target_blocked, which your retry logic can catch like any other error status.

import requests

r = requests.post(
    "https://app.quantumproxies.io/api/v1/scraper/unlock",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    json={"url": "https://example.com/", "country": "us"},
    timeout=130,
)
page = r.json()["payload"]
print(page["status"], page["blocked"], page["latency_ms"], page["cost"])
print(page["body"][:500])

Blocked pages and fail_on_block in the API docs

What each field means on an Imperva site

FieldMeaning
statusThe status the site sent. On Imperva it can be 200 for a check page, which is why blocked exists.
blockedtrue when the page that came back is the check or the incident page, whatever its status.
blockClassjs_challenge for the check that a browser clears, captcha when the page asks a person, access_denied for the incident page.
vendorimperva, also for sites that still print Incapsula.
tier_servedtls or browser, the tier that produced the page.
attemptsThe number of attempts. attempt_log names the tier, the exit country and the outcome of each.
costBytes moved by the call and their cost.

Blocked on Imperva: which page was it

The incident ID in the body means something only to the site's administrators. Keep the request_id of our answer instead: it identifies the call, its attempts and its exits.

The block class tells you which of the two layers answered, and so what is worth trying.

Find the check pages you already stored

If an older scraper has been running against an Imperva site, some of what it saved may not be data. Look for rows whose body is tiny and contains _Incapsula_Resource or "Request unsuccessful". Those URLs were never fetched.

Send them again through the Web Unlocker and compare. The blocked flag of the answer and the size of body tell you in one pass which pages were real. From then on, the session reuse does the upkeep: after a pass the cookies stay with the client that earned them, and your next calls to that site do not start from zero.

Proxies, browsers and the Web Unlocker on Imperva sites

A residential proxy answers the firewall layer's question about the IP and leaves the handshake and the script unanswered. A browser farm of your own answers all three and is yours to maintain. The Web Unlocker sits between the two: one URL in, one page out, and a flag you can trust when the page is not the page.

What it leaves alone

Captchas that need a person are reported, not solved. Pages behind a login stay behind it: the Web Unlocker does not sign in or open accounts. Banking, payment and login-provider destinations are refused outright, with the error target_denied.

What you may collect from a public page is decided by the site's terms and by the law that covers the data. The presence of Imperva changes neither.

Frequently Asked Questions

Can the Web Unlocker scrape Imperva (Incapsula) protected sites?

Yes, for public pages. The request leaves from a residential or mobile IP as a browser-grade request, and a real browser clears the JavaScript check when Imperva serves it. The answer carries the page, the tier that served it and a blocked flag that is true whenever the page is still a block page.

Why read blocked instead of the status code on Imperva sites?

Because Imperva often serves its check page with status 200. The body then holds only a frame that loads /_Incapsula_Resource, and code that trusts the status stores it as content. The Web Unlocker reads the page before returning it and sets blocked true when it is the check or the incident page.

How do I make a blocked page an error in my pipeline?

Add fail_on_block true to the request. A page that is still blocked after every attempt then answers 424 target_blocked instead of 200 with a flag, with the same blockClass, vendor and attempt log in the error payload. On the forward proxy the matching header is x-qp-fail-on-block.

Does it work on sites that still say Incapsula?

Yes. Incapsula is the former name of Imperva's cloud firewall, and many sites still print it in the incident page, in the x-cdn header and in the incap_ses_ cookie. The Web Unlocker treats them as one system and names it imperva in the vendor field of the answer.

What if Imperva asks for a captcha?

Then the call reports it. A captcha that asks for a person is not solved: the answer has blocked true and blockClass captcha, and body holds the page that was served. The check that a browser clears by itself is a different case and is handled by the browser tier.

The Web Unlocker by protection system

Sources

Related

If you're an AI agent

Skip the marketing. QuantumProxies.io publishes a machine-readable site map, Markdown for every page, a free MCP server, and APIs billed from the same balance as your proxies.

Connect in one command: npx -y quantumproxies-mcp