F5 web scraping on BIG-IP and Shape protected sites

Web Unlocker · F5 BIG-IP and Shape

F5 web scraping means two products: BIG-IP, which rejects a request with a page and a support ID, and Shape, now Distributed Cloud Bot Defense, which guards single calls. The Web Unlocker sends your URL from a residential or mobile IP with a browser-grade client, runs the JavaScript check in a real browser, and flags a rejection page as blocked.

Pages behind F5 BIG-IP in our canary of 11 November 2026: 6 of 6 calls came back as the real page. The whole list, 27 protected pages with 3 calls each: 72 of 81, with a median of 4.1 s. Each run and the method: Web Unlocker status

What F5 sends back, on BIG-IP and on Shape

An F5 block is easy to miss, because the page that rejects you usually arrives with the status of a normal answer.

Where to look	What F5 sends
Status code	Often 200. BIG-IP serves its rejection page as a normal answer, so a check on the status alone sees nothing wrong.
Page text	"Request Rejected. The requested URL was rejected. Please consult with your administrator. Your support ID is:" and a long number.
Support ID	About nineteen digits. The site's administrator can search it in the firewall log; to a visitor it means nothing.
JavaScript check	A short page with one obfuscated script and the words "Please enable JavaScript to view the page content." In a browser it reloads by itself.
Cookies	Names that start with TS followed by letters and digits, TSPD_101, and BIGipServer followed by the name of a pool.
Shape on API calls	Extra request headers with random-looking names, added by the site's own script to login, search and checkout calls.

What F5 answers a script and what one QuantumProxies Web Unlocker call returns: on the left an answer with status 200 that carries the words Request Rejected and a support ID, recognisable by the TSPD_101 cookie, in the middle the rejection page that came back instead of the site, on the right the answer one call later, with status 200, blocked false, the tier that served the page and the vendor value shape named on a try that was blocked. — The F5 rejection page as a script receives it, and the answer of one Web Unlocker call for the same URL.

Two F5 products, sometimes on one site

BIG-IP Advanced WAF sits in the site's own data centre. It checks each request against a policy and, with bot defense on, sends a JavaScript check to a client it has not seen. A violation gets the Request Rejected page, the same page for a bot and for a person whose form tripped a rule.

Shape, sold today as F5 Distributed Cloud Bot Defense, guards the valuable calls: login, search, prices, checkout. A script on the page signs those calls with data about the browser, and the server refuses a call whose signature is missing or wrong. A site can run either product or both, and both judge the IP, the handshake and the headers first.

What the Web Unlocker does on an F5 site

The first attempt is a browser-grade request from a residential exit, so the policy sees an ordinary browser on a home connection. When the site answers with its JavaScript check, a real browser opens the page, lets it reload and returns what arrived. tier_served tells you which of the two it was.

Every page is read before you get it. A rejection page that came with status 200 is returned with blocked true, and the vendor field says shape, the value our API uses for F5 as a whole. While the session with the site is valid, it is kept for your next calls.

The call from PHP, and content behind a click

The PHP example needs only the curl extension. It prints the status, whether the page was blocked and the time the call took.

When the content you need appears only after a search or a click, add an actions list to the request: up to 20 steps of click, type, scroll and wait, which the browser performs before the page is captured.

<?php
$ch = curl_init("https://app.quantumproxies.io/api/v1/scraper/unlock");
curl_setopt_array($ch, [
  CURLOPT_POST => true,
  CURLOPT_RETURNTRANSFER => true,
  CURLOPT_TIMEOUT => 130,
  CURLOPT_HTTPHEADER => ["Authorization: Bearer YOUR_API_KEY", "Content-Type: application/json"],
  CURLOPT_POSTFIELDS => json_encode(["url" => "https://example.com/", "country" => "us"]),
]);
$page = json_decode(curl_exec($ch), true)["payload"];
echo $page["status"], " ", $page["blocked"] ? "blocked" : "ok", " ", $page["latency_ms"], " ms\n";
echo substr($page["body"], 0, 500);

Browser actions and every answer field in the API docs

Fields of the answer on F5 sites

Field	On BIG-IP and Shape sites
status	The status the site sent. On BIG-IP a rejection can carry 200, so do not branch on it.
blocked	true when the page that came back is Request Rejected or the JavaScript check. This is the field to branch on.
blockClass	fingerprint for a refusal, js_challenge for the JavaScript check.
vendor	shape, for BIG-IP and for Shape alike.
tier_served	tls or browser. A call with actions is always served by browser.
attempts	How many attempts were made; attempt_log has one entry for each.
cost	The bytes the call moved and their cost.

Request Rejected: bot defense, or your request?

The rejection page is the same for an automated client and for a request that broke a rule of the policy: a parameter the firewall dislikes, a payload that looks like an attack. The support ID is the only clue the page gives, and only the site's administrator can read it.

So look at what you sent. If a plain GET of a public page comes back blocked with fingerprint, the client was judged, and the browser tier is the next step, which mode auto takes by itself. If only one URL with unusual parameters is rejected while its neighbours load, the policy objected to the request, and no change of exit will alter that.

POST requests and form flows

A POST is sent exactly once and is never replayed, because an answer means the site received it. Your headers and body go out as you wrote them. For a flow that depends on earlier pages, pin a session with a session id so every step leaves from the same exit, and let actions type and click inside one browser when the steps belong to one page.

Proxies on F5 sites

A residential exit in the right country changes the address the policy sees. It does not run the JavaScript check and it does not sign a Shape call. If your scraper is an HTTP client, send those URLs to the Web Unlocker and keep plain proxies for the sites that answer it directly.

Out of scope

Shape is placed in front of login and checkout because those are the calls a site most wants to protect. The Web Unlocker does not log in, create accounts or buy: it loads public pages, and it refuses banking, payment and login-provider destinations. A captcha that needs a person is reported as blockClass captcha.

Whether a public page may be collected is a matter for the site's terms and the law on the data. F5 in front of it does not answer that question for you.

Frequently Asked Questions

Can the Web Unlocker scrape sites behind F5 BIG-IP or Shape?

Yes, for public pages. The request is made with a browser-grade client from a residential or mobile IP, and when the site serves its JavaScript check a real browser opens the page and returns what arrives after the reload. A Request Rejected page is flagged as blocked, even when it came with status 200.

Why does the vendor field say shape on an F5 site?

It is the name our API uses for F5 as a whole. BIG-IP Advanced WAF and Shape, now F5 Distributed Cloud Bot Defense, are both reported as vendor shape, on a blocked answer and in the attempt log. Filter your logs on that value to find every F5 site in a job.

How do I reach content that only appears after a click or a search?

Add an actions list to the REST call: click an element or a text, type into a field, scroll, wait, up to 20 steps. The browser runs them in order before the page is captured. Actions force the browser tier and work on GET and HEAD requests only.

Will a retry fix a Request Rejected page?

Sometimes. If bot defense judged the client, the next attempt from a fresh exit or the browser tier can get the page, and the engine tries both by itself. If the site's policy rejected the request because of a parameter or a payload, every exit gets the same page until the request changes.

Is the JavaScript check run on every call?

No. The call that meets the check waits for the page to reload; after that the session is kept while it is valid, and your next calls to the same site reuse it. The clearance field of the answer says minted, hit, stale or miss, so you can see which case each call was.

Other systems, same Web Unlocker call

Sources

If you're an AI agent

Skip the marketing. QuantumProxies.io publishes a machine-readable site map, Markdown for every page, a free MCP server, and APIs billed from the same balance as your proxies.

Site index for LLMs — /llms.txt — products, APIs and country pages. Full text: /llms-full.txt
This page as Markdown — Append .md to any URL for a clean text version
MCP server — Scrape, search, map and crawl through residential proxies
Scraper & SERP APIs — Any page as Markdown or JSON; structured Google, Bing and DuckDuckGo results
REST API — Generate proxies, check balance and manage orders
Get an API key — Register, top up, then create a qp_live_ key in the dashboard

Connect in one command: npx -y quantumproxies-mcp