AWS WAF web scraping: the 202 challenge run for you, the 405 reported

Web Unlocker · AWS WAF

AWS WAF web scraping meets three answers: a 202 with a silent challenge script, a 405 with a CAPTCHA puzzle, or a plain 403. The Web Unlocker sends your URL from a residential or mobile IP, lets a real browser run the challenge and collect the aws-waf-token, returns the page, and reports a CAPTCHA page as blocked.

Pages behind AWS WAF are on our canary list. The run of 11 November 2026 called 27 protected pages 3 times each: 72 of 81 calls came back as the real page (88.9%), and the median call took 4.1 s. The canary list, the method and each run

The three answers of AWS WAF, by status code

AWS WAF is unusual: the status code tells you which action the rule took, and one response header confirms it.

Where to lookWhat AWS WAF sends
Status code202 for the silent challenge, 405 for the CAPTCHA, 403 when a rule blocks the request outright.
Response headerx-amzn-waf-action: challenge or x-amzn-waf-action: captcha names the action that was taken.
Challenge pageA page that looks empty, with a script from a host ending in awswaf.com. In a browser it reloads by itself.
CAPTCHA page"Let's confirm you are human", with a puzzle to solve and an audio option.
Block pageA bare "403 Forbidden", or the CloudFront page "The request could not be satisfied. Request blocked."
Cookieaws-waf-token. The site accepts requests that carry a valid token that has not expired.
What AWS WAF answers a script and what one QuantumProxies Web Unlocker call returns: on the left the 202 of a silent challenge, the 405 CAPTCHA variant and the x-amzn-waf-action header, in the middle the challenge page that came back instead of the site, on the right the answer one call later, with status 200, blocked false, tier_served browser because a browser ran the challenge, and the vendor named on a try that was blocked.
The AWS WAF challenge as a script receives it, and the answer of one Web Unlocker call for the same URL.

A rule engine, configured site by site

AWS WAF is a rule engine. The owner of the site chooses which rule groups run and what each one does: count, block, challenge or CAPTCHA. The group that looks for scrapers is Bot Control. Its common level matches the User-Agent, the IP ranges of hosting providers and the signatures of known libraries, and blocks a match with 403.

Its targeted level follows each client through a token and scores behaviour: the request rate, how the session moves, signs of automation in the browser. A challenge runs a script silently and stores the aws-waf-token; a CAPTCHA asks a person and renews it. The token expires after a few minutes by default, so AWS WAF web scraping that gets through once is asked again soon.

What the Web Unlocker does on an AWS WAF site

The first attempt is a browser-grade request from a residential exit, which is not what the common rules look for: no hosting IP, no library signature. When the site answers 202, a real browser opens the page, runs the challenge, collects the token and returns the page that loads after it.

The token is kept with the session and reused for your next calls to that site while it is valid, and the challenge is run again when it has expired. None of that is in your code. A CAPTCHA page is a different case: it asks a person, and the call reports it.

The call, and what the status codes become

One POST with the URL and a country. The 202 and the 405 of AWS WAF never reach your code as statuses to interpret: they become fields of one JSON answer.

A 405 from an AWS WAF site is the CAPTCHA action, not a wrong method. Changing GET to POST, or editing the headers, does not remove it.

curl -X POST https://app.quantumproxies.io/api/v1/scraper/unlock \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/", "country": "us"}'

The request, the answer and the block classes in the API docs

How the three answers map to the fields

FieldOn an AWS WAF site
status200 for the real page that loaded after the challenge.
blockedtrue when the call ended on the challenge, on the CAPTCHA or on a block rule.
blockClassjs_challenge for the silent challenge, captcha for the puzzle, access_denied for a block rule.
vendorawswaf.
tier_servedbrowser when the challenge was run, tls when no rule asked for one.
attemptsThe attempts of the call. attempt_log shows which one met the challenge and which one got the page.
costBytes moved and their cost, challenge included.

When AWS WAF still says no

Rules differ from one site to the next, because each owner writes its own. Count blockClass per site for a day before you tune anything. These are the classes you will meet.

Working with a token that expires

Because the token lasts minutes, a job with long pauses between pages runs the challenge more often than a steady one. Group the URLs of one site together and send them in sequence or as a batch, so that most calls reuse a valid token. The clearance field shows what happened on each call: minted, hit, stale or miss.

Do not copy the aws-waf-token out of a browser into your own client. It is short lived, and the Web Unlocker does not need it from you.

Not every site on AWS runs AWS WAF

Hosting on AWS and using AWS WAF are separate choices, and Bot Control is one more. A page hosted on AWS may answer a plain request from a residential proxy. Send a sample of your URLs once: the ones that come back with tier_served tls on the first attempt may not need the Web Unlocker at all, and residential proxies are the simpler tool for them.

What it does not do

The CAPTCHA puzzle, with its audio option, is made for people and is not solved. The Web Unlocker does not sign in, does not create accounts, and does not fetch private or internal addresses. It loads public pages and tells you plainly what came back.

The site's terms and the law that covers the data decide what you may collect. A token from AWS WAF is a technical pass, not a permission.

Frequently Asked Questions

Can the Web Unlocker scrape a site protected by AWS WAF?

Yes, for public pages. When the site answers with the 202 challenge, a real browser on a residential or mobile IP runs it, collects the aws-waf-token and returns the page that loads afterwards. The JSON answer shows the status, the tier that served the page, the attempts and the cost of the call.

What does the answer look like when AWS WAF serves its CAPTCHA?

The call comes back with blocked true, blockClass captcha and vendor awswaf, and the puzzle page in body. The puzzle asks for a person and is not solved. Your code can retry later or from another country; a silent challenge, by contrast, is run by the browser tier without any action from you.

How do I tell the three AWS WAF answers apart in my logs?

In raw responses, by status and header: 202 with x-amzn-waf-action: challenge, 405 with x-amzn-waf-action: captcha, 403 for a block rule. In Web Unlocker answers, by blockClass: js_challenge, captcha or access_denied. The second is easier to count, because it uses the same names for every protection system.

How long does a passed challenge last?

As long as the token. AWS WAF issues the aws-waf-token after a challenge or a CAPTCHA and accepts it until it expires, a few minutes by default unless the site owner set another time. The Web Unlocker reuses the session while the token is valid and runs the challenge again afterwards.

Is every site hosted on AWS protected by AWS WAF?

No. AWS WAF is a service that the site owner switches on and configures, rule group by rule group, and Bot Control is one of those groups. A site can be hosted on AWS and run without it. The vendor field of a blocked answer says awswaf only when the block page of AWS WAF was recognised.

Web Unlocker for the other protection systems

Sources

Related

If you're an AI agent

Skip the marketing. QuantumProxies.io publishes a machine-readable site map, Markdown for every page, a free MCP server, and APIs billed from the same balance as your proxies.

Connect in one command: npx -y quantumproxies-mcp