How SERP Scraping Actually Works in 2026 (After Google Killed num=100 and Required JavaScript)
Two quiet changes — mandatory JavaScript and the death of the num=100 parameter — rewrote the economics of search scraping. Here is what actually changed, why your rank tracker broke, and how to collect SERP data reliably in 2026.
If your rank tracker suddenly went quiet in late 2025, or your homegrown Google scraper started returning empty pages, you were not imagining it. In the span of nine months Google made two changes that, individually, looked like minor housekeeping and, together, quietly rewrote the economics of search scraping. This is a practical guide to what actually changed, why it broke so much tooling at once, and how to collect SERP data reliably in 2026 without burning your infrastructure — or your IPs.
Change one: Google made JavaScript mandatory
On January 17, 2025, Google began requiring JavaScript to use Google Search. Turn it off and you get a short message — "Turn on JavaScript to keep searching" — instead of results. A company spokesperson told TechCrunch the goal was to "better protect our services and users from bots and evolving forms of abuse and spam," and confirmed that fewer than 0.1% of searches come from users with JavaScript disabled.
That 0.1% figure is the tell. At roughly 8.5 billion searches a day, a tenth of a percent is still millions of requests — and a disproportionate share of them were never humans at all. They were rank trackers, SEO crawlers, and cheap scrapers hitting the lightweight, no-JavaScript version of the results page. Search Engine Journal confirmed the read directly: Google is requiring JavaScript to block bots and scrapers, including SEO tools.
Enabling JavaScript allows us to better protect our services and users from bots and evolving forms of abuse and spam, and to provide the most relevant and up-to-date information.
The practical consequence: the era of parsing Google by fetching a URL and running a regex over static HTML is over. The results are now assembled client-side. If your scraper cannot execute JavaScript, it does not see a degraded page — it sees no page.
Change two: the death of num=100
The second change was smaller in appearance and larger in impact. Around September 11, 2025, Google disabled the &num=100 URL parameter — the little flag that told Search to return 100 results on a single page instead of the default 10. For a decade it was the backbone of deep rank tracking: one request, one hundred positions.
When it vanished, the math inverted overnight. To see the top 100 results you now need up to ten paginated requests instead of one. Keyword Insights put the cost bluntly on the day it happened.
Google has killed the n=100 SERP parameter. Instead of 1 request for 100 SERP results, it now takes 10 requests (10x the cost). This impacts our rankings module. We're reviewing options and will update the platform soon.
The ripple hit Google Search Console too. Starting around September 10, SEO teams saw desktop impressions fall sharply while average position appeared to improve. Analyst Brodie Clark's theory — widely discussed but never confirmed by Google — is that a large share of those earlier impressions were never human: they were bots loading 100-result pages via num=100, each registering ten times the impressions a normal page would. Remove the parameter, and the phantom impressions disappear with it. It is a community interpretation, not an official explanation, but it fits the timing exactly.
Why this broke so many tools at once
Most SERP tooling was built on two assumptions that were both true for years and are both false now: that Google would serve usable HTML without JavaScript, and that you could fetch 100 results per request. Remove both in the same year and the failure modes stack:
- Static fetchers return blank or challenge pages instead of results.
- Pagination-based collectors need 10x the request volume for the same coverage, which means 10x the proxy bandwidth and 10x the block surface.
- More requests from the same IP ranges trip anti-bot systems faster, so success rates fall exactly as volume needs to rise.
- Search Console baselines shifted, so historical dashboards no longer compare like-for-like.
As SEO engineer Ryan Jones framed it, the aggressive scraping wave — much of it now feeding AI products — is precisely what pushed Google to fight back, breaking rank checkers and SERP scrapers as collateral. The tooling did not fail because it was badly written. It failed because the ground moved.
What reliable SERP scraping looks like in 2026
The new baseline has three non-negotiables. Miss any one and your success rate collapses under load.
1. Execute JavaScript — but only when you must
Because results are assembled client-side, you need a client that runs JavaScript for the pages that demand it. But spinning up a headless browser for every request is slow and expensive. The efficient pattern is to try a lightweight, real-TLS-fingerprint fetch first and escalate to a full browser render only when the page actually challenges you. Most requests never need the browser; the ones that do get it automatically.
2. Rotate residential exits with real browser fingerprints
Datacenter IPs and default HTTP-client fingerprints are the fastest way to a CAPTCHA. Requests need to arrive from residential IPs, carrying the TLS and header fingerprint of a genuine browser like Chrome. When one exit gets flagged, rotation to a fresh one usually clears the block without any change to the request itself — which is exactly how a resilient collector recovers from an "unusual traffic" page instead of dying on it.
3. Parse into structured data, not raw HTML
Google's markup changes constantly, and post-num=100 the layout is more fragmented, not less. Maintaining your own selectors against a moving target is a full-time job. A structured SERP endpoint that returns clean JSON — organic results, ads, shopping, knowledge panels, People Also Ask, AI Overviews — insulates your pipeline from every cosmetic change on Google's side.
The pattern in practice
Instead of maintaining browsers, proxy pools, and parsers yourself, you hand a query to an endpoint that does all three and returns clean structured results. With the QuantumProxies SERP API a single authenticated request handles JavaScript execution, residential rotation, and parsing for you:
curl --location --request GET \
"https://app.quantumproxies.io/api/v1/serp/google/search?q=best+running+shoes&cc=us" \
--header "Authorization: Bearer YOUR_API_KEY"The response is JSON, not HTML — organic results with positions, ads, shopping units, the knowledge graph, People Also Ask, and AI Overviews where present. No headless browser to babysit, no selector to repair when Google reshuffles the page, and pagination handled so you are not manually rebuilding what num=100 used to give you for free. When one exit trips an anti-bot page, the service retries on a fresh residential IP automatically — the same rotate-to-recover behavior that keeps a pipeline alive under real load.
If you would rather see the full request and response shape before writing any code, the interactive SERP API docs at https://quantumproxies.io/serp-api/docs let you run a live query and copy ready-made snippets for cURL, Python, JavaScript, PHP, Go, and Ruby.
The takeaway
SERP scraping did not get harder in some vague, incremental way — it changed shape twice in one year. JavaScript is now mandatory, so you need a real client. num=100 is gone, so deep coverage costs an order of magnitude more in requests, which means the quality of your proxy layer and the intelligence of your escalation logic now matter more than raw request volume ever did. The teams still collecting clean SERP data at scale in 2026 are not the ones sending more requests. They are the ones sending smarter ones — JavaScript-capable, residential-fingerprinted, and structured from the first byte.