How to Scrape Reddit Data in 2026: JSON Endpoints and the 1,000 Cap

Since Reddit's API went paid, the goldmine got harder to reach — but the public .json endpoint still works. Here's how to page it, beat the 1,000-item ceiling, and collect at scale without getting rate-limited.

Reddit is one of the richest opinion datasets on the web — real people arguing about products, tools, brands and news in searchable communities. Since Reddit moved its official API to paid pricing in 2023 (widely reported at $0.24 per 1,000 calls, which shut down third-party apps like Apollo), collecting that data at scale got trickier. But there's a route the pricing change didn't close: the public .json endpoint that backs every Reddit page. This guide covers how to page it, how to beat the 1,000-item ceiling, and how to collect without tripping the rate limit.

Append .json to any Reddit URL

Almost every Reddit URL returns structured JSON if you add .json. No OAuth, no client secret — just an HTTP GET. A subreddit listing, a post with its comment tree, a user's history: all of it. Start here:

import requests

headers = {"User-Agent": "research-script/1.0 by u/yourname"}
proxies = {"http": "http://USER:PASS@gate.quantumproxies.io:8000",
           "https": "http://USER:PASS@gate.quantumproxies.io:8000"}

url = "https://www.reddit.com/r/webscraping/new.json?limit=100"
r = requests.get(url, headers=headers, proxies=proxies, timeout=20)
data = r.json()

for child in data["data"]["children"]:
    post = child["data"]
    print(post["title"], post["ups"], post["num_comments"])

Set a descriptive User-Agent — Reddit is stricter with generic ones. The JSON gives you title, score (ups), upvote ratio, comment count, author and timestamps directly, so there's nothing to parse out of HTML.

Pagination with the after token

Listings page with a cursor, not page numbers. Each response includes an after token (the fullname of the last item, like t3_abc123); pass it back to get the next batch. Loop until after comes back null:

def fetch_listing(subreddit, sort="new", pages=10):
    after, out = None, []
    for _ in range(pages):
        url = f"https://www.reddit.com/r/{subreddit}/{sort}.json?limit=100"
        if after:
            url += f"&after={after}"
        data = requests.get(url, headers=headers, proxies=proxies, timeout=20).json()
        out += [c["data"] for c in data["data"]["children"]]
        after = data["data"].get("after")
        if not after:
            break
    return out

Key numbers for scraping Reddit: 1,000-item list cap, $0.24 per 1,000 API calls, the .json endpoint, and combining four sorts — The 1,000-item cap is per sort — combining sorts and time filters is how you get past it.

The 1,000-item ceiling (and how to break it)

Here's the limit that surprises people: any Reddit listing stops at roughly 1,000 items. Page a subreddit's new feed and after ~1,000 posts the after token runs dry, no matter how many more posts exist. This is a platform-wide behaviour, not a scraper bug. It does not apply to comments inside a single post — you can pull thousands of those. To collect more than 1,000 posts from a community, multiply your views of it:

Combine sorts. Scrape new, then top, hot and controversial. Each is a different 1,000-item window into the same subreddit, and the overlap is partial — together they surface far more unique posts.
Add time filters. On top, iterate ?t=hour, day, week, month, year, all to reveal different slices.
Use search. Keyword searches surface older posts that fell off the main lists entirely.
Run incrementally. Poll new on a schedule and dedup by post id, so you capture everything before it ages past the 1,000-item window.

# combine sorts to break past the 1,000-item cap
seen, posts = set(), []
for sort in ("new", "top", "hot", "controversial"):
    for post in fetch_listing("webscraping", sort=sort, pages=10):
        if post["id"] not in seen:
            seen.add(post["id"])
            posts.append(post)
print(f"{len(posts)} unique posts across four sorts")

Rate limits and staying unblocked

One IP polling the JSON endpoints hard gets throttled quickly — Reddit returns 429s and eventually blocks. Two things keep a collection job healthy: pace it (a short delay between requests beats a burst that trips the limiter) and spread it across IPs so no single address carries the whole load. Routing through residential proxies lets a research crawl look like many ordinary readers rather than one aggressive client, and a Scraper API handles that rotation and pacing for you. If you're hitting 429s constantly, our guide on rate limits and backoff covers the pacing maths.

What to do with it: listening at scale

The reason to collect Reddit data is to listen. Track mentions of your brand or a competitor across communities, run sentiment on the comment trees, watch a niche subreddit for recurring problems worth building around, or spot a trend before it hits the mainstream. For anything past keyword matching, feeding the raw text through an LLM extraction step turns messy threads into structured signals — themes, complaints, feature requests. Reddit's public data is fair game to collect, but it is people's posts; if you plan to process usernames or personal detail, our overview of scraping legality in 2026 is worth a read (informative, not legal advice).

Reddit collection loop: paging the .json endpoint through a residential exit, deduping by post id, and using the data for listening — Page the JSON, pace it across clean IPs, dedup by fullname, and the data feeds brand listening or trend research.

Collect Reddit data on clean residential IPs

Frequently asked questions

Can you still scrape Reddit after the API changes?

Yes. The official API is now paid, but the public .json endpoint behind every Reddit page still returns structured data over a plain HTTP GET with no OAuth. It's rate-limited, so pace requests and spread them across IPs, but for research-scale collection it remains the most practical route. old.reddit.com and RSS feeds are additional lightweight sources.

Why does Reddit stop at 1,000 posts?

Reddit caps any single listing — a subreddit sort, a user's history, a search — at about 1,000 items platform-wide; the pagination cursor simply stops. It's not your scraper. Get past it by combining sorts (new, top, hot, controversial), applying time filters, using search, and running incremental jobs that capture posts before they age out of the window.

Do I need proxies to scrape Reddit?

For a handful of requests, no. For sustained collection you do: a single IP hammering the JSON endpoints gets 429-throttled and then blocked. Residential proxies spread the load so the traffic reads as many ordinary readers, and pacing between requests keeps you under the limiter. Together they let a research crawl run continuously without tripping defences.

Is scraping Reddit legal?

Collecting publicly visible pages is broadly defensible, but Reddit's terms restrict automated access and the data includes user-generated content and usernames. Keep to public data, respect rate limits, and be careful with anything that identifies individuals. This is general information, not legal advice — check Reddit's current terms and your jurisdiction before a large project.

The API going paid didn't lock the door — it just moved the handle. The .json endpoint, cursor pagination, and a strategy for the 1,000-item cap get you the data; residential IPs and polite pacing keep the collection running. From there it's a listening problem, and Reddit is one of the best signals you can point it at. If you also want the API-pricing story on the other platforms, our post on X/Twitter data without the API maps the same terrain.

Scrape Reddit at scale with a Scraper API