RAG That Isn't Stale: Fresh Web Data and Refresh Loops

A RAG system is only as fresh as its last crawl. The retrieval math is the easy part — the hard part is a refresh loop that re-crawls, re-chunks and re-embeds each source on its own clock before the answers go stale.

Retrieval-augmented generation is now in roughly half of enterprise AI systems — adoption jumped to 51% in the latest surveys, up from 31% a year earlier. Yet most RAG projects share the same failure mode: they were populated once, and then the world moved on. The retrieval math (embed the query, search the vectors, stuff the top chunks into the prompt) is the easy, well-documented part. The hard part, the part that decides whether your assistant is trustworthy or confidently wrong, is freshness — a pipeline that keeps re-crawling, re-chunking and re-embedding each source before its answers go stale. This guide is about building that refresh loop, with markdown-first ingestion and per-source TTLs at its core.

Stale retrieval is worse than no retrieval

The whole promise of RAG is that the model answers from retrieved evidence instead of frozen training data. Break the freshness and you break the promise: a support bot cites last quarter's return policy, a pricing assistant quotes a discontinued figure, a research copilot misses the update that changed the answer. Worse, the model states it with full confidence because the retrieved chunk looks authoritative. A stale index does not fail loudly — it fails quietly and plausibly, which is the most dangerous kind. Keeping the corpus current is not a nice-to-have; it is the difference between grounding and hallucinating with extra steps.

Ingest markdown, not raw HTML

The quality of everything downstream is set at ingestion. Raw HTML is a disaster for retrieval — navigation, cookie banners, footers and script tags become noise chunks that pollute similarity search and waste context tokens. The clean approach is markdown-first: crawl each page and extract only the primary content as structured markdown, discarding boilerplate before anything is chunked. Our Scraper API returns markdown directly, renders JavaScript on demand and rotates IPs so blocked pages do not leave holes in your corpus — it replaces the brittle Puppeteer-and-BeautifulSoup ingestion layer that most RAG stacks limp along on. The web-data-for-LLMs endpoint wraps crawl, clean and structure into one call built for exactly this.

import requests

# markdown-first ingestion: clean text, JS rendered, IPs rotated
def fetch_markdown(url):
    r = requests.post(
        "https://api.quantumproxies.io/scrape",
        headers={"Authorization": f"Bearer {QP_KEY}"},
        json={"url": url, "format": "markdown", "render": True},
        timeout=60,
    )
    doc = r.json()
    return {
        "url": url,
        "markdown": doc["markdown"],       # no nav, no boilerplate
        "fetched_at": doc["fetchedAt"],     # timestamp for TTL logic
        "content_hash": doc["contentHash"], # skip re-embed if unchanged
    }

Feed your RAG stack clean web data

Chunk small, carry metadata

Once you have clean markdown, split it into chunks of roughly 300 to 500 tokens — small enough that a retrieved chunk is tightly on-topic, large enough to keep a coherent thought. Use a recursive splitter (LangChain or LlamaIndex both ship one) that respects sentence and heading boundaries rather than cutting mid-word. The critical, often-skipped step is metadata: every chunk must carry its source URL, a timestamp, and a content hash. That metadata is what makes freshness possible — retrieval can filter by recency, and your refresh loop can tell which chunks belong to a source that just changed.

from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=1800, chunk_overlap=200,  # ~400 tokens per chunk
)

def to_chunks(doc):
    parts = splitter.split_text(doc["markdown"])
    return [{
        "text": p,
        "source": doc["url"],
        "fetched_at": doc["fetched_at"],   # drives recency filtering
        "ttl": ttl_for(doc["url"]),          # per-source refresh clock
    } for p in parts]

Embed those chunks with a current model — text-embedding-3-large, a Cohere model, or an open Sentence-BERT variant — and store the vectors alongside the metadata in a vector database such as Pinecone, Weaviate, Milvus or FAISS. Tune similarity thresholds and, importantly, enable metadata filtering so a query can demand, say, only chunks fetched in the last 30 days for a pricing question.

Per-source TTLs: the heart of freshness

The mistake most teams make is refreshing everything on one schedule — a nightly full re-crawl that is simultaneously too slow for prices and wasteful for reference docs. Different sources age at different rates, so give each a time-to-live that matches how fast its content actually moves. Live prices and stock might need a TTL of minutes; news, listings and forum threads an hour; rankings and catalogues a day; documentation, policies and reference material a week or more. Tag each chunk with its source's TTL, and a scheduler re-crawls only the sources whose clock has expired. The content hash is your efficiency valve: if the re-crawled page is byte-identical, skip the embedding step entirely and just reset the timestamp.

import time

TTL = {"prices": 300, "news": 3600, "catalog": 86400, "docs": 604800}

def needs_refresh(chunk, now=None):
    now = now or time.time()
    return (now - chunk["fetched_at"]) > chunk["ttl"]

# refresh only what has expired; re-embed only what actually changed
for src in due_sources(TTL):
    doc = fetch_markdown(src.url)
    if doc["content_hash"] != src.last_hash:
        reindex(to_chunks(doc))     # re-chunk + re-embed
    src.touch(doc["content_hash"])  # reset the clock either way

This is the loop that separates a demo from a production system. It keeps retrieval predictable, spends compute only where the world actually changed, and lets you make a real freshness guarantee per source. For the broader picture of grounding LLMs on live data, see our guide to feeding LLMs fresh web data, and for turning a single site into a citeable knowledge base, building a support-bot knowledge base.

Retrieve for coverage, not just similarity

One last upgrade at query time. Pure vector similarity misses exact terms — product codes, error strings, proper nouns — that a keyword search nails. Hybrid retrieval blends the two, then applies your recency filter so a time-sensitive query prefers fresh chunks. Log the retrieved chunks with every answer so you can audit what evidence the model actually used; that traceability is what makes RAG explainable, and it is how you catch a stale chunk before a user does. Techniques from LLM-powered extraction pair well here when you need structured fields out of the retrieved pages rather than prose.

Frequently asked questions

How do I keep a RAG pipeline's data fresh?

Assign each source a time-to-live that matches how fast its content changes, tag every chunk with a source URL and timestamp, and run a scheduler that re-crawls only expired sources. Use a content hash to skip re-embedding pages that did not actually change, and enable recency filtering at retrieval so time-sensitive queries prefer fresh chunks.

Why ingest markdown instead of HTML for RAG?

Raw HTML carries navigation, banners, footers and scripts that become noise chunks, polluting similarity search and wasting context tokens. Markdown-first ingestion extracts only the primary content, so chunks are clean and on-topic. A scraping API that returns rendered markdown removes the brittle custom parsing most RAG stacks rely on.

What chunk size should I use for RAG?

Roughly 300–500 tokens is the common sweet spot: small enough that a retrieved chunk stays tightly relevant, large enough to preserve a complete thought. Use a recursive splitter that respects sentence and heading boundaries, add a small overlap so context is not lost at the seams, and attach source metadata to every chunk.

Do I need proxies to build a RAG data pipeline?

If you crawl the open web at any scale, yes — sites rate-limit and block repeated requests from one IP, leaving gaps in your corpus. A scraping API or rotating proxies keep the crawl reliable so sources refresh completely and on schedule, which is exactly what per-source TTLs depend on.

RAG lives or dies on freshness. Ingest clean markdown, chunk small with metadata, and drive re-embedding from per-source TTLs so each source refreshes on its own clock. Build that loop once and your assistant answers from today's web, not last month's snapshot.

Build a fresh RAG pipeline on the Scraper API