How to Collect Training Data for Machine Learning From the Web
The model is the easy part. Sourcing a clean, deduplicated, legally sane and refreshable training set from the open web is where projects actually stall. Here's the collection pipeline, end to end.
Training a model is often the easy part. The step that stalls machine-learning projects is upstream: assembling a clean, deduplicated, legally sane and refreshable training set from the open web. Getting raw pages is cheap — a crawler does that — but turning them into usable rows is the real work, and it's what this guide covers: sourcing, cleaning to structured text, deduplication, sampling, licensing, and the refresh pipeline that keeps a model from going stale.
Start with the signal, not the scraper
Before writing any collection code, define exactly what the model needs to learn and which sources carry that signal. Data comes in three shapes and each has different collection costs: structured (tables, product feeds — easy to parse), semi-structured (JSON, CSV, XML endpoints — often the cleanest thing a site exposes), and unstructured (article text, reviews, forum threads). Unstructured content is 80-90% of all data and reportedly only about 0.5% of it ever gets used, which is exactly the gap a well-built web dataset closes. Decide which shape you're after before you scale anything.
Crawl to clean text, not raw HTML
Feeding raw HTML into a training pipeline means feeding it navigation, ads, cookie banners and script tags — noise that degrades the dataset. The durable move is to crawl straight to clean, structured text. A Scraper API that returns markdown does the boilerplate stripping for you, so a crawl of thousands of pages lands as readable content instead of tag soup:
import requests
def fetch_clean(url):
r = requests.get(
"https://api.quantumproxies.io/scrape",
params={"url": url, "format": "markdown"},
headers={"Authorization": "Bearer YOUR_API_KEY"},
timeout=60,
)
return r.json()["content"] # boilerplate-stripped markdown
corpus = [fetch_clean(u) for u in seed_urls]Markdown is a good target because it keeps structure (headings, lists, tables) while dropping the presentation layer — the same reason it's the preferred input for RAG pipelines and LLM training alike. Route the crawl through rotating residential IPs so a large collection run doesn't get the whole job blocked halfway through.
Deduplicate before it poisons the model
Near-duplicate documents are the silent killer of web datasets — the same article syndicated across ten sites, boilerplate footers, reposted content. Train on them and you over-weight whatever repeats. Hash each document's normalised content and drop collisions; for near-duplicates, a shingle or MinHash approach catches the ones an exact hash misses:
import hashlib
def content_hash(text):
norm = " ".join(text.lower().split()) # normalise whitespace/case
return hashlib.sha256(norm.encode()).hexdigest()
seen, unique = set(), []
for doc in corpus:
h = content_hash(doc)
if h not in seen:
seen.add(h)
unique.append(doc)
print(f"kept {len(unique)} of {len(corpus)} after dedup")The content hash you compute here does double duty — it's also how you detect change on the next crawl, so keep it.
Sample deliberately, license honestly
More data isn't automatically better data. A representative sample beats a lopsided pile — if 90% of your crawl is one site or one language, the model learns that skew. Sample stratified across your sources and classes so the set reflects the distribution you actually want to model. On licensing: web data isn't automatically free to train on. Filter by source terms, honour robots directives where they apply, and keep provenance (the source URL and fetch date) alongside every row so you can prove where each example came from. Our note on scraping legality in 2026 covers the personal-data and licensing lines (informative, not legal advice).
Ground truth: the part the web won't give you
For supervised learning you need labels, and the open web rarely hands them over. Some datasets are inherently labelled (a rating alongside a review, the next word in a sentence), which is why those are cheap to collect at scale. Everything else needs a ground-truth step: labelling by domain experts, or crowdsourcing through a marketplace. If you crowdsource, guard quality by seeding known-answer tasks and rejecting workers who fail them — that's a well-studied failure mode. Where possible, prefer naturally-labelled web data; it's the difference between a dataset you can build in a week and one that needs a labelling budget.
Refresh, because data goes stale
Collection is never one-and-done. Prices change, pages get rewritten, new content appears — a model trained on a frozen snapshot drifts away from the world it's meant to predict. Build the refresh in from day one: re-crawl on a schedule, compare each page's content hash to last run, and only re-process what actually changed. That keeps refresh cost proportional to real change rather than re-downloading everything:
def refresh(url, last_hash):
text = fetch_clean(url)
h = content_hash(text)
if h == last_hash:
return None # unchanged, skip re-processing
return {"url": url, "text": text, "hash": h, "fetched": now()}For fast-moving domains, or when you'd rather consume a managed feed than run the crawl yourself, web data built for LLMs gives you clean, refreshed content without maintaining the pipeline.
Crawl the web to clean markdown at scale
Frequently asked questions
Where do you get training data for machine learning?
Three main sources: existing public datasets (Kaggle, Google Dataset Search, academic corpora), internal first-party data, and the open web via crawling. The web is the largest and freshest source but needs the most work — cleaning, deduplication, sampling and licensing. For many modern models, a crawl of relevant sites to clean markdown is the fastest way to a domain-specific dataset that doesn't already exist.
How much training data do I need?
It depends on the task and model, and the honest answer is: start small and scale until performance plateaus. Train on a representative sample, measure accuracy on a held-out test set (a 70/30 split is a common starting point), then add more data and watch whether the metric keeps improving. Quality and representativeness usually matter more than raw volume.
Is it legal to scrape web data for ML training?
Collecting public data is broadly defensible, but training use raises licensing and, where personal data is involved, privacy questions. Filter by source terms, avoid personal data you don't need, keep provenance for every example, and respect robots directives where they apply. This is general information, not legal advice — get counsel for a commercial dataset at scale.
How do I keep a training dataset fresh?
Schedule re-crawls and use a content hash to detect change, re-processing only pages that actually moved. Version your dataset so you can reproduce which snapshot trained which model, and keep fetch dates on every row. Incremental, change-detected refresh keeps cost proportional to real change instead of re-downloading the entire corpus each cycle.
The model gets the headlines, but the dataset decides the outcome. Source the right signal, crawl to clean markdown, dedup hard, sample honestly, keep provenance, and build refresh in from the start. Get the collection pipeline right and everything downstream gets easier. If you're heading toward fine-tuning specifically, our guide on building a fine-tuning dataset from the web picks up where this leaves off.