Build a Fine-Tuning Dataset From the Web: URLs to JSONL
A good fine-tune is 90% dataset. This is the pipeline from raw URLs to clean JSONL — sourcing at scale, dedup, license filters, and the chat format trainers actually expect.
A fine-tune is only as good as the data you feed it, and the hardest part of fine-tuning is almost never the training run — it's building a clean dataset. The web is the richest source of that data, but raw pages are messy, duplicated, unlabelled and legally uneven. This guide is the pipeline from URLs to training-ready JSONL: how much data you actually need, how to source it at scale, how to clean and deduplicate it, how to handle licensing, and how to format it in the chat structure trainers expect. It assumes you're building an instruction or chat fine-tune from public web content.
How much data do you actually need?
Start with realistic targets so you don't over- or under-build. The practical floor for a fine-tune that produces reasonable results is around 100 rows; for solid performance you generally want over 1,000. More usually helps, but only if it's clean — a thousand well-labelled examples beat ten thousand noisy ones. For scale reference, the classic Alpaca dataset was 52,000 instruction/output pairs, generated by prompting a stronger model. And if you're using a large model to generate synthetic examples, seed it with at least ten hand-written examples so it learns the exact structure and tone you want before it produces more.
Sourcing: crawl to clean markdown
The goal at the sourcing stage is clean text, not raw HTML. A fine-tuning row wants the substance of a page — the article, the docs, the Q&A — without the nav bars, cookie banners and ad markup that pollute a naive scrape. So crawl your target pages and convert them to markdown at ingestion, which strips boilerplate while keeping structure like headings and lists. At any real volume you'll hit rate limits and IP blocks, so route the crawl through a rotating pool. A Scraper API that returns markdown directly does both jobs — clean output and block resistance — in one call, which is why we cover it in our guide to feeding LLMs fresh web data.
# Sketch: crawl target URLs to clean markdown, ready for shaping into rows.
import requests
API = "https://api.quantumproxies.io/scrape" # returns markdown, handles rotation
def fetch_markdown(url):
r = requests.get(API, params={"url": url, "format": "markdown"},
headers={"Authorization": "Bearer YOUR_KEY"}, timeout=60)
return r.json()["markdown"]
docs = [fetch_markdown(u) for u in seed_urls] # clean text, no boilerplate
The JSONL format trainers expect
Most modern fine-tuning tooling defaults to a conversational JSONL format: one JSON object per line, each with a messages array of role/content pairs, where roles are system, user and assistant and the user/assistant turns alternate. That's the chat format Hugging Face and OpenAI-style trainers ingest. Older instruction datasets use a three-field shape (instruction, input, output) — fine for single-turn supervised tuning — and multi-turn conversation formats exist too, but they convert cleanly into the messages shape. Pick one target format and standardise everything to it.
// One training example per line (.jsonl). Chat / messages format:
{"messages":[
{"role":"system","content":"You classify support tickets by urgency."},
{"role":"user","content":"My payment failed three times and the event is tomorrow."},
{"role":"assistant","content":"high"}
]}
{"messages":[
{"role":"system","content":"You classify support tickets by urgency."},
{"role":"user","content":"How do I change my avatar?"},
{"role":"assistant","content":"low"}
]}Match the format to the training objective: raw text for continued pretraining, instruction-plus-output or multi-turn chat for supervised fine-tuning, ranked responses for preference-based methods. If you're building a reasoning fine-tune, keep the question and answer but rewrite the answer to include the chain-of-thought steps you want the model to learn.
Cleaning: where quality is won
This is the stage that decides your result, and it's mostly unglamorous filtering. Never assume web-sourced or model-generated rows are correct — inspect them. The recurring failure modes are specific and worth checking for by name:
- Label inconsistency: the same category written as "Positive" in one row and "positive" in another — normalise case and spelling.
- Off-target answers: the assistant returns a sentence or a follow-up question instead of the exact label or format you're training for.
- Mislabelled rows: an ambiguous input assigned the wrong output — the single most damaging kind of noise.
- Boilerplate residue: nav text, cookie notices or "read more" fragments that survived the markdown conversion.
- Duplicates: identical or near-identical examples that inflate row counts and bias the model toward repeated content.
A good habit is to keep both the pre-clean and cleaned JSONL so you can measure what you removed. And balance the set across categories — an over-represented class teaches the model to over-predict it.
Deduplication and licensing
Web data is full of repeats — syndicated articles, mirrored docs, boilerplate paragraphs — and duplicates quietly hurt a fine-tune by over-weighting whatever's repeated. Do two passes: exact-match dedup by hashing the normalised text to catch identical rows, then near-duplicate detection (a similarity or shingling technique like MinHash) to catch reworded copies that an exact hash misses. On licensing, be deliberate: not everything public is freely reusable. Track the source and license of each document, filter out content whose terms prohibit training use, and prefer sources with clear reuse permissions. It's far cheaper to record provenance while you crawl than to reconstruct it after you've mixed everything together.
Source clean web data for LLMs
Combine, card and split
When you're pulling from several sources, standardise them all to one format and merge into a single unified dataset before training — one clean fine-tune on the combined set beats sequentially fine-tuning on each source, which tends to erode what the model learned earlier. Then write a dataset card: a short record of where the data came from, how it was cleaned, its size and label distribution, its licensing, and any known limitations. It's what makes the dataset reproducible and auditable months later. Finally, hold out an evaluation split you never train on, so you can measure the fine-tune honestly instead of scoring it on data it memorised. If you're feeding a retrieval system rather than a fine-tune, our RAG pipeline guide covers the refresh side of the same problem.
Frequently asked questions
How many rows do I need to fine-tune an LLM?
Around 100 rows is the practical minimum for reasonable results, and over 1,000 is a good target for solid performance. More generally helps, but only if the data stays clean — a smaller, well-labelled set beats a large noisy one. For synthetic generation, seed the generator with at least ten hand-written examples so it learns your exact format first.
What format should fine-tuning data be in?
JSONL — one JSON object per line. Most modern trainers default to a conversational format with a messages array of system/user/assistant role-content pairs. Older instruction datasets use instruction/input/output fields. Pick one target format based on your training objective and standardise every source to it before training.
How do I clean a scraped dataset for fine-tuning?
Convert pages to markdown to strip boilerplate, then filter by hand for the common failures: inconsistent labels, off-target answers, mislabelled rows, and leftover nav or cookie text. Deduplicate with an exact hash and a near-duplicate pass, balance the categories, and keep both the raw and cleaned versions so you can measure what you removed.
Can I use any web content to train a model?
Not automatically — public does not mean freely reusable. Some content carries terms that prohibit training use, and personal data brings its own rules. Track each document's source and license as you collect it, filter out anything whose terms disallow training, and prefer clearly permissioned sources. This is general information, not legal advice; get counsel for commercial datasets.
Building a fine-tuning dataset from the web is a filtering pipeline: source clean markdown, shape it into the messages format your trainer expects, ruthlessly dedup and label-check, filter for license, and card the result. Get roughly a thousand clean rows and hold out an honest eval split, and you'll spend your training budget on signal instead of noise.