llms.txt Explained: The Spec, Real Adoption, and How to Publish
llms.txt promises to hand AI a clean map of your site. Here is what the spec really requires, an honest read on whether crawlers use it yet, and how to publish one.
llms.txt is a simple idea getting a lot of attention: a markdown file at the root of your site that hands large language models a clean, curated map of what you offer, instead of making them wrestle meaning out of ad-laden HTML. It is worth understanding precisely - because there is real substance to the spec, and real over-promising in the marketing around it. This guide covers what the format actually requires, an honest read on whether AI crawlers respect it today, and how to publish one.
Why the file exists
The proposal, introduced by Answer.AI in September 2024, starts from a real constraint: LLM context windows are too small to swallow most websites whole, and converting complex HTML - navigation, ads, scripts - into clean text is lossy and imprecise. llms.txt sidesteps that by giving models concise, expert-level information in one predictable place. The key word is inference: this file is meant to help a model at the moment a user asks it something, not to feed training runs. It is a curated overview, hand-authored, not a dump of everything.
The spec, precisely
The file lives at /llms.txt and is written in markdown, in a fixed section order so it can be parsed programmatically as well as read by a model:
- An H1 with the project or site name. This is the only required section.
- A blockquote summary giving the key context needed to understand the rest.
- Zero or more markdown sections (paragraphs, lists - no headings) with further detail.
- Zero or more H2 'file lists': each item is a markdown link
[name](url)followed by an optional:and a short note. - A special ## Optional section whose links can be skipped when a shorter context is needed.
There is a companion proposal: serve a clean markdown version of any page at the same URL with .md appended (and index.html.md for directory URLs), so a model can fetch the readable version directly. Here is a minimal, valid file:
# QuantumProxies
> Residential, ISP, datacenter, mobile and IPv6 proxies plus SERP, Scraper and
> Extract APIs for web data collection at scale.
Stable API endpoints, per-request rotation, and 200+ country geo-targeting.
## Docs
- [Scraper API](https://quantumproxies.io/scraper-api): Markdown/JSON/HTML output, JS rendering on demand
- [SERP API](https://quantumproxies.io/serp-api): Google, Bing and DuckDuckGo plus verticals
## Guides
- [Feeding LLMs fresh web data](https://quantumproxies.io/blog/feeding-llms-fresh-web-data-2026): grounding and RAG patterns
## Optional
- [All blog posts](https://quantumproxies.io/blog): the full archive, skip for short context
How it differs from robots.txt and sitemap.xml
It sits alongside the two files you already know, following the same root-path convention. robots.txt states access rules - who may crawl what - and we cover that in our robots.txt guide. sitemap.xml lists every indexable human page for search engines. Neither substitutes for llms.txt: a sitemap rarely links the markdown versions of pages, never points to helpful external resources, and in aggregate is far too large to fit in a context window. llms.txt is the curated, LLM-sized overview those two were never designed to be.
Do AI crawlers actually respect it? An honest answer
Here is where candour matters. llms.txt is a proposal, not an official standard, and adoption is uneven. As of 2025 no major AI crawler has publicly confirmed that it consumes llms.txt to rank or answer - Google has said plainly that it does not use the file, and Chrome's own documentation describes it as 'an emerging convention'. The honest position: publishing one is cheap and harmless, it can help tools that do ingest documentation (some developer-focused assistants and doc platforms use it), and it costs you nothing to future-proof. But do not expect an SEO or AI-traffic windfall. Anyone selling llms.txt as a ranking hack is overstating a proposal. Treat it as low-effort hygiene, not a growth lever.
How to publish one
Write the file by hand for a small site - it is meant to be curated, and a thoughtful 30-line file beats an auto-generated 300-line one. For larger sites, plugins exist for VitePress, Docusaurus, Mintlify and Drupal that generate it from your content, and directories like llmstxt.site catalogue published files. The reference tooling, llms_txt2ctx, expands your file into a single context blob you can test against real models - it produces a lean version without the Optional links and a full version with them:
# Install the reference CLI and expand your file into an LLM context blob
pip install llms-txt
# Lean context (skips the ## Optional links)
llms_txt2ctx llms.txt > llms-ctx.txt
# Full context (includes Optional)
llms_txt2ctx --optional true llms.txt > llms-ctx-full.txtThe write-once trap is staleness. An llms.txt that points to renamed or deleted pages actively misleads. If yours is generated from docs, wire it into your build; if hand-authored, review it on the same cadence as your sitemap. Keeping the linked .md versions fresh is the same discipline behind any grounding pipeline - our guide to RAG pipelines that don't go stale applies directly.
Where clean web data comes in
llms.txt is about publishing clean context for your own site. The mirror problem - getting clean, LLM-ready markdown out of everyone else's sites for your own model or agent - is what our web data for LLMs service exists for. The Scraper API can return any page as markdown, which is exactly the shape both an llms.txt .md mirror and a RAG ingestion pipeline want. If you are building an assistant that needs to read the live web, that markdown-first output is the missing piece; our overview of feeding LLMs fresh web data ties it together.
Frequently asked questions
What is llms.txt?
It is a proposed markdown file placed at your site root (/llms.txt) that gives large language models a concise, curated overview of your content and links to detailed markdown versions of key pages. It is aimed at helping models at inference time - when a user asks a question - rather than at training, and requires only an H1 name as its single mandatory section.
Do AI models actually use llms.txt?
Inconsistently. It is a proposal, not an adopted standard; as of 2025 no major AI crawler has confirmed using it to rank or answer, and Google has said it does not. Some developer-focused assistants and documentation tools do ingest it. Publishing one is cheap and future-proofs your site, but treat it as hygiene, not a proven traffic or ranking lever.
How is llms.txt different from robots.txt?
They serve opposite intents. robots.txt sets access rules - which bots may crawl which paths - and is often used to keep crawlers out. llms.txt invites AI in and hands it a curated map of what to read. They follow the same root-path convention and coexist; llms.txt does not replace robots.txt or your XML sitemap.
How do I create an llms.txt file?
For a small site, write it by hand: an H1 name, a blockquote summary, then H2 sections listing annotated markdown links, with a ## Optional section for skippable extras. For larger sites, use a generator plugin for your platform. Test it by expanding it with the llms_txt2ctx CLI and asking a model questions about your content.
llms.txt is a small, sensible idea worth adopting for what it is - a curated context file - and worth being sceptical about for what it is sold as. Publish a clean one, keep it fresh, and pair it with genuinely LLM-ready web data, and you have covered both sides: what you expose, and what your models consume.