# llms.txt Explained: The Spec, Real Adoption, and How to Publish

llms.txt promises to hand AI a clean map of your site. Here is what the spec really requires, an honest read on whether crawlers use it yet, and how to publish one.

**llms.txt** is a simple idea getting a lot of attention: a markdown file at the root of your site that hands large language models a clean, curated map of what you offer, instead of making them wrestle meaning out of ad-laden HTML. It is worth understanding precisely - because there is real substance to the spec, and real over-promising in the marketing around it. This guide covers what the format actually requires, an honest read on whether AI crawlers respect it today, and how to publish one.

## Why the file exists

The proposal, introduced by Answer.AI in September 2024, starts from a real constraint: LLM context windows are too small to swallow most websites whole, and converting complex HTML - navigation, ads, scripts - into clean text is lossy and imprecise. llms.txt sidesteps that by giving models concise, expert-level information in one predictable place. The key word is *inference*: this file is meant to help a model at the moment a user asks it something, not to feed training runs. It is a curated overview, hand-authored, not a dump of everything.

## The spec, precisely

The file lives at `/llms.txt` and is written in markdown, in a fixed section order so it can be parsed programmatically as well as read by a model:

- An **H1** with the project or site name. This is the only required section.
- A **blockquote** summary giving the key context needed to understand the rest.
- Zero or more markdown sections (paragraphs, lists - no headings) with further detail.
- Zero or more **H2** 'file lists': each item is a markdown link `[name](url)` followed by an optional `:` and a short note.
- A special **## Optional** section whose links can be skipped when a shorter context is needed.

There is a companion proposal: serve a clean markdown version of any page at the same URL with `.md` appended (and `index.html.md` for directory URLs), so a model can fetch the readable version directly. Here is a minimal, valid file:

```markdown
# QuantumProxies

> Residential, ISP, datacenter, mobile and IPv6 proxies plus SERP, Scraper and
> Extract APIs for web data collection at scale.

Stable API endpoints, per-request rotation, and 200+ country geo-targeting.

## Docs

- [Scraper API](https://quantumproxies.io/scraper-api): Markdown/JSON/HTML output, JS rendering on demand
- [SERP API](https://quantumproxies.io/serp-api): Google, Bing and DuckDuckGo plus verticals

## Guides

- [Feeding LLMs fresh web data](https://quantumproxies.io/blog/feeding-llms-fresh-web-data-2026): grounding and RAG patterns

## Optional

- [All blog posts](https://quantumproxies.io/blog): the full archive, skip for short context
```

![Three files at the root of one site: robots.txt states access rules for bots, sitemap.xml lists every indexable page for search engines, and llms.txt gives language models a curated, context-sized map; the card shows who reads each one, since llms.txt complements the other two rather than replacing them.](https://res.cloudinary.com/dmqliab0t/image/upload/f_auto,q_auto,w_1200/v1785335210/blog/generated/qp-b9-083-diag1.png)
*llms.txt complements robots.txt and sitemaps - it does not replace either. Different files, different jobs.*

## How it differs from robots.txt and sitemap.xml

It sits alongside the two files you already know, following the same root-path convention. `robots.txt` states access rules - who may crawl what - and we cover that in our [robots.txt guide](https://quantumproxies.io/blog/robots-txt-web-scraping-ethics). `sitemap.xml` lists every indexable human page for search engines. Neither substitutes for llms.txt: a sitemap rarely links the markdown versions of pages, never points to helpful external resources, and in aggregate is far too large to fit in a context window. llms.txt is the curated, LLM-sized overview those two were never designed to be.

## Do AI crawlers actually respect it? An honest answer

Here is where candour matters. llms.txt is a proposal, not an official standard, and adoption is uneven. As of 2025 no major AI crawler has publicly confirmed that it consumes llms.txt to rank or answer - Google has said plainly that it does not use the file, and Chrome's own documentation describes it as 'an emerging convention'. The honest position: publishing one is cheap and harmless, it can help tools that *do* ingest documentation (some developer-focused assistants and doc platforms use it), and it costs you nothing to future-proof. But do not expect an SEO or AI-traffic windfall. Anyone selling llms.txt as a ranking hack is overstating a proposal. Treat it as low-effort hygiene, not a growth lever.

## How to publish one

Write the file by hand for a small site - it is meant to be curated, and a thoughtful 30-line file beats an auto-generated 300-line one. For larger sites, plugins exist for VitePress, Docusaurus, Mintlify and Drupal that generate it from your content, and directories like llmstxt.site catalogue published files. The reference tooling, `llms_txt2ctx`, expands your file into a single context blob you can test against real models - it produces a lean version without the Optional links and a full version with them:

```bash
# Install the reference CLI and expand your file into an LLM context blob
pip install llms-txt

# Lean context (skips the ## Optional links)
llms_txt2ctx llms.txt > llms-ctx.txt

# Full context (includes Optional)
llms_txt2ctx --optional true llms.txt > llms-ctx-full.txt
```

The write-once trap is staleness. An llms.txt that points to renamed or deleted pages actively misleads. If yours is generated from docs, wire it into your build; if hand-authored, review it on the same cadence as your sitemap. Keeping the linked `.md` versions fresh is the same discipline behind any grounding pipeline - our guide to [RAG pipelines that don't go stale](https://quantumproxies.io/blog/fresh-web-data-rag-pipelines) applies directly.

![A minimal llms.txt file: an H1 with the site name, a blockquote summary, an H2 Docs list of annotated links and an Optional section; the card explains each part: the H1 is the only required section, the blockquote gives key context, file lists pair links with notes, and Optional links are skipped for short context.](https://res.cloudinary.com/dmqliab0t/image/upload/f_auto,q_auto,w_1200/v1785335211/blog/generated/qp-b9-083-diag2.png)
*The structure is fixed and small: a required H1, a summary blockquote, then H2 lists of annotated links.*

## Where clean web data comes in

llms.txt is about publishing clean context for your own site. The mirror problem - getting clean, LLM-ready markdown out of everyone else's sites for your own model or agent - is what our [web data for LLMs](https://quantumproxies.io/web-data-for-llms) service exists for. The Scraper API can return any page as markdown, which is exactly the shape both an llms.txt `.md` mirror and a RAG ingestion pipeline want. If you are building an assistant that needs to read the live web, that markdown-first output is the missing piece; our overview of [feeding LLMs fresh web data](https://quantumproxies.io/blog/feeding-llms-fresh-web-data-2026) ties it together.

[Get clean, LLM-ready web data](https://quantumproxies.io/web-data-for-llms)

## Frequently asked questions

### What is llms.txt?

It is a proposed markdown file placed at your site root (`/llms.txt`) that gives large language models a concise, curated overview of your content and links to detailed markdown versions of key pages. It is aimed at helping models at inference time - when a user asks a question - rather than at training, and requires only an H1 name as its single mandatory section.

### Do AI models actually use llms.txt?

Inconsistently. It is a proposal, not an adopted standard; as of 2025 no major AI crawler has confirmed using it to rank or answer, and Google has said it does not. Some developer-focused assistants and documentation tools do ingest it. Publishing one is cheap and future-proofs your site, but treat it as hygiene, not a proven traffic or ranking lever.

### How is llms.txt different from robots.txt?

They serve opposite intents. robots.txt sets access rules - which bots may crawl which paths - and is often used to keep crawlers out. llms.txt invites AI in and hands it a curated map of what to read. They follow the same root-path convention and coexist; llms.txt does not replace robots.txt or your XML sitemap.

### How do I create an llms.txt file?

For a small site, write it by hand: an H1 name, a blockquote summary, then H2 sections listing annotated markdown links, with a `## Optional` section for skippable extras. For larger sites, use a generator plugin for your platform. Test it by expanding it with the `llms_txt2ctx` CLI and asking a model questions about your content.

llms.txt is a small, sensible idea worth adopting for what it is - a curated context file - and worth being sceptical about for what it is sold as. Publish a clean one, keep it fresh, and pair it with genuinely LLM-ready web data, and you have covered both sides: what you expose, and what your models consume.

[Turn any page into LLM-ready markdown](https://quantumproxies.io/web-data-for-llms)