Reference guide

llms.txt: the complete guide

Format, implementation and limits: what the convention proposes, what Google documents and what remains unsupported by public evidence at other operators.

Updated 12 August 2026 16 min read

Origin and context: why llms.txt was created

In September 2024, Jeremy Howard, co-founder of Answer.AI and creator of the fast.ai framework, published a convention proposal called llms.txt. The starting point was simple: LLMs read HTML pages designed for humans. Navigation, menus, banners, scripts, all this structural noise must be filtered before useful information can be extracted. This process introduces loss and imprecision.

llms.txt proposes a response to this problem: give a compatible tool a clean Markdown file that says "here is what I am, here are my most important pages, here is how to describe them". It is a documentation format for automated retrieval, not a demonstrated ranking signal.

The convention draws inspiration from robots.txt: a text file at the domain root, readable without authentication, with minimal syntax. It does not replace robots.txt (which manages crawl permissions) or sitemap.xml (which lists URLs for indexing), but complements them with a semantic layer oriented towards LLMs.

If you are looking for a tool to generate your llms.txt automatically, the sister site llmtxt.info offers a free online generator based on the official specification.

The specification: what llms.txt contains

An llms.txt file is a Markdown file located at https://yourdomain.com/llms.txt. It follows a three-zone structure:

Zone 1: header with entity description

The file opens with a description of the entity: what the site is, who its audience is and what its value proposition is. Keep it concise, factual and consistent with the linked pages. An explicitly compatible tool may use it as initial context, but that use must not be assumed for public search engines.

Zone 2: thematic sections with annotated links

The body of the file lists your key pages, organised by theme, with a description of each page. The syntax is standard Markdown: H2 headings for sections, Markdown links for pages with 1 to 3 lines of description.

Zone 3: optional section for contextual exclusions

An ## Optional section can list URLs to de-prioritise for LLM context: legal pages, transactional pages without informational value. This is not a crawl exclusion directive (do not confuse with robots.txt), it is a contextual relevance signal.

llms-full.txt: the full-content variant

The convention provides for a second file, /llms-full.txt, containing the full text of selected pages in Markdown. It can serve as a corpus for an agent, analysis tool or RAG system configured to load it.

Documented adoption state in 2026

The presence of a file on a domain measures editorial adoption, not use by a search engine. For decisions, use each operator's documented status as of 12 August 2026:

Operator or use case Public status Operational conclusion
Google Search Explicitly ignored No positive or negative effect on Google visibility and rankings
OpenAI Ingestion not documented on its official crawler page Do not attribute a crawl or citation to llms.txt without system-specific evidence
Perplexity Ingestion not documented on its official crawler page Treat the file as experimental, not as a confirmed search lever
Anthropic Ingestion not documented in its official bot policy Keep bot policy separate from any assumption about llms.txt
Internal agent, RAG or analysis tool Verifiable through configuration and application logs Relevant when the consumer is explicitly configured to read the file

Practical conclusion: publish llms.txt when you have an identified consumer or want a reusable Markdown index. Do not fund it at the expense of indexability, original content, internal linking, Merchant Center or applicable structured data.

Implementation: step-by-step guide

Step 1: write llms.txt

Recommended minimal structure:

  1. H1 title = site name
  2. Blockquote (>) = 2 to 4 sentence description (who you are, what you do, for whom)
  3. Section ## Main pages with your 5 to 15 most important pages
  4. Section ## Recent articles if you have an editorial section
  5. Section ## Optional to signal pages with no contextual value

To generate your file automatically, llmtxt.info provides a free online tool that analyses your site and produces a ready-to-use llms.txt structure.

Step 2: publish at the domain root

The file must be accessible at https://yourdomain.com/llms.txt, not in a subdirectory, not behind authentication. The server must return Content-Type text/plain with HTTP 200.

Step 3: verify and monitor

After publishing, verify accessibility with curl -I https://yourdomain.com/llms.txt. You should see a 200 and the correct Content-Type. Then monitor your server logs for AI crawler requests.

Best practices and pitfalls to avoid

Do not duplicate robots.txt content in llms.txt. They are two files with distinct functions. llms.txt is not an access control mechanism.

Avoid generic descriptions. "Our site homepage" adds no contextual value. Each description should contain precise and informative terms.

Update regularly. An llms.txt that omits recent content becomes misleading documentation for tools and people who consult it. Automate coverage checks without removing editorial curation.

Frequently asked questions

Is llms.txt an official standard recognised by Google or OpenAI?
No. llms.txt is a convention proposed by Jeremy Howard (Answer.AI) in September 2024. Google explicitly says Search ignores the file. The public crawler documentation from OpenAI, Perplexity and Anthropic does not document its ingestion as a search signal.
Should I choose between llms.txt and llms-full.txt, or do I need both?
No. llms.txt is a curated index while llms-full.txt aggregates full content. Their usefulness depends on a system explicitly configured to read them. No official documentation supports the claim that publishing both automatically improves visibility in an AI search engine.
What is the impact of llms.txt on classic Google SEO?
Google Search says it ignores llms.txt: the file neither helps nor harms visibility or rankings in Google Search, including its generative features. It does not replace robots.txt, sitemap.xml or indexable HTML pages.
How do I know if an LLM has read my llms.txt?
Server logs can prove that an HTTP client requested /llms.txt. They do not prove use, indexing or citation. For an agent or RAG system you control, instrument the file load directly and retain the HTTP response.
Is there a tool to generate llms.txt automatically?
Yes. llmtxt.info is a free online generator that produces a structured llms.txt file from your site URL. It is based on the official Answer.AI specification. For static sites like Astro or Next.js, programmatic approaches also allow generating the file automatically at build time.

Official sources and specification

À lire ensuite

Technical optimisation robots.txt, HTTP headers, schema.org, the complete AI bot configuration. Content structure for LLMs Self-contained format, chunking, citation-friendliness. GEO: Generative Engine Optimization The definitive pillar on how GEO works and how to implement it.