llms.txt
llms.txt is a proposed convention for a markdown file at the root of a website that lists its most important pages with short descriptions, so a language model can find the relevant material without parsing navigation, scripts and layout. It was proposed by Jeremy Howard in 2024.
What the file looks like
It lives at /llms.txt and is markdown, not a directive format. The convention is an h1 with the site name, a short blockquote summary, then lists of links with a one-line description of each:
# NorthernGo
> An AI app builder that can generate applications locally in the browser.
## Guides
- [Connect Supabase](https://northerngo.com/resources/connect-supabase/): Add a database to a generated app.
- [Local AI with WebGPU](https://northerngo.com/resources/webgpu-react-local-ai/): How browser-based generation works.A companion /llms-full.txt holds the full text of that documentation in one file, for cases where a model should read everything rather than navigate.
The reasoning is straightforward. HTML is a rendering format; a model retrieving a page pays context window budget for markup, menus and cookie banners before reaching the sentence it needs. Markdown removes that overhead, and a curated list is more useful than a sitemap because it says what each page is for.
Who it is for
The clearest fit is documentation, APIs and reference material — sites where an assistant helping a developer benefits from being pointed at the right page. Several developer-tool companies publish one, and a few AI coding tools will read it if you point them at it.
What to watch out for
It is not robots.txt, and it does not control anything. This is the most common confusion. robots.txt is a long-established convention that crawlers respect for access; llms.txt grants no permission and forbids nothing. Adding it does not stop AI training on your content, and removing it does not either. If access control is what you want, that is robots.txt, a firewall, or a licence agreement.
The second thing to be honest about: adoption is unproven. As of now no major AI provider has publicly committed to fetching or using the file as part of normal retrieval, and the search engines have not endorsed it. Publishing one is cheap and plausibly useful; treating it as a ranking or citation lever is not supported by evidence.
If you do publish one, keep it accurate. A file listing pages that have moved is worse than no file, and it is exactly the kind of artifact that gets written once and never updated. Genuine visibility in AI answers comes from the work described under generative engine optimization, not from a manifest.
Frequently asked questions
Does llms.txt stop AI companies from training on my site?
No. It is a helpful index, not a permission file, and it has no restrictive effect whatsoever. To limit crawling you need robots.txt directives for the specific crawlers, and for anything you need enforced rather than requested, authentication or a firewall rule is the only reliable mechanism.
Is it worth publishing an llms.txt file?
If you have documentation and can generate the file automatically from content you already maintain, yes — the cost is close to zero and the downside is only a stale file. If it would be handwritten and updated by hand, the effort is better spent making the pages themselves clearer and more quotable.
Build it yourself
NorthernGo turns a plain-text description into a working web app with a database, login and a live URL. Local AI generation runs on your own GPU, is unlimited, and is free on every plan.