What is llms.txt, and does your site need one? The honest 2026 answer

What is llms.txt, exactly?

It is a plain-text proposal for a hard problem: large language models have context windows smaller than most websites, and when an AI tool fetches a page it receives raw HTML stuffed with navigation, cookie banners, scripts and ads that crowd out the content that matters (DerivateX, 2026). llms.txt answers that by placing a Markdown file at your domain root — example.com/llms.txt — that links to your most important pages with a one-line description of each, giving a model a clean, curated map of your site (Codersera, 2026).

Two facts about its origin set expectations correctly. It was proposed by Jeremy Howard of Answer.AI in September 2024, and it was explicitly framed as a proposal to help LLMs use a website at inference time — not as a tool to manipulate rankings (Greadme, 2026). And it is a community convention with no backing from any recognized standards body and no enforcement mechanism, so AI providers adopt it, or ignore it, entirely on their own terms (DerivateX, 2026). Everything below follows from those two facts.

Do AI crawlers actually use it?

The log data from 2026 is consistent and unkind: almost no relevant AI crawler requests the file. One 90-day study of more than 500 million AI-bot visits found only 408 requests aimed at llms.txt (Limy, 2026). A separate 90-day measurement found that out of tens of thousands of AI-bot requests, roughly 0.1% touched the file — performing worse than an average content page on the same domain (OtterlyAI, 2026). Ahrefs, looking at about 38,000 domains that had a valid llms.txt, found that 97% received zero requests for it in a single month (Ahrefs, 2026).

Adoption is just as thin. A study of 300,000 domains put adoption at roughly 10% — about one in ten sites after more than a year of discussion — and found it spread evenly across the web rather than concentrated among leaders, which is the opposite of how a winning best practice usually looks (SE Ranking, 2025). The crucial distinction underneath all of this: an occasional fetch by a bot like GPTBot is not the same as a model using the file to source or cite content. A restaurant having a menu does not mean it reads other restaurants’ menus before cooking.

Does it help my Google or AI search rankings?

No, not in any way that the evidence supports. Google has addressed this directly and twice: its team confirmed on the record in 2025 that it does not support llms.txt and is not planning to, with one engineer comparing it to the long-discredited keywords meta tag, and its May 2026 guide on generative-AI features repeats the point in a section literally about mythbusting (Limy, 2026; Ahrefs, 2026). No major provider — OpenAI, Google, Anthropic, Meta, Mistral — has committed to reading or acting on the file in production (DerivateX, 2026).

The statistical work points the same way. The 300,000-domain study found no correlation between having an llms.txt and being cited by LLMs, and reported that removing the variable from its predictive model actually improved accuracy — a sign the file was adding noise rather than signal (SE Ranking, 2025). If your goal is to be cited more in ChatGPT, Perplexity, Claude or Google’s AI surfaces, the file is not the lever. Anyone selling it as a shortcut to AI visibility is selling the menu, not the meal.

So where does llms.txt actually do real work?

In one specific lane: the agentic and developer-tools layer, which is also its original purpose. IDE coding agents — Cursor, Windsurf, Claude Code, GitHub Copilot, Cline, Aider — routinely fetch /llms.txt and /llms-full.txt when pointed at a documentation site, using the file as a structured entry point so the agent can pull only the relevant pages before writing code (Limy, 2026). This is exactly the use Jeremy Howard described, and it is why developer-first companies like Anthropic, Stripe, Mintlify and Cursor publish llms.txt on their documentation sites (Codersera, 2026).

There is also a forward-looking case worth stating fairly, because not every argument for the file is hype. Yoast’s founder, Joost de Valk, made the honest counter-point that web standards do not get adopted because bots start requesting something that does not yet exist — they get adopted because publishers start serving a thing, which gives crawlers a reason to look, which gives more publishers a reason to serve it (Medium / Santhosh J, 2026). On that view, shipping one now is a cheap option on a future that may or may not arrive. Google’s own Lighthouse added an agentic-browsing audit that checks for the file, which is a small straw in that wind (Greadme, 2026).

What actually drives AI citations instead?

The same things that have always driven visibility, captured during normal crawling without any special file. The sites that show up most often in ChatGPT, Perplexity and Claude answers are the ones with genuine authority on a topic, consistent mentions across high-quality external sources, content that answers questions directly, and clean entity signals that let a model form a reliable picture of what you do (DerivateX, 2026). One analysis found high-E-E-A-T content cited roughly 340% more often than generic information, regardless of any access file (AEO Engine, 2026). This is the substance of our pillar guide on whether your site is readable by AI, and none of it depends on llms.txt.

For the part of the job that is about access rather than content, the working standard is the one from 1994: robots.txt. Every relevant AI crawler — GPTBot, ClaudeBot, Google-Extended, PerplexityBot — respects robots.txt user-agent rules, and Anthropic’s own crawler documentation points to robots.txt, not llms.txt (Medium / Spriestersbach, 2026). If you want to allow AI systems to cite you while disallowing training scrapes, that control lives in robots.txt today — it is the file that actually changes outcomes.

Are there risks to shipping one?

A few, all manageable, and worth knowing before you commit. The most common mistake is the implementation that generates an indexable Markdown copy of every page on the site: at scale, those duplicates dilute crawl budget and can suppress rankings for the original pages (DerivateX, 2026). There is also a small disclosure cost — a tidy map of your best content makes life marginally easier for competitors who want to scrape you (Ahrefs, 2026).

Both are avoidable. Keep the file a single, hand-curated list rather than an auto-generated mirror of the whole site, and follow the suggestion from Google’s John Mueller to noindex the llms.txt itself so it stays available to agents that request it without being indexed as an ordinary page (Greadme, 2026). Done that way, the downside is close to nothing — which is the whole reason the honest verdict lands where it does.

What does a good llms.txt look like?

If you do ship one, the correct version is small and hand-made. The format the proposal describes is simple Markdown: a heading with your site or project name, an optional one-line summary of what you do, and then sections of links — each a page title linked to its URL, followed by a short description of what the reader will find there (Codersera, 2026). The goal is a curated table of contents to your best material, not a dump of every URL on the site.

A few principles keep it useful. List only the pages you would genuinely want an AI to read and cite — documentation, key guides, product or pricing pages — and write the descriptions specifically enough that an agent can tell which link answers which question. Keep it short, because a focused file an agent can scan beats an exhaustive one it has to wade through. And maintain it by hand alongside your sitemap so it does not drift out of date. The discipline that makes the file useful to an agent — clarity, specificity, curation — is the same discipline that makes your actual pages citable, which is the more important work.

So should you create one?

The balanced answer is: it is fine to ship one, as long as you ship it for the right reasons and with the right expectations. The cost is genuinely near zero — a short Markdown file you can write in a few minutes — and the confirmed payoff is also near zero for AI search, so this is a low-stakes decision either way (Greadme, 2026). Ship one if you have developer documentation, if you want to be early on agent-driven traffic, or if you value the clean inventory of your best content that writing it forces you to produce. Skip it, or deprioritize it, if you were hoping it would lift your AI citations — that hope is not supported by the data.

What you should not do is let it displace the work that matters. The file has shown up in too many 2026 audit decks as its own line item, presented as a lever it is not (Medium / Spriestersbach, 2026). An hour spent on original, experience-led content and clean technical health does more for your AI visibility than an hour spent perfecting a file almost no model reads.

Why we ship an llms.txt anyway

For transparency, our own site serves an llms.txt — and the reasons are exactly the honest ones above, not a ranking trick. It costs us almost nothing to maintain; it gives us optionality if the agentic web does standardize on it; and writing it forced a clean, deliberate inventory of the content we would most want an AI to find. We do not count it as an AI-visibility tactic, we do not generate per-page Markdown duplicates, and we noindex the file itself. If the data changes and major answer engines begin using it, we are ready; if it never does, we have lost nothing.

That posture is the point of this whole guide. The honest position on a contested tactic is to state plainly what the evidence shows, adopt the cheap and harmless version where it has narrow value, and refuse to sell it as more than it is. For the work that genuinely earns AI citations — being readable, authoritative and worth quoting — see the pillar this guide supports, on whether your site is readable by AI.