What Is llms.txt and Why It Matters for AI Visibility

A client forwarded me a note from his web team last quarter. A scanning tool had flagged his site for a “missing llms.txt file” and ranked it as an issue, right alongside broken links and slow pages. He wanted to know whether his reputation was at risk over a text file he had never heard of. The short answer was no. The longer answer is worth your time, because llms.txt sits at the center of a genuine debate about how brands surface inside AI assistants, and most of what gets written about it lands in one of two ditches: breathless overselling, or flat dismissal. Neither one helps you decide anything.

Here is the honest version.

What llms.txt actually is

llms.txt is a proposal. Jeremy Howard of Answer.AI published it in September 2024, and the idea is to standardize on a single file at the root of your website, at /llms.txt, that hands large language models a clean, curated guide to your most important content. It is written in Markdown, the plain-text format AI systems read most reliably.

The reasoning behind it holds up. AI assistants work inside a context window, a fixed budget of text they can hold at one time. Most websites are far too large to fit, and a typical page buries its substance under navigation, ads, banners, and scripts that a model has to wade through before it reaches anything useful. The proposal’s own framing is blunt about the problem: context windows are “too small to handle most websites in their entirety.” Rather than make the model guess what matters, llms.txt lets the site owner hand over a short map. A title, a one-line summary, and a curated set of links to the pages that count.

That is the whole idea. You are not changing what your site says. You are telling an AI where the good material lives, so it does not have to reconstruct your site from scratch.

How it differs from robots.txt and sitemaps

People reach for two familiar comparisons, and both teach you something.

robots.txt

robots.txt is a gatekeeper. It tells automated crawlers what they may and may not access. Its whole job is permission and exclusion. llms.txt takes the opposite posture. It is an invitation, a recommendation of what an AI should read when someone asks about you. One closes doors selectively. The other points at the rooms worth entering. The proposal treats the two as complementary, not competing.

Sitemaps

A sitemap (sitemap.xml) is an exhaustive inventory of every indexable page, built for search engines. llms.txt is the opposite of exhaustive. It is editorial. The proposal notes that a sitemap usually lists far more than fits in a model’s context, carries no external links, and rarely points to clean, machine-readable versions of pages. A sitemap says, here is everything. llms.txt says, here is what matters, and here is the short version.

That editorial quality is the real point. A sitemap is generated. An llms.txt is decided. Someone has to choose what represents the brand.

The format itself is deliberately plain. The proposal calls for a single H1 with the site name, an optional one-line summary set off as a blockquote, optional notes, then sections of links, each a plain Markdown hyperlink with a short description. It also suggests offering clean Markdown versions of key pages at the same address with .md appended, so a model can pull the substance of a page without the surrounding clutter. None of this needs special tooling, and that low barrier explains both why so many sites added one quickly and why its real-world value is still unproven.

The part nobody should skip: adoption is not there yet

Here is where I have to be straight with you, because the distance between the marketing and the reality is wide.

As of early 2026, no major AI provider has publicly committed to reading llms.txt files. Not OpenAI, not Google, not Anthropic, not Perplexity. Several of them publish their own llms.txt files, which is an entirely different thing from agreeing to consume yours.

Google has been the most direct about it, twice. In mid-2025, Google’s John Mueller observed that none of the AI services had said they use llms.txt, and pointed out that you can confirm this in your own server logs, where the crawlers do not even check for the file. He likened it to the old keywords meta tag, the discredited tag where a site owner simply declared what a page was about, which search engines learned to ignore because it was trivial to game. His worry was specific. An AI that trusted an llms.txt would still have to verify the real content anyway, to guard against a site showing one story to the model and another to everyone else. Google’s Gary Illyes went further the same year, stating plainly that Google does not support the file and has no plans to.

Independent analyses landed in the same place. Ahrefs, reviewing the evidence, found no major provider supporting the file. Server-log studies through 2025 showed AI crawlers requesting llms.txt at vanishingly low rates, a rounding error against their total traffic.

And yet sites kept adding it. By late 2025, BuiltWith’s crawl detected the file on more than 800,000 sites, a count that sweeps up everything from a one-line stub to a carefully built guide, and includes names like Anthropic, Cloudflare, and Stripe. Strip out the stubs and the verified, substantive implementations number in the hundreds. Either way, the shape of the situation is the same and a little strange. Plenty of supply. Almost no confirmed demand.

That does not make llms.txt a scam. It makes it a bet on the future. Howard was careful to frame the file as something for inference, the moment a user asks an assistant about you, not for training. The infrastructure exists. The agreement to use it does not, yet.

What it practically means for your brand

So should you care? For the clients I advise, my answer is measured, and it turns on who you are.

If you run documentation, a developer platform, or a content library that AI tools already pull into coding environments and research workflows, an llms.txt earns its keep today. That was the original use case, and there it works. Some IDE assistants and research tools genuinely read these files.

If you are an executive, a public figure, or a brand whose real concern is what an assistant says when someone types your name, the math changes. An llms.txt file will not, on its own, alter your answer in ChatGPT tomorrow. The systems that generate that answer are not reading it. Selling the file as a reputation lever right now means selling a tactic the platforms have not agreed to honor.

What does move the needle is the discipline that producing a good llms.txt forces on you. You have to decide, deliberately, which content best represents you. You have to make sure it is clean, accurate, and structured so a machine can parse it. You have to strip out the noise that muddies the picture. That work pays off across every channel an AI does read today, which is the actual substance of Search and AI Visibility. The file is cheap and harmless to publish as a forward-looking hedge. Just publish it with clear eyes about what it does and does not do.

This is the same principle I come back to with clients weighing the shift from search to AI answers, which I get into in GEO vs SEO. The goal is not to chase every proposed standard. It is to own the substance that every system, present and future, has to draw from. The file is a pointer. Your reputation is the thing it points at.

llms.txt is a hedge, not a lever. The brands that win in AI answers are not the ones with the tidiest text file.

They are the ones whose underlying reputation is strong, consistent, and well-documented enough that any assistant, reading anything, arrives at the right conclusion about them. If you want to know where you actually stand, start by learning what ChatGPT says about you. That answer matters far more than whether a file sits at your root today.

Sources

• The /llms.txt file (official proposal)
• Jeremy Howard, Answer.AI: the original llms.txt announcement
• Ahrefs: What Is llms.txt, and Should You Care About It?
• Search Engine Land: llms.txt, a proposed standard for AI website content crawling
• PPC Land: llms.txt adoption stalls as major AI platforms ignore proposed standard
• Search Engine Journal: Google Says llms.txt Comparable To Keywords Meta Tag
• Search Engine Land: Google says normal SEO works for AI Overviews and llms.txt won’t be used