llms.txt: What It Is, Who Actually Reads It, and What the Logs Say

llms.txt is a Markdown file served from a website's root directory (yoursite.com/llms.txt). It gives large language models a curated index of the site's most important pages. Answer.AI co-founder Jeremy Howard proposed it on September 3, 2024. It remains a proposal, not an adopted standard: no major AI company has confirmed that its crawlers fetch the file on their own.

Twenty-one of the 89 results Google served for "llms.txt" on July 9, 2026 carry doubt in the title: "should you care about it?", "does it actually do anything?", "hype or real?", "stop trying to make llms.txt happen". That is 24% of a results page interrogating its own subject. robots.txt never had to prove it "actually works" on its own SERP. Another 7 results are generator tools. 49 are what-is explainers. The spec's own site sits at #2. The demand is real. The skepticism is real. Most explainers resolve the tension by ignoring one side of it.

This page keeps both sides. It documents the spec, then lays out the evidence with dates. One side: Google's public dismissals and an 80,000-blog server-log null result. The other: the AI companies' own live llms.txt files, Chrome's new Lighthouse audit, and working agent workflows. It ends with a verdict on when the file is worth 30 minutes and when it's cargo cult.

What is llms.txt?

llms.txt is a plain Markdown file that tells AI systems which of your pages matter and where clean, machine-readable versions live. The community shorthand is "robots.txt but for LLMs". The shorthand gets the location right and the function backwards. robots.txt restricts crawlers. llms.txt offers them a reading list and restricts nothing.

The shorthand is worth documenting because it travels everywhere the file is explained. A practitioner on r/GenEngineOptimization put it this way on June 23, 2026: "llms.txt is basically a plain text file at yoursite.com/llms.txt that tells AI models who you are, what you do, how to cite you. Think robots.txt but for LLMs." The same framing appears in SERP titles ("The New Robots.txt for AI") and HN comments. We logged it at least three times, independently, across our July 2026 community corpus. Mintlify's docs pick a more accurate comparison. The file "helps LLMs index content more efficiently, similar to how a sitemap helps search engines." Closer. But a sitemap lists everything. This file is selective by design, with prose notes a sitemap doesn't carry. The honest one-liner: a curated Markdown index. It is closer to a well-edited README for your whole site than to either file it gets compared to.

The problem the file was written to solve is context windows, not crawler politics. An LLM answering a question at inference time can't read your whole site. What it can read arrives as HTML noise: navigation, scripts, ads, cookie banners. llmstxt.org proposes a single, predictable address. There, the site hands over its own summary in the format models parse best. A commenter in the original September 3, 2024 discussion drew the line precisely: "llms.txt isn't really designed to help with scraping; it's designed to help end-users use the information on web sites with the help of AI." The SEO industry adopted the file in 2025 as a visibility lever. It was the one tactic every generative engine optimization checklist could ship in an afternoon. A large share of today's "does it work" fight is two groups holding the file to jobs it never claimed. Skeptics test it as a rankings lever. Evangelists sell it as one.

The spec: llms.txt, llms-full.txt, and Markdown mirrors

The llms.txt specification at llmstxt.org requires exactly one element: an H1 with the site or project name. Convention adds a blockquote summary, H2 sections of annotated links, and an "Optional" section agents may skip when context is short. Two companions grew alongside the spec. llms-full.txt holds the site's full content in one Markdown file. Per-page .md mirrors serve clean copies.

A minimal valid file looks like this:

Three format details carry most of the file's value. The blockquote under the H1 is the summary a model reads first. Give it one or two sentences stating what the site is and who it serves, in the vocabulary your users actually type. Each link line pairs a URL with a one-line description. The description is the point: it lets an agent decide what to fetch without fetching it. The "Optional" H2 is load-bearing vocabulary from the spec. Everything above it signals "read this first." Everything under it signals "only if context allows."

What the file is not: a URL dump. "llms.txt is not a docs url dump," as an HN submission titled it on June 3, 2026. Piping your sitemap through a script produces a file that defeats its own purpose, because curation is the mechanism. Twenty selected links with honest descriptions beat two hundred unsorted ones for the only reader the file has.

The two companions extend the same idea in opposite directions. llms-full.txt concatenates the actual content, so an agent gets everything in one fetch with no navigation. Mintlify defines it as "a single Markdown file containing the full plain text content of your site." The trade-off is size. When an HN reader pointed to Anthropic's platform.claude.com/llms-full.txt in June 2026, he attached a warning: it is "much bigger" than the index file. Per-page .md mirrors go the other way. The llmstxt.org proposal suggests serving a clean Markdown version of any page at its URL plus .md. The index can then link to noise-free versions of exactly the pages it curates. Some practitioners trust a fourth convention more than the file itself: a <link rel="alternate" type="text/markdown"> tag in each page's HTML. One HN commenter defended the tag on June 19, 2026 because it is "discoverable (just read the HTML) and naturally adapts to any website size and structure." The four artifacts are one strategy — machine-readable content plus an index of it. The index is the least important part.

The four artifacts of an llms.txt strategy. The curated index is the least important part.

Does llms.txt actually work? Both camps, dated

As a push lever, no. No major AI platform has confirmed its crawlers request llms.txt on their own. Google states in writing that its systems don't use it. The largest published log sample, 80,000 blogs, recorded zero requests. As a pull asset, yes: agents pointed at the file consume it today. That split decides whether your 30 minutes is invested or wasted.

"Does llms.txt actually work?" is one of the four People Also Ask questions Google attaches to this query. Searchers are asking. The ranked answers split into camps that mostly refuse to quote each other. Here is the ledger, both camps, dated:

The strongest exhibit on the skeptic side is not an opinion. It's a measurement. Herman Martinus runs Bear Blog, a platform hosting roughly 80,000 blogs. His server logs are one of the largest single windows into what AI crawlers request in the wild. His report from June 5, 2026: "I can definitively say llms.txt is not used by any AI players. I run a blogging platform with around 80k blogs and /llms.txt is not requested by anything (other than humans checking to see if there's an llms.txt path). All regular pages are aggressively scraped to the extent it's a problem I have to consistently manage, but not llms.txt." The bots are there. They want the content, badly enough to be a cost problem. They still don't ask for the file that was written for them. Independent nulls agree. The January 2026 experiment across 10 sites found no AI-traffic lift from the file. Google's own John Mueller made the same two claims a full year earlier: no confirmed consumers, no bot requests. His employer put it in formal guidance on May 15, 2026.

The strongest exhibit on the believer side is who publishes the file. In the same June 2026 thread where Bear Blog's logs landed, a commenter replied to "no AI players use it" with six live URLs. OpenAI's developer docs, Anthropic's docs, AWS documentation, GitHub, OpenRouter, and the Gemini CLI site all serve llms.txt today. This isn't stray adoption. Anthropic recommends the file in its Writing for Agents guidance. OpenAI ships the file for its Agents SDK and Agentic Commerce Protocol. The companies whose crawlers ignore the file at scale still consider it worth publishing for their own developer docs. That is not hypocrisy; it is a scope statement. Their docs teams are optimizing for a reader their crawler teams don't operate: an agent that arrives on instruction.

Google contradicts itself the same way, and the contradiction is archived. Search's May 15, 2026 guidance says skip the file. Chrome's Lighthouse 13.3, shipped days earlier, audits for it under Agentic Browsing. And on the "llms.txt" query itself, Google's AI Overview promotes the file. Our July 9, 2026 snapshot captured it describing the file as "an open standard designed to guide AI crawlers and Large Language Models (LLMs) to the most authoritative, machine-readable content on a website." Google's answer engine endorses the file on the exact SERP where Google's search team would tell users to ignore it. Passionfruit's llms.txt explainer — a competitor, credited because they got this right — decomposes the split correctly. Search is answering "does this affect ranking?". Lighthouse is answering "can a browser agent parse this site?". Neither team is wrong inside its own scope.

The reconciliation: llms.txt is pull, not push

Both ledgers are right, because they measure different verbs. Every skeptic exhibit measures push: whether crawlers discover and fetch /llms.txt on their own while scraping. GPTBot, ClaudeBot, and their peers demonstrably don't. Every believer exhibit that survives scrutiny is pull: a reader pointed at the file on purpose. A developer on HN launches his platform with "To get started ask your agent to read https://walnut.sh/llms.txt" (June 29, 2026). Another HN user asks a SaaS founder to add the file so he can "just tell my agent to build something and point it at your llms.txt" (June 25, 2026). Lighthouse audits readiness for browser agents that navigate on instruction. Even the corpus's funniest data point fits the pattern. The one confirmed reader of these files in the wild is a human. An HN user got 36 upvotes on June 5, 2026 for admitting he manually browses /llms.txt versions of websites "because I find the content for LLMs straight to the point and clear."

The verdict this evidence supports: llms.txt works as a pull standard and fails as a push standard. In July 2026 it is mostly being sold as a push standard. Every disappointment in the skeptic ledger comes from expecting hands-off pickup. Every real win in the believer ledger involves someone aiming an agent at the file. Whether it's worth your 30 minutes reduces to one question. Does anyone point an agent at your site on purpose?

Who has adopted llms.txt? The adoption statistics

Adoption is concentrated in developer documentation and near-absent elsewhere. An independent January 2026 crawl of 1,500 sites found 0.2% publishing llms.txt. Google's index holds 30,000–60,000 of the files, per Wix's October 2025 research. Both adoption statistics describe the same fact. Tens of thousands in absolute terms is a fraction of a percent of the web.

The roster of named adopters reads like a developer-tools conference: OpenAI, Anthropic, AWS docs, GitHub, OpenRouter, Cloudflare, Search Engine Land, Pydantic, Laravel Cloud, weather.com. Cloudflare's developers.cloudflare.com/llms-full.txt circulated on HN as the canonical example back in July 2025. The distribution engine behind the long tail is platforms, not decisions. Mintlify generates llms.txt and llms-full.txt automatically for the documentation sites it hosts. "These work really well," per a Laravel ecosystem developer on HN, June 5, 2026. GitBook does the same. Wix exposes one for its sites. When a docs platform flips a default, thousands of files appear overnight. No site owner is required.

That supply-side pattern explains the entire skeptic/believer split. Publishing the file costs almost nothing. So publishing happened, tens of thousands of times. Consumption was always the contested half, and the logs say it never caught up. A standard can rack up impressive-sounding adoption statistics while its intended reader ignores it, because "adoption" here measures writers, not readers. The counter-signal is worth logging too. Astro's audience is exactly the demographic that pulls these files, and Astro still removed its own in May 2026. The maintenance cost skeptics predicted is real. A file nobody fetches is a file nobody notices going stale.

What the debate lacks is longitudinal data collected by someone with no position to defend. A quarterly llms.txt adoption study is the next data project on this site's roadmap. Sample, method, and raw counts will be published, including our own server's fetch logs for /llms.txt. Both camps get the same numbers, whichever way they point.

How to create an llms.txt file

Creating an llms.txt file takes four steps. Choose the 10–50 pages an AI agent should read first. Write the Markdown file: H1 name, blockquote summary, H2 link sections with one-line descriptions. Serve it from your root directory. Put a quarterly review in the calendar. A stale map misleads agents, which is worse than no map.

Step 1 — curate, don't export. List the pages that answer two questions: what is this site, and what would an agent need from it? Core docs, pricing, the pages that define your product. If the list passes 50, you're exporting a sitemap, not editing an index.

Step 2 — write the file. Follow the skeleton from the spec section above: an H1 site name (the only required element), a blockquote stating what the site is in your users' vocabulary, H2 sections of link lines in the form - [Title](https://example.com/page.md): one-line description, and an "Optional" section for the long tail. Write descriptions as answers, not marketing. An agent choosing what to fetch is the whole audience.

Step 3 — serve it at the root. The file must respond at yourdomain.com/llms.txt as plain text or Markdown. Can your stack also serve .md mirrors of the linked pages and an llms-full.txt? Then the index gets a payload worth pointing to.

Step 4 — schedule the refresh. Quarterly, minimum, for a site that publishes regularly. Every dead link and outdated description in the file is served directly to the most literal reader on the internet.

Annotated real-site files and copy-paste example templates by site type live in our llms.txt examples breakdown. It covers SaaS docs, blog, ecommerce, and local business. If you want a first draft in seconds, the llms.txt generator builds one from your sitemap. Edit the curation by hand afterward. Selection is the one step no generator tool does well.

Go deeper

• llms.txt Examples: 12 Real Files, Fetched and Torn Down — 12 real llms.txt examples, fetched and annotated.

llms.txt vs robots.txt: what's actually different

robots.txt is an exclusion protocol. It tells crawlers where not to go, every major search engine honors it, and it has been an IETF standard (RFC 9309) since 2022. llms.txt is an invitation. It suggests what to read first, carries no enforcement, and remains an unratified proposal. One controls access. The other curates attention.

The verdict the table supports: these files aren't rivals, and neither substitutes for the other. They answer opposite questions — "what may you take?" versus "what should you read?". Is your actual goal controlling AI crawler ingestion, deciding which bots get your content at all? That is robots.txt and firewall territory, managed per user agent. Start with the AI crawlers hub and the per-bot references like ClaudeBot . Compliance there is strong but not absolute. A February 2026 HN report logged GPTBot requests against a disallowed private mirror. Enforcement-grade blocking happens at the WAF, not in a text file.

One popular critique lands on the wrong file. "If they want it, they will take it, polite directives in text files will have no effect," an HN commenter wrote of the file on June 19, 2026. That describes robots.txt, which issues directives and depends on voluntary compliance. llms.txt demands nothing, so it has nothing to be ignored except an offer. Its failure mode isn't defiance. It's indifference.

The verdict: worth 30 minutes, never worth a strategy

Publish llms.txt if you run developer docs, agents consume your product, or your platform generates the file for free. The 30 minutes buys real pull-workflow value plus a passing Lighthouse agentic audit. Skip it if the goal is Google rankings, AI Overview citations, or ChatGPT mentions. No published evidence supports that outcome, and Google says so in writing.

Ship it for documentation sites. This is the one segment where the pull workflow is routine today. Developers point coding agents at docs. Anthropic's agent guidance recommends the file. On Mintlify or GitBook the marginal cost is zero, because the platform already generates it. Not shipping it there is leaving a free artifact unshipped.

Ship it if agents consume your product. APIs, SDKs, MCP servers, integrations — anywhere "point your agent at our docs" is a real onboarding sentence, the file is that sentence's landing page. The walnut.sh launch pattern ("ask your agent to read our llms.txt") is what working consumption looks like.

Ship the Markdown, not just the map. The durable half of this work is clean .md versions of your key pages. The file is merely their index. A site with Markdown mirrors and no index is more agent-readable than a site whose llms.txt points at HTML noise.

Skip it as a Google tactic. The May 15, 2026 guidance is explicit: not needed for AI Overviews, AI Mode, or any generative Search feature. Anyone selling llms.txt as AI-Overview insurance is selling against Google's own documentation.

Skip it as a citation shortcut. The documented pattern runs the other way. An agency operator on r/MarketingandAI (June 29, 2026, 15 upvotes) ran the full on-site checklist for a B2B client: "Schema, an llms.txt file, rewrote half the site into FAQ blocks. Nothing. Genuinely zero change over like two months." The client started appearing in ChatGPT answers only after a third-party "best companies" roundup added him: "That was it. That was the whole thing." A root-directory file doesn't build entity trust. Sources talking about you do.

Never pay for it as a line item. Seven of the 89 results on this SERP are generator tools, most of them free, ours included. The file is a 30-minute artifact with a quarterly touch-up. An agency invoice with "llms.txt optimization" on it is a tell about the rest of the invoice.

Ours is live at /llms.txt , because this site doesn't recommend artifacts it won't run. The current file follows the structure this page teaches. Trimmed excerpt:

Line by line: the blockquote carries the three entity facts an agent needs to describe us correctly. What we are. What we test. Which engines we cover. The Guides section holds pillar pages only. Descriptions state what each page settles, in query vocabulary, not slogans. The long tail sits under Optional, where the spec says a context-limited reader may stop. We serve llms-full.txt and .md mirrors alongside it. Our fetch logs go into the adoption study when it ships. If no crawler ever requests our own file unprompted, that number gets published with the rest.

Частые вопросы