llms.txt and AI Visibility: What It Is—and What It Isn't

Key Takeaways

• llms.txt is a community proposal for a concise Markdown guide that helps compatible agents find useful information on a website. It is not a crawler-control or indexing standard.
• The file does not block access, protect private content, grant or withhold permission, require attribution, or guarantee that an AI system will use the listed pages.
• Support is system-specific. Google states that Search—including its generative AI features—ignores llms.txt, while some documentation tools and agents use it as a navigation aid.
• The current v2 proposal allows a file at the site root or within a subpath and adds standard link relations so agents do not have to guess where the file or a Markdown page version lives.
• Create and maintain llms.txt only when a defined audience or compatible workflow can benefit. Do not prioritize it over accessible webpages, sound crawler controls, internal links, sitemaps, and useful source content.

---

llms.txt is easy to misunderstand because its filename resembles two established web conventions: robots.txt and sitemap.xml.

That resemblance has encouraged a much larger promise than the proposal makes.

An llms.txt file has been described as a place to allow or block AI crawlers, prevent model training, protect private pages, demand attribution, improve AI rankings, and make a website discoverable to language models. Those are different jobs involving different systems. The file cannot perform them simply because a publisher writes instructions in Markdown.

The useful idea behind llms.txt is narrower: give a compatible agent a concise, curated starting point for understanding and navigating a website.

That can be valuable for complex documentation, APIs, policies, and other information-heavy sites. It is not proof that a provider will look for the file, follow its links, retrieve the content for a buyer's question, cite the source, or recommend the organization.

The right way to evaluate llms.txt is therefore not "Does the file exist?" It is:

Which agent or workflow is expected to use this file, how will it discover it, and what observable task should become easier?

---

What llms.txt Actually Is

The current llms.txt v2 proposal describes a Markdown file that provides background information, guidance, and links to more detailed, LLM-friendly resources.

The file can sit at the origin root, such as:

https://example.com/llms.txt

Version 2 also defines files within subpaths, such as:

https://example.com/docs/llms.txt

A subpath file covers the pages beneath that path. When several files apply, the proposal says the most specific one should be used.

The format is intentionally small and predictable:

1. an H1 containing the project or site name—the only required element;
2. an optional blockquote with a concise summary;
3. optional non-heading details that help interpret the resources; and
4. optional H2 sections containing Markdown lists of links, with a name and optional description for each resource.

The Optional section is a convention for secondary links an agent can skip when context is limited. Under v2, that label does not carry a special processing rule.

The proposal expects an agent to view or search the concise file, select the relevant resource, and follow its link. Detailed content stays on the linked page rather than turning llms.txt into a copy of the whole site.

Why it matters: llms.txt is best understood as a curated navigation document. Its value depends on a client that knows about it and a publisher that maintains useful destinations.

---

What Changed in llms.txt Version 2

Version 2 was published on August 10, 2026. It responds to an important weakness in the original proposal: discoverability.

If an agent lands on an ordinary webpage, it should not have to guess whether a related llms.txt or Markdown version exists. Version 2 proposes standard link relationships that can appear as HTML link elements or HTTP Link response headers:

• rel="alternate" type="text/markdown" points to a Markdown version of the current page.
• rel="describedby" points to the llms.txt file that covers the page.

For example:

<link rel="alternate" type="text/markdown" href="/docs/getting-started.md">
<link rel="describedby" href="/docs/llms.txt">

The proposal permits a Markdown alternative to append .md to the original URL or replace its extension with .md. The exact pattern matters less than advertising the relationship consistently.

Version 2 also clarifies subpath scope and removes the earlier context-expansion tooling from the proposal. The current specification centers on a small guide plus linked Markdown-friendly resources.

This has a practical implication for another file publishers sometimes encounter: llms-full.txt. The current v2 specification does not require or assign a standard role to that filename. A platform may still generate it as its own convention, but it should not be presented as a required part of the v2 proposal. Page-level Markdown alternatives are the documented v2 pattern for detailed content.

Why it matters: A file at a predictable URL is useful only if the intended client knows to check it. Version 2 supplies an explicit discovery mechanism without turning the file into a crawler directive.

---

llms.txt Is Not robots.txt

robots.txt communicates crawler access preferences using the Robots Exclusion Protocol. An llms.txt file supplies descriptive context and links.

The difference is not cosmetic.

What is its primary job? robots.txt states which URLs identified crawlers are allowed to request. llms.txt gives compatible agents a curated overview and links.

Is it a crawler-control convention? robots.txt is. llms.txt is not.

Can it express user-agent-specific access rules? robots.txt can. llms.txt cannot under the proposal.

Does it enforce compliance? Neither does. A crawler must choose to obey robots.txt. A client must choose to discover and use llms.txt.

Can it protect confidential information? Neither can.

Can it request attribution? Either file can state a preference, but neither creates technical enforcement.

Does it guarantee indexing or AI visibility? Neither does.

Google's robots.txt guidance makes two limitations especially important:

1. crawler instructions cannot enforce behavior; and
2. robots.txt should not be used to protect private information.

The same security boundary applies even more strongly to llms.txt, which is a public content guide. Never list a secret, unpublished, gated, or sensitive resource in it. Protect private material with authentication and server-side authorization. Do not rely on a public text file to make a public URL private.

If the goal is to manage an identified AI crawler, use the mechanism documented by that crawler's operator—often a robots.txt user-agent rule—then verify whether a CDN, firewall, or bot-management layer is applying the same policy. The technical-readiness guide covers that delivery path.

Why it matters: Treating llms.txt as a permission file can create a false sense of control while the underlying content remains publicly accessible.

---

llms.txt Is Not a Sitemap

A sitemap supplies a search engine with a structured inventory of URLs the publisher considers important. An llms.txt file supplies a much smaller, curated path to selected resources and can include contextual descriptions.

Typical format. A sitemap is XML, RSS, Atom, or text, depending on the sitemap type. llms.txt is Markdown following the proposal's structure.

Intended scope. A sitemap covers important crawlable URLs, potentially at large scale. llms.txt covers a concise selection of resources useful to an agent.

Primary use. A sitemap supports URL discovery and crawl planning for supporting search engines. llms.txt supports on-demand context and navigation for compatible agents.

URL metadata. A sitemap can include supported details such as modification dates and language relationships. llms.txt uses headings, links, and short natural-language descriptions.

Inclusion, citation, or ranking. Neither file guarantees that listed URLs will be crawled, indexed, cited, or ranked.

Google explains that a sitemap can help search engines discover URLs, but it does not guarantee that every listed item will be crawled or indexed. llms.txt has an even narrower and less universal role: it can help only when the receiving system supports or is explicitly directed to use it.

Do not copy an entire sitemap into llms.txt. A long, indiscriminate URL dump defeats the proposal's purpose. Select the resources an agent would need to answer common, high-value questions accurately.

Why it matters: A sitemap is an inventory. llms.txt is an editorially selected guide. Neither is a command to include the site in an answer.

---

llms.txt Is Not Structured Data, Metadata, or an API

Several machine-readable layers can describe or expose a website, but they are not interchangeable.

• Structured data attaches defined types and properties to a page or entity. It can support eligible search features and explicit page interpretation.
• Metadata describes the document through titles, descriptions, canonical URLs, robots directives, and social properties.
• An API or feed can expose structured, queryable, and frequently updated data under a defined contract.
• llms.txt provides a short Markdown overview and routes a compatible agent toward selected resources.

If an agent needs current product inventory, pricing, availability, or account-specific data, a maintained API or feed may be more appropriate than prose links. If the goal is to identify an article's author and publication date, accurate Article structured data can express that relationship. If the goal is to specify a canonical URL, use the canonical mechanism.

Adding the same claim to every layer does not make it more true. The values should be consistent, but each mechanism should perform its documented job.

Why it matters: Choosing the right layer prevents llms.txt from becoming an unstructured container for crawler rules, SEO directives, legal terms, and data that belongs elsewhere.

---

Which Systems Support llms.txt?

There is no universal support answer.

The v2 proposal documents growing adoption among documentation platforms, content-management tools, and agent-oriented workflows. Several AI companies publish llms.txt files for their developer documentation. Perplexity's own documentation pages, for example, direct documentation readers and agents to its llms.txt documentation index.

That demonstrates a useful publishing pattern. It does not establish that the same company's consumer answer engine automatically looks for and follows llms.txt on every website.

This distinction is essential:

Publishing an llms.txt file for your own documentation is evidence that you make the file available. It is not, by itself, evidence that your crawler, search index, model-training pipeline, or consumer assistant honors other sites' files.

Google provides unusually explicit guidance. Its current generative AI search documentation says Google Search does not use llms.txt, including for its generative AI capabilities, and that publishing one neither helps nor harms Search visibility or rankings.

For every other target system, look for an affirmative statement in the operator's current documentation. Check whether the support applies to:

• an agent or coding tool supplied with a website URL;
• a developer-documentation interface;
• a web-search crawler;
• an answer-time retrieval agent;
• a training crawler;
• a consumer assistant; or
• a manual workflow in which the user provides the file.

Those are different use cases. Support in one does not imply support in the others. The absence of published documentation also does not prove that a system never fetches the file; it means the publisher should not build a visibility forecast on undocumented behavior.

Why it matters: "Supported by AI" is too broad to act on. Name the client, product surface, use case, and documented behavior.

---

Can llms.txt Improve AI Visibility?

It can improve navigation for a compatible agent under the right conditions. That is not the same as improving market visibility.

Consider the full path:

file discovery → file retrieval → link selection → page retrieval → interpretation → answer use → citation → recommendation

An llms.txt file addresses the first few steps only when the client supports the pattern or is explicitly directed to the file. It does not control later decisions.

Even a successful agent interaction establishes only that the file helped in that task. It does not show that:

• an unrelated provider will use the same file;
• the listed page will be indexed;
• the page will be retrieved for an unbranded buyer question;
• the content will be preferred over competing sources;
• a visible citation will be provided;
• the brand will be framed accurately; or
• the response will recommend the brand.

This is the same measurement boundary discussed in AEO vs GEO vs SEO. A technical or navigational aid should be measured at the layer it can affect.

The defensible claim is:

A maintained llms.txt file can provide a compatible agent with a concise, publisher-curated route to relevant content.

The unsupported claim is:

Adding llms.txt makes the website discoverable, trusted, or citation-ready across AI platforms.

Why it matters: Accurate scope allows teams to test a real agent workflow without mistaking the result for proof of broad AI visibility.

---

When Creating llms.txt Is Worthwhile

The file is most plausible when four conditions are present:

1. There is a defined agent audience. Developers, customers, employees, or partners are using tools that can follow the file or can be directed to it.
2. The site is difficult to navigate programmatically. The useful material spans extensive documentation, policies, reference pages, tutorials, or multiple content formats.
3. Clean destinations exist. The linked pages are accessible, authoritative, current, and preferably available in concise Markdown-friendly form.
4. Someone will maintain it. The owner can remove retired links, update descriptions, and keep the guide aligned with the site.

Strong use cases can include:

• developer documentation and API references;
• technical product manuals;
• policy and compliance libraries;
• complex educational resources;
• support knowledge bases; and
• controlled agent workflows where the file is an agreed entry point.

The case is weaker when:

• the site contains only a few clearly linked pages;
• no target client has documented or tested support;
• the file would duplicate a sitemap without curation;
• the only objective is a speculative ranking or citation gain;
• the organization cannot keep the links and descriptions current; or
• important access, rendering, indexability, or content problems remain unresolved.

Why it matters: A low-cost file can still create maintenance debt and false confidence. Build it for a user or workflow, not for the presence of a passing audit check.

---

How to Create a Useful llms.txt File

1. Define the scope

Decide whether the guide represents the whole site or one section. Place it at /llms.txt for the origin or within the relevant path for a scoped documentation area.

2. Identify the questions an agent should answer

Start with real tasks: understanding the product, selecting the correct API endpoint, finding a policy, comparing plans, or troubleshooting a known issue. Then select the minimum set of resources needed to complete those tasks.

3. Write a factual summary

The opening blockquote should identify the organization, product, project, or documentation set without marketing superlatives. State important limits or distinctions in the detail that follows.

4. Organize links by user purpose

Use H2 sections such as "Start Here," "Concepts," "API Reference," "Policies," or "Troubleshooting." Link names should explain the destination. Descriptions should state what the agent will find there.

5. Prefer clean, stable destinations

Use canonical URLs that return the expected content without authentication, challenge pages, redirect chains, or empty client-rendered shells. Where practical, provide the Markdown alternatives described by v2.

6. Advertise the relationships

Add rel="describedby" from covered pages to the applicable llms.txt file. Add rel="alternate" type="text/markdown" where a page has a Markdown version. Apply the relationship consistently through HTML or HTTP headers.

7. Keep secondary material secondary

Use the Optional section for material that is helpful but not necessary for the primary tasks. Do not force every page into the file.

8. Assign an owner and review cadence

Review the file after documentation restructures, product changes, migrations, and important content updates. A concise guide with dead or misleading links is worse than a smaller accurate one.

---

Example llms.txt File

This example follows the v2 structure without pretending to set crawler permissions:

# Acme Analytics Documentation

> Acme Analytics is a reporting platform for subscription businesses. These resources explain its data model, API, authentication, and reporting workflows.

Important notes:
- API examples use version 3 unless a page states otherwise.
- Account-specific data requires authenticated API access.
- The public documentation does not contain customer records.

## Start Here

- [Platform overview](https://example.com/docs/overview.md): Core concepts, terminology, and product boundaries.
- [Quick start](https://example.com/docs/quick-start.md): Create a test project and run the first report.

## API Reference

- [Authentication](https://example.com/docs/api/authentication.md): Supported authentication methods and token handling.
- [Reports API](https://example.com/docs/api/reports.md): Endpoints, parameters, response fields, and examples.

## Policies

- [Data retention](https://example.com/policies/data-retention.md): Retention periods and deletion behavior.

## Optional

- [Release notes](https://example.com/docs/releases.md): Version history and recent changes.

Notice what the file does not contain:

• Allow or Disallow rules;
• a demand that every AI system cite the company;
• confidential links or credentials;
• claims that conflict with the source pages;
• every URL on the site; or
• invented permissions for training, indexing, or answer generation.

The file tells a compatible agent where useful public information lives. Actual access control, licensing, and data authorization remain elsewhere.

---

How to Audit llms.txt Without Turning It Into a Score

A practical audit should answer four groups of questions.

Delivery

• Does the intended URL return a successful response?
• Is the final response the expected file rather than a redirect loop, error page, or bot challenge?
• Does a subpath file sit within the scope it is intended to describe?
• Can the target agent fetch it through the CDN and security layers?

Structure

• Is the file valid, readable Markdown?
• Does it contain the required H1?
• Are the summary, detail, sections, links, and descriptions concise?
• Are secondary resources separated from the primary path?

Destination quality

• Do all links resolve to the intended canonical resources?
• Are the pages public, current, accurate, and useful for the named task?
• Are Markdown alternatives present and advertised where the workflow expects them?
• Do descriptions agree with the linked content?

Supported use case

• Which client or agent is expected to use the file?
• Is support documented, configured, or supplied through the user's workflow?
• Can a controlled test show that the agent discovers the file, follows the relevant link, and answers the target question more reliably?
• Are server logs or tool traces available to distinguish a fetch from an assumption?

Report the evidence, not a binary "AI-ready" result.

/llms.txt returns 200. That establishes the file is publicly fetchable. It does not establish that any target system requested or used it.

The file follows v2 structure. That establishes a compatible parser can interpret its organization. It does not establish provider support or visibility impact.

Every link resolves. That establishes the curated path is technically usable. It does not establish that the destinations will be retrieved or cited.

A selected agent follows the file. That establishes the tested workflow supports it under the observed conditions. It does not establish automatic use by other agents or consumer products.

Answers improve in a controlled task. That establishes the file may help that task and configuration. It does not establish broad ranking, citation, or recommendation gains.

Why it matters: A useful audit separates file quality from client support and separates client support from market visibility.

---

Where llms.txt Belongs in the Priority List

For most public websites, llms.txt is an optional enhancement—not the first technical task.

Prioritize these foundations first:

1. important pages return the expected content;
2. intended crawlers are allowed through robots.txt, the CDN, and the application;
3. public pages are indexable where appropriate;
4. canonical URLs, redirects, sitemaps, and internal links identify the correct pages;
5. primary content is available in a form the target system can retrieve and render;
6. metadata and structured data accurately describe the visible document; and
7. the content contains useful, supported information for the questions being tested.

Then add llms.txt when a compatible agent workflow or documentation audience gives it a clear purpose.

Viziquo's Technical Readiness review treats the file proportionately: it can test whether one exists and whether it is usable, but its absence is not equivalent to a crawler block or an inaccessible page. Page Analysis examines the destination pages that still carry the substantive content.

---

The Right Conclusion

llms.txt is a sensible proposal for a real problem: helping agents navigate websites that were designed for people and wrapped in complex layouts, scripts, and navigation.

Version 2 makes that proposal more useful by defining subpath scope, Markdown alternatives, and explicit discovery links.

It still is not a universal AI-control plane.

Use it to curate public information for compatible agents. Do not use it to protect private content, replace crawler controls, duplicate a sitemap, demand attribution, or predict citations.

Most importantly, do not record "llms.txt present" as evidence that a brand is visible in AI answers. Visibility requires observing the brand in defined buyer conversations, inspecting the sources and framing, and repeating the test across relevant providers. That is the job of an AI visibility audit, not a file-presence check.

If you'd rather see which technical findings actually deserve attention for your brand, fill out the form below.