llms.txt: complete guide for commercial websites

The llms.txt file has become one of the most discussed artefacts in the field of machine readability. Inspired by robots.txt, which guides search engines, it aims to offer language models a structured entry point for understanding a site. But between promise and reality, there are important nuances, especially for a commercial website.

This guide covers everything you need to know: what the file is, how to structure it, what to include, how to deploy it, and above all, why it is not sufficient on its own.

What is the llms.txt file

The llms.txt is a text file placed at the root of a website (accessible at yoursite.com/llms.txt). Its purpose is to provide language systems with a structured summary of the site: who you are, what you do, which pages are most important.

Unlike robots.txt, which tells crawlers what they can or cannot explore, llms.txt tells language models what they should read first and how to interpret the site.

The format is not yet formally standardized, but a convention is emerging around a few key sections:

• Identity: organization name, field of activity, positioning.
• Primary routes: the most important pages of the site, ranked by priority.
• In-depth corpus: editorial resources that support the offer.
• Proof: verifiable elements that reinforce credibility.
• Policies: conditions for content use by automated systems.

Why a commercial site needs one

A commercial site has specific stakes. Unlike a technical documentation site or a personal blog, it must communicate a structured offer, a clear positioning and proof of credibility. Yet language models do not always have the means to distinguish a sales page from a background page, proof from a claim, a primary service from a secondary one.

The llms.txt helps guide this reading. It acts as a machine-oriented table of contents, designed to reduce ambiguity and favour a more faithful interpretation.

For a site that adopts a machine-first approach, this file fits naturally into the overall architecture.

How to structure an effective llms.txt

Section 1: Identification

Start with a clear declaration of who you are. Include the organization name, domain (canonical URL), a positioning statement and the primary language.

Section 2: Priority routes

List the most important pages in order of priority. For a commercial site, this generally includes:

• the homepage;
• main service pages;
• the methodology or approach page;
• proof or case study pages;
• the FAQ;
• the contact or diagnostic page.

Each entry should include the full URL and a brief description of the content.

Section 3: Editorial corpus

Signal in-depth resources that support your offer: blog articles, guides, glossary, publications. These contents help models understand the depth of your expertise.

Section 4: Proof

Indicate verifiable elements: case studies, documented testimonials, external publications, structured data. AI gives more weight to claims supported by proof.

Section 5: Usage policy

Specify the conditions under which your content may be used by automated systems. This includes permissions, restrictions and attribution preferences.

What to include and what to avoid

Include

• Pages that best represent your offer and identity.
• In-depth content that demonstrates genuine expertise.
• Inspectable and verifiable proof.
• A clear hierarchical structure (from most important to least important).
• Concise, factual descriptions.

Avoid

• Weak or generic pages (pages “under construction”, empty content).
• Duplicates or near-duplicates.
• Temporary pages (ephemeral promotions, past events).
• Claims unsupported by proof.
• A flat inventory without hierarchy.

Technical implementation

Deploying an llms.txt is technically straightforward:

1. Create a structured text file following the sections described above.
2. Place it at the root of your site (accessible via /llms.txt).
3. Ensure the file is served with a text/plain MIME type.
4. Verify it is accessible without authentication or redirection.
5. Update it whenever the site structure changes significantly.

For sites built with a framework such as Astro, the file can be dynamically generated from content collections, which ensures it stays synchronized with the actual site.

The limitations of llms.txt

This is where most guides stop. But it is precisely where the reflection must begin.

It does not replace a clear architecture

If your site is poorly structured, the llms.txt will only point to confusing pages. The artefact does not fix the problem; it makes it more visible.

It does not compensate for a weak corpus

Signalling pages in a file does not make them more substantial. If your content lacks depth, proof or coherence, the llms.txt cannot invent what does not exist.

It does not resolve entity conflicts

If multiple sites or sources send contradictory signals about your organization, a local file will not settle the matter. Resolution requires broader governance.

It is only one layer among others

The llms.txt is part of a set of machine governance artefacts: entity-graph in JSON-LD, ai-manifest.json, usage policies, schema.org structured data. Taken in isolation, it does not constitute a strategy.

We have already explored these limitations in a dedicated article: llms.txt alone will not fix how AI understands you.

llms.txt within a governance strategy

The file reaches its full value when it sits within an interpretive governance approach. In that framework, it is not an isolated gesture, but one piece of a coherent set:

• a machine-first web architecture that structures pages for all readers;
• an entity-graph that stabilizes the organization’s identity;
• structured data that maps the relationships between services, proof and content;
• an ai-manifest.json that declares the organization’s policies toward AI systems;
• an llms.txt that condenses all of this into a readable entry point.

It is this coherence that produces value, not the file alone.

Frequently asked questions

Do I need an llms.txt if I already have JSON-LD?

Yes. The two play complementary roles. JSON-LD structures data for engines and systems. The llms.txt offers a narrative summary oriented toward language models.

How often should it be updated?

Whenever the site structure, offer or proof changes significantly. An outdated file is worse than an absent one.

Do search engines read the llms.txt?

Not directly in their ranking algorithms. But AI search systems (Bing Chat, Google AI Overviews, Perplexity) may use it as a complementary signal.

Conclusion

The llms.txt is a useful and increasingly expected artefact. For a commercial website, it offers a real opportunity to guide machine reading. But the artefact must not be confused with the strategy.

A good llms.txt reveals the quality of the site it describes. If that quality is weak, the file will not mask it. If it is strong, the file will make it more accessible to systems that matter more and more.

To go further, explore the AI governance and machine readability service or start with a diagnostic.