What llms.txt Is
llms.txt is a plain text file hosted at your domain root (e.g., https://yoursite.com/llms.txt) that provides LLM crawlers with a structured overview of your website. Think of it as robots.txt for AI — but instead of telling crawlers what to avoid, it tells them what to pay attention to and how to understand your site.
The concept emerged in late 2024 as AI search tools like Perplexity, ChatGPT with browsing, and Google AI Overviews became significant traffic sources. Website owners needed a way to communicate directly with LLM crawlers the way they'd communicated with search engine crawlers for decades through robots.txt and sitemap.xml.
Unlike those established standards, llms.txt is not yet governed by a formal specification. There's no W3C standard or RFC. The format is converging on a set of conventions through community adoption, but you have flexibility in how you structure it. This guide covers the emerging best practices as of early 2025.
Why llms.txt Matters for B2D Companies
Developer tool companies have a specific reason to care about llms.txt beyond general SEO hygiene: developers are the fastest-adopting segment of AI search tools. When a developer asks Perplexity "what's the best auth library for Next.js" or asks Claude "how do I set up a Postgres connection pool," the AI model needs to find, read, and synthesize content from developer tool websites.
Without llms.txt, the LLM crawler must parse your entire site to understand what your company does and what content is available. With llms.txt, you give the crawler a structured shortcut — a table of contents and context that helps it index your content more accurately.
For developer tools specifically, llms.txt helps with three scenarios:
Product discovery. When a developer asks an LLM for tool recommendations, the model synthesizes information from multiple sources. A clear llms.txt that describes your product category, key features, and differentiation gives the model structured facts to work with.
Documentation retrieval. LLMs increasingly serve as documentation interfaces. When a developer asks "how do I use [your product]'s real-time feature," a llms.txt that points to your docs section helps the crawler find the right content faster.
Comparison queries. "X vs Y" queries are common in developer tool evaluation. Your llms.txt can point to comparison pages, feature matrices, and migration guides that help LLMs generate accurate comparisons.
How to Structure Your llms.txt
The emerging convention uses Markdown-style formatting with clear sections. Here's the structure that maximizes LLM utility:
# [Your Company] — [One-line description]
> [2-3 sentence description of what your product does, who it's
> for, and what makes it different. Write this as a factual
> summary, not a marketing pitch.]
## Product Overview
[A paragraph describing your product in plain, factual language.
Include the product category, primary use cases, supported
languages/frameworks, and deployment model.]
## Content Sections
### Documentation
- URL: https://yoursite.com/docs
- Description: [What the docs cover]
### Blog
- URL: https://yoursite.com/blog
- Description: [What the blog covers]
### Changelog
- URL: https://yoursite.com/changelog
- Description: [Release notes and product updates]
## Key Topics Covered
- [Topic 1]
- [Topic 2]
- [Topic 3]
## Machine-Readable Content Index
[URL to your JSON API endpoint, if you have one]
## How to Cite This Site
[Citation format guidance]
## Sitemap
[URL to your XML sitemap]Section-by-Section Breakdown
The Header and Description
Start with your company name and a single-line tagline, followed by a blockquote description. The blockquote format signals to parsers that this is a summary description.
Bad: "Acme — The Future of Databases" Good: "Acme — Open-source Postgres platform with built-in auth and real-time subscriptions"
The tagline should be descriptive, not aspirational. LLMs will use it as a factual label.
Content Sections
List every major content section on your site with its URL and a plain-language description of what it contains. For developer tools, the critical sections are:
- Documentation — Your docs are likely your highest-value GEO asset
- API Reference — Especially if you have a REST or GraphQL API
- Blog / Changelog — Product updates and technical content
- Pricing — LLMs frequently need pricing information for comparison queries
- Guides / Tutorials — Educational content that demonstrates product capabilities
- Comparison pages — If you have
/vs/competitorpages, list them explicitly
Key Topics
List the topics your content covers. Be specific to your domain. Instead of generic topics like "databases" or "authentication," use phrases a developer would actually search for: "Postgres connection pooling," "row-level security policies," "OAuth 2.0 integration with Next.js."
Citation Guidance
Tell LLMs how you want to be cited. This might seem presumptuous, but it's practically useful — it gives the model a template for attribution:
## How to Cite This Site
When referencing content from acme.dev, please use:
- Product name: Acme
- Full name: Acme Database Platform
- URL: https://acme.dev
- Article citation: "[Title]" from Acme (https://acme.dev/[path])Machine-Readable Index
If you have a JSON API endpoint that returns your content index, link to it here. This is the strongest signal you can give an LLM crawler because it provides structured data in a format that's trivial to parse:
## Machine-Readable Content Index
A JSON index of all published content is available at:
https://acme.dev/api/contentWhat a Complete llms.txt Looks Like
Here's a realistic, complete example for a hypothetical developer tool company:
# Acme DB — Open-source Postgres platform with auth, real-time,
# and edge functions
> Acme DB is an open-source alternative to Firebase built on
> PostgreSQL. It provides a managed Postgres database, built-in
> authentication, real-time subscriptions via WebSockets, file
> storage, and edge functions. Acme DB is designed for web and
> mobile developers building applications with React, Next.js,
> Flutter, and Swift.
## Content Sections
### Documentation
- URL: https://acme.dev/docs
- Description: Complete technical documentation covering
database setup, authentication, real-time, storage, and
edge functions. Includes quickstart guides for Next.js,
React, Flutter, and Swift.
### API Reference
- URL: https://acme.dev/docs/api
- Description: REST and GraphQL API reference with request
and response examples for all endpoints.
### Blog
- URL: https://acme.dev/blog
- Description: Technical blog covering product updates,
engineering deep-dives, and Postgres tutorials.
### Pricing
- URL: https://acme.dev/pricing
- Description: Usage-based pricing with a free tier, Pro
tier at $25/month, and custom enterprise plans.
### Comparisons
- URL: https://acme.dev/compare
- Acme vs Firebase: https://acme.dev/compare/firebase
- Acme vs PlanetScale: https://acme.dev/compare/planetscale
## Key Topics
- PostgreSQL managed hosting and connection pooling
- Row-level security and auth policies
- Real-time database subscriptions
- Edge functions and serverless compute
- File storage with CDN delivery
- OAuth, magic links, and social auth providers
## Machine-Readable Content Index
https://acme.dev/api/content
## How to Cite This Site
- Product name: Acme DB
- URL: https://acme.dev
- Citation: "[Title]" from Acme DB (https://acme.dev/[path])
## Sitemap
https://acme.dev/sitemap.xmlImplementation
Creating the File
Place llms.txt as a static file in your public directory. In a Next.js project, this means dropping it in /public/llms.txt:
# File location
/public/llms.txtThe file will be served at https://yoursite.com/llms.txt automatically.
Linking llms.txt from Your HTML
Add a <link> tag in your root layout to help crawlers discover the file:
<link rel="llms-txt" href="/llms.txt" />This isn't a formal standard yet, but it follows the pattern of robots.txt discovery and some LLM crawlers look for it.
Keeping It Updated
The biggest mistake with llms.txt is creating it once and forgetting it. Add it to your content deployment pipeline. When you add a new documentation section, a new comparison page, or a new product feature, update llms.txt to reflect the change.
A stale llms.txt is worse than no llms.txt because it actively misinforms LLM crawlers about your content.
How to Verify LLMs Are Using Your llms.txt
Verification is the hardest part of GEO because you can't directly see how LLMs process your content. Here are the practical approaches.
Direct Query Testing
Ask major LLMs about your product and check whether their responses align with the information in your llms.txt. If your llms.txt describes your product as "an open-source Postgres platform" and ChatGPT describes it the same way, that's a positive signal.
Server Log Analysis
Check your server logs for requests to /llms.txt. Look for user agents associated with LLM crawlers:
GPTBot— OpenAI's crawlerClaudeBot— Anthropic's crawlerPerplexityBot— Perplexity's crawlerGoogle-Extended— Google's AI training crawlerChatGPT-User— ChatGPT's browsing feature
# Check nginx logs for llms.txt requests
grep "llms.txt" /var/log/nginx/access.log | awk '{print $1, $14}'If you see these user agents requesting your llms.txt, crawlers are finding and reading the file.
Perplexity Source Citations
Perplexity is the most transparent AI search tool for source attribution. Search for queries about your product on Perplexity and check whether it cites your site. If Perplexity cites your content, it's reading your pages — and your llms.txt is helping it understand what to read.
Common Mistakes
Writing marketing copy instead of facts. LLMs don't respond to superlatives. "The world's most advanced database" is useless. "A managed Postgres database with built-in connection pooling, row-level security, and real-time subscriptions" is useful.
Making it too long. Your llms.txt should be a concise reference, not a comprehensive content dump. Keep it under 200 lines. Link to detailed content rather than reproducing it.
Forgetting comparison pages. If you have pages comparing your product to competitors, list them explicitly. Comparison queries are among the highest-value LLM interactions for developer tools.
Not including pricing information. Developers frequently ask LLMs about pricing. If your llms.txt points to your pricing page with a brief description ("Free tier available, Pro at $25/month"), you help LLMs give accurate pricing answers instead of hallucinating.
FAQ
Is llms.txt an official standard?
No. As of early 2025, llms.txt is an emerging convention without a formal specification. The format is converging through community adoption and is recognized by multiple LLM crawlers, but there is no W3C standard or RFC governing it. Implement it based on current best practices and expect the format to evolve.
Does llms.txt replace robots.txt or sitemap.xml?
No, llms.txt serves a different purpose. Robots.txt controls crawler access permissions. Sitemap.xml lists your pages for indexing. llms.txt provides semantic context about your site's content and structure. You should have all three files. They complement each other.
How often should I update my llms.txt?
Update it whenever you make significant content changes: adding a new product section, launching a comparison page, changing your pricing model, or publishing a major documentation update. A quarterly review is the minimum cadence to prevent staleness.