llms.txt: Making Services Agent-Readable

Introduction

Every website that exists today was designed for a human to navigate with a browser. The human opens a page, reads menus, understands the visual hierarchy, clicks around, and figures out what the service offers. This is fine for humans. For AI agents, it's like reading a book by staring at the outside of the cover.

When an agent needs to use a service — understand its capabilities, find the relevant endpoints, know what it can extract or do — it currently has two options: browse the site like a human (slow, fragile, expensive), or read documentation that may be extensive, poorly organized, and not designed for machine consumption.

The llms.txt pattern is a third option: a that a service publishes intentionally, telling agents exactly what it offers.

Core Concept

What llms.txt Is

llms.txt is a plain text file that a service publishes at a well-known path (/llms.txt) on its domain. It contains a structured description of the service's capabilities, key resources, and how an agent should interact with it. Think of it as a for the agentic internet.

A minimal llms.txt might look like this:

# Example Service

> A data extraction platform providing structured access to web content.

## What We Offer
- Structured JSON extraction from common web sources
- 274+ pre-built extraction skills organized by domain
- CLI and API access with API key authentication

## Key Resources
- [Skill Catalog](https://example-service.com/catalog): Full list of available skills with schemas
- [API Docs](https://example-service.com/api): REST API reference
- [Getting Started](https://example-service.com/quickstart): 5-minute guide

## How to Use
Install the CLI: `npm install -g example-cli`
Authenticate: `example-cli auth --key YOUR_API_KEY`
List skills: `example-cli list`
Run a skill: `example-cli run skill-name --url TARGET_URL`

An extended version (llms-full.txt) often includes the full documentation content in a single file, optimized for a language model to ingest in one context window load rather than crawling through 50 separate documentation pages.

Why This Matters as Agents Proliferate

Right now, most agent interactions with services involve a browser skill navigating the service's website to figure out what it does. This is functional but inefficient — the agent is doing the same work every time it encounters the service, interpreting visual layout that was designed for humans.

As agent usage grows, services that publish llms.txt gain a structural advantage: agents that encounter them can onboard faster, with less token spend, with fewer errors, and with a description that the service itself has vetted as accurate.

The analogy to robots.txt is useful but inverted. robots.txt says "here's where crawlers are NOT allowed to go" — it's a restriction mechanism. llms.txt says "here's what agents ARE allowed and encouraged to do" — it's a discovery mechanism.

The browse.sh project implemented this pattern early: publishing both /llms.txt and /llms-full.txt at its domain. An agent encountering browse.sh can load the llms.txt file and immediately understand what the service offers, what skills are available, and how to install and invoke them — without scraping the site or reading through a human-oriented documentation structure.

For Operators: The Two Sides of llms.txt

If you're building automation, llms.txt gives you two opportunities:

Consuming llms.txt: When your agent needs to understand a new service, check for a /llms.txt at the root of the domain first. If it exists, load it as context. Your agent gets an accurate, service-sanctioned description of what's available — faster and more reliable than scraping the docs.

Publishing llms.txt: If you build internal services, tools, or agent-facing APIs, publishing a llms.txt makes those services dramatically easier for your own agents to discover and use correctly. You write the catalog once; every agent session benefits from it.

For an internal toolset — a gateway fronting a fleet of agents and services — a llms.txt for each service eliminates the need for agents to figure out capabilities from scratch every session. The catalog is the memory — explicit, current, and under your control.

Practical Application

Here's a concrete workflow for using llms.txt in your automation stack.

Checking for llms.txt before a browser skill:

Before dispatching a browser agent to scrape documentation from an unfamiliar service, check if the service has published a machine-readable catalog:

curl -s https://target-service.com/llms.txt
# If a file is returned: use it as agent context
# If 404: proceed with browser-based doc scraping

If the file exists, pass its contents as context to your agent before asking it to interact with the service. The agent has an accurate map before it starts navigating.

What to look for in a quality llms.txt:

A useful llms.txt answers three questions within the first 20 lines:

1. What does this service do in one sentence?
2. What are its primary capabilities?
3. Where do I find the technical documentation or API reference?

A llms.txt that requires extensive reading to get the basic picture is not serving its purpose. The format rewards concision.

Common Mistakes

Assuming llms.txt exists for every service. Adoption is growing but not universal. Always have a fallback — if no llms.txt, fall back to browser-based documentation discovery. Treat llms.txt as an enhancement, not a dependency.

Using a stale llms.txt as the only source of truth. A llms.txt published six months ago may not reflect current capabilities. When accuracy matters, verify key claims against the live service, especially for API endpoints and authentication methods.

Publishing a llms.txt that is really marketing copy. A useful llms.txt is precise and technical. It tells agents how to interact with the service, not why the service is great. Resist the temptation to make it a pitch document.

Ignoring llms-full.txt. Many services that publish llms.txt also publish llms-full.txt with complete documentation content in one file. When you need deep capability understanding (not just discovery), the full version is worth loading.

Summary

• llms.txt is a machine-readable service catalog, published at /llms.txt on a domain, designed for agent consumption
• It enables agent-first service discovery — understanding what a service offers without scraping or reading human-oriented docs
• robots.txt restricts crawlers; llms.txt invites agents — opposite in purpose
• Check for llms.txt before browser-based doc scraping; use it as context when it exists
• Publishing llms.txt for your own internal services makes them dramatically easier for your own agents to use correctly

What's Next

You now have all five layers of the Conductor Stack mapped, from conductor vision through browser skills, community libraries, structured output, execution environments, and service discovery. The final lesson brings it all together: why one agent is a toy, and what it means to conduct an orchestra of them. This is the capstone.