llms.txt for API docs, measured on two Twitter (X) API doc sets
Most llms.txt explainers describe the format and stop. This page measures it on two real doc sets for the same API surface: X's official developer docs at docs.x.com, and twitterapi.io's site and docs. Both publish llms.txt and llms-full.txt, both sites are built on the same docs platform (Mintlify, which generates the files), and the numbers still come out very differently, which tells you more about how to write your own file than the spec alone.
Three things are here that the usual explainer skips. First, file-by-file measurements: bytes, links, sections, estimated tokens and whether each file actually follows the llmstxt.org v2 shape, including where our own file does not. Second, a study of 959 X posts that mention llms.txt, to see what developers ask and which tools they connect it to. Third, an honest line between llms.txt, MCP and SKILL.md, since X ships all three as its Agent Resources and so do we.
What this page does not do: it does not claim which files Cursor, Claude Code or any other client fetches on its own, because we did not log client traffic, and it quotes no adoption counts beyond what the spec's author and X state in their own words.
What the llms.txt spec says (v2), in one screen
The proposal lives at llmstxt.org, written by Jeremy Howard of Answer.AI. It was first published in September 2024; version 2 arrived in August 2026. The idea is simple: HTML pages wrap their content in navigation and scripts, context windows are finite, so a site should hand agents a short, expert-written map and clean Markdown copies of the pages behind it.
The file layout, in order. An optional byte-order mark; an H1 with the project or site name (the only required part); a blockquote with a short summary; zero or more Markdown sections of any type except headings; then zero or more H2 sections, each a "file list" of Markdown links written [name](url), optionally followed by a colon and a note. An H2 named Optional is, by convention, for links an agent can skip.
Where it lives. At /llms.txt, or at any subpath such as /docs/llms.txt. v2 defines scope: a file covers the pages under its path, and the most specific file wins. The spec argues against /.well-known/ as the only location because path owners (for example a project site on a shared host) cannot write there.
Markdown copies of pages. Pages that agents may need should be available as Markdown at the same URL with .md appended (page.html.md) or with the extension replaced (page.md). v2 allows both.
Discovery. v2 recommends standard link relations: rel="alternate" type="text/markdown" for a page's Markdown copy and rel="describedby" for the llms.txt that covers it, as HTML <link> elements or an HTTP Link: header.
How agents use it. v2 states the expectation plainly: agents view or search the llms.txt, then follow the relevant links. The v1 context-expansion tool (llms_txt2ctx) was dropped from the proposal, and with it the mechanical meaning of the Optional section. The spec's author writes that thousands of sites publish the file and that docs platforms generate it automatically; Chrome's Lighthouse now includes an llms.txt check in its agentic-browsing audits, which per its docs fails only on a server error and marks a 404 as not applicable, since the file is optional.
What it is not. It is not access control (that is robots.txt) and not a full page list (that is sitemap.xml). And llms-full.txt does not appear anywhere in the v2 text: it is a convention popularized by docs platforms, which is why its contents vary so much from site to site.
What X ships at docs.x.com/llms.txt and llms-full.txt
X lists llms.txt, llms-full.txt, skill.md and two hosted MCP servers under the Agent Resources section of its developer docs. Its own page on the topic (docs.x.com/tools/llms-txt) says the files "give AI tools like Cursor, Windsurf, and Grok structured access to X API documentation", and suggests pasting docs.x.com/llms-full.txt into Grok or adding it as documentation context in Cursor or Windsurf.
The root file is small and deliberately nested. It has an H1, a blockquote, four H2 sections and 26 links, 17 of them to .md pages, and it points to section indexes instead of listing every page. We fetched every index it links to:
Across the seven index files that is 125.4 KB, 964 links and about 32,085 tokens; almost every link ends in .md, because every page on docs.x.com also serves a Markdown copy (we checked docs.x.com/x-api/posts/get-post-by-id.md: HTTP 200, text/markdown). The same root is mirrored at docs.x.com/.well-known/llms.txt, and the site's robots.txt carries a Content-Signal: ai-train=yes, search=yes, ai-input=yes line.
The llms-full.txt is the outlier. It concatenates 1,148 pages (each begins with a Source: line) into one 4.9 MB file with 1,890 code blocks. At the characters-divided-by-4 estimate that is about 1.23 million tokens, roughly 1,424 times the root index. X's own docs tell agents to start from the root llms.txt and read the AGENTS.md first; the full file is there for tools that search it rather than read it end to end.
Discovery headers. Both docs.x.com URLs we requested (an HTML page and its .md copy) returned an HTTP Link: header with rel="llms-txt" and rel="llms-full-txt" (plus api-catalog, MCP server card, agent card and agent-skills relations). Those are the docs platform's relation names, not the rel="describedby" that v2 recommends, so a strictly v2 client would not recognize them yet.
What twitterapi.io ships, and where our own file breaks the spec
We publish two pairs of files, and they were written for different readers. The marketing site (twitterapi.io/llms.txt and twitterapi.io/llms-full.txt) is hand-written and served from the repository's public/ folder. The docs site (docs.twitterapi.io/llms.txt and llms-full.txt) is generated by the docs platform from the pages themselves.
The hand-written root is useful but off-spec. It opens correctly with an H1 and a blockquote, then puts install commands in three code blocks and H3 subheadings inside its sections, and every link is a <https://...> autolink rather than a [name](url): note list item. A parser that follows the v2 grammar finds zero file-list links in it. The content (what the API is, how to connect the MCP server, where the long-form docs live) is what agents need; the shape is what a strict parser cannot read.
The generated docs index is closer, but it skips the summary. docs.twitterapi.io/llms.txt lists 69 links, 68 of them to .md pages, but they sit directly under the H1 with no blockquote and only one H2 (for the OpenAPI specs) at the end. The pages are all reachable; the agent just gets no one-paragraph orientation first.
Our llms-full.txt files are product briefs, not dumps. The marketing one is a 2.8k-token summary with category labels, the 12 MCP tools, pricing and FAQs. The docs one concatenates 68 pages into about 13.0k tokens, small enough to paste into a single prompt, which is the practical reason to keep a reference compact. X's docs cover the whole platform (Ads API, Enterprise, SDKs), so its full file cannot be small; ours covers one product.
Gaps we measured on ourselves. twitterapi.io/.well-known/llms.txt returns 404, marketing pages send no Link: header pointing at the file, and marketing pages have no .md copies (twitterapi.io/pricing.md is a 404). The docs site does all three the same way docs.x.com does, because the same platform serves both. These are on our list to fix; they are printed here because a page about the format should not hide its own author's misses.
llms.txt vs llms-full.txt: which one an agent should read
The two files answer different questions. llms.txt answers "where is the page about X?"; llms-full.txt answers "what does the whole manual say?". On a small product they overlap; on a platform the size of X they do not.
Rule of thumb from the numbers. If your llms-full.txt stays under a few tens of thousands of tokens, it can be pasted or attached whole and will be read. Past that, agents should search it or fetch the per-page .md copies through the index. X's own guidance follows the same logic: start at the root llms.txt, follow the section index, fetch the page.
Keep each link's note specific. The note after the colon is what an agent uses to choose a link without opening it. X's indexes show the risk of generated notes: 159 of the 384 links in /x-api/llms.txt and 162 of the 343 in /enterprise-api/llms.txt carry the same note, "Reference documentation for the endpoint and related functionality", with the endpoint name missing. The link titles still work, but the notes add nothing an agent can choose by.
llms.txt vs SKILL.md vs MCP: when each one wins
X ships all three as Agent Resources, and so does twitterapi.io. They are not competing formats; they sit at different points between reading and acting.
llms.txt wins when the agent is writing code against your API and needs the exact parameter name, error shape or pagination rule. It costs nothing to serve, works in any tool that can fetch a URL, and needs no install or credentials. It is also the only one of the three that a person can read and fix in a minute.
llms.txt loses when the agent must act. Reading the search reference does not return a single post; an MCP tool call does. It also loses on judgment: a SKILL.md can say "use a since_time: cursor and stop at has_next_page: false", which is an instruction, while llms.txt can only point at the page that explains it. And it loses on very large doc sets if the agent is handed the full file instead of the index.
MCP has costs llms.txt does not. It needs a client that supports MCP, credentials, and an install step; X's hosted server requires your own X developer app and the xurl bridge for OAuth, per X's MCP page. Every tool call is a billed API call on whichever rate card you use. llms.txt is a static file.
What the conversation on X suggests. In our sample, 12.6% of llms.txt posts also mention MCP, while only 1.1% mention SKILL.md and 1.6% AGENTS.md, so for now people pair llms.txt with MCP far more than with the instruction formats. The deeper SKILL.md comparison, with X's skill next to ours, is on our SKILL.md explainer.
What people say about llms.txt on X: 959 posts, measured
To see how developers actually talk about the file, we sampled posts containing "llms.txt" (reposts excluded) through /twitter/tweet/advanced_search: 900 posts from the Latest feed, covering about 26 days at roughly 34.7 posts a day, plus 120 from the Top feed, 959 unique posts after deduplication. Classification uses transparent keyword rules, listed in the method section.
Two audiences use the same word. Roughly a quarter of posts frame llms.txt as a developer-docs tool and almost as many frame it as an SEO or AI-search visibility tactic, often next to robots.txt (8.7%), structured data (9.0%) or sitemaps (5.6%). The spec itself says the file is used mainly at inference time, when an agent is helping a user, and makes no ranking claim; if you are reading about llms.txt as a search-ranking lever, that claim comes from the post, not from the proposal.
Questions are common. About one post in five asks something, and 47.7% carry a link. Of posts that link anywhere, 17.9% link to another X post and 7.2% to GitHub; 15.5% of all posts link directly to an llms.txt or llms-full.txt file, usually someone showing the file they just shipped.
Tool names are rarer than you might expect. Only 18.7% of posts name any AI product; ChatGPT / OpenAI and Claude lead, Perplexity (3.2%), Gemini (2.5%), Cursor (2.2%) and Grok (2.1%) trail. Explicit skepticism ("doesn't work", "no evidence", "nobody reads it") shows up in 1.7% of posts by our keyword rule, which undercounts sarcasm.
Reach is uneven. The median post has 2 likes and 101 views; the 90th percentile has 48 likes and 5,230 views. Posts in X's Top feed have a median of 231 likes, so the ranking surfaces a handful of widely shared posts over a long tail.
A copy-paste llms.txt template for an API vendor
This template follows the v2 order exactly: H1, blockquote, a short notes block without headings, then H2 file lists of [name](url): note items, with secondary links under Optional. Replace the placeholder host with your docs domain and keep every link pointing at a .md copy where you have one.
Put rules in the notes block, not only links. The three lines an agent most often gets wrong on a data API are authentication, pagination and cost per call. Saying them in the file means an agent that reads nothing else still writes a correct loop.
Split by section once you pass a few hundred links. X's root lists section indexes instead of 964 links; copy that pattern for a large platform and keep each section file under roughly 15k tokens, as X's largest index is.
Serve it with the right headers. Content-Type: text/plain or text/markdown works; add a Link: </llms.txt>; rel="describedby" header to pages if you want v2 discovery, and keep a copy at /.well-known/llms.txt if your platform expects it.
Test it the way the spec suggests. Give an agent only the llms.txt and ask it three real questions about your API. If it cannot answer which header authenticates, how to get page two, and what an empty result costs, the file is missing a line. The short linter below checks the structural part.
# Acme API
> Acme API returns X (Twitter) posts and profiles as JSON. One header, `X-API-Key`, authenticates every call. Base URL: https://api.acme.example
Important notes for agents:
- Paginate with `cursor`; stop when `has_next_page` is false.
- Errors carry a `msg` field; HTTP 429 means slow down, not retry instantly.
- Prices are per result returned; see the Pricing page before looping.
## Start here
- [Quickstart](https://docs.acme.example/quickstart.md): Key, first request, reading the JSON
- [Authentication](https://docs.acme.example/authentication.md): The one header every call needs
- [Pagination](https://docs.acme.example/pagination.md): Cursor rules for search, timelines, followers
## Endpoint reference
- [Search posts](https://docs.acme.example/api/search.md): Query operators, sort order, page size
- [User profile](https://docs.acme.example/api/user-info.md): Fields returned and the not-found shape
- [Errors](https://docs.acme.example/errors.md): Status codes and what client code should do
## For agents
- [OpenAPI spec](https://docs.acme.example/openapi.json): Machine-readable contract
- [MCP server](https://docs.acme.example/mcp.md): Hosted tools for chat clients
- [SKILL.md](https://docs.acme.example/skill.md): Task instructions for coding agents
## Optional
- [Changelog](https://docs.acme.example/changelog.md): Breaking changes and new endpoints
- [llms-full.txt](https://docs.acme.example/llms-full.txt): Every page above in one file
Check any llms.txt with a short Python script
The script fetches a file and prints the same measurements used on this page: bytes, lines, estimated tokens, link counts by kind, .md links, and four v2 shape checks. Run it as python3 llms_lint.py https://docs.x.com/llms.txt, then on your own file.
Run against the files above, it reports all four checks passing for docs.x.com/llms.txt and each of X's section indexes, fails the file-list check for twitterapi.io/llms.txt (its 16 autolinks are not [name](url) list items), and fails the blockquote check for docs.twitterapi.io/llms.txt. Tokens are estimated as characters divided by 4, a rough rule for English text and Markdown; real tokenizer counts differ by model.
"""llms_lint.py - fetch an llms.txt and report size, structure and llmstxt.org v2 shape."""
import re
import sys
import requests
url = sys.argv[1] if len(sys.argv) > 1 else "https://docs.x.com/llms.txt"
r = requests.get(url, timeout=30)
r.raise_for_status()
text = r.text
lines = text.splitlines()
nonblank = [l for l in lines if l.strip()]
md_links = re.findall(r"\[([^\]]+)\]\((https?://[^)\s]+)\)", text)
autolinks = re.findall(r"<(https?://[^>\s]+)>", text)
h2 = [l[3:].strip() for l in lines if l.startswith("## ")]
link_items = [l for l in lines if re.match(r"^\s*[-*]\s+\[[^\]]+\]\(https?://", l)]
checks = {
"H1 is the first line": nonblank[0].startswith("# ") if nonblank else False,
"blockquote summary follows the H1": len(nonblank) > 1 and nonblank[1].startswith(">"),
"has H2 file-list sections": bool(h2),
"file-list items use [name](url)": bool(link_items),
}
print(f"{url}\n {len(text.encode()):,} bytes, {len(lines)} lines, ~{len(text) // 4:,} tokens (chars / 4)")
print(f" {len(md_links)} [name](url) links, {len(autolinks)} <url> autolinks, {len(h2)} H2 sections")
print(f" links ending in .md: {sum(1 for _, u in md_links if u.split('#')[0].endswith('.md'))}")
for name, ok in checks.items():
print(f" [{'ok' if ok else '--'}] {name}")
What the study cost, on two rate cards, and how we collected it
Question. How large and how well-formed are the llms.txt and llms-full.txt files on the two X API doc sets, and what do people posting about llms.txt on X talk about?
Files. Fourteen files fetched once each with curl: the docs.x.com root llms.txt, its six nested indexes, its llms-full.txt and skill.md; twitterapi.io's llms.txt and llms-full.txt; docs.twitterapi.io's llms.txt, llms-full.txt and skill.md. Measured with a script that counts bytes, lines, H2 headings, [name](url) links, <url> autolinks, code fences and Source: lines, and checks the v2 order. Tokens are estimated as characters / 4; no tokenizer was used.
Posts. Query "llms.txt" -filter:retweets on /twitter/tweet/advanced_search, 45 pages of Latest (900 posts, about 26 days) and 8 pages of Top (120 posts), 959 unique. Collected in 2026. Labels come from case-insensitive keyword rules (question = contains a question mark or opens with a question word; SEO framing = words such as SEO, GEO, AI search, ranking, visibility; docs framing = docs, documentation, API, SDK, developer, coding agent), so a post can match several labels or none (55.5% matched neither framing).
Cost. 53 requests returned 1,020 posts. At twitterapi.io's $0.15 per 1,000 tweets returned that is about $0.15 in total. The same 1,020 post reads on X's pay-per-usage rate card (docs.x.com/x-api/getting-started/pricing, $0.005 per post read) would be $5.10, about 33 times more per 1,000 posts ($5 versus $0.15).
What the sample cannot show. It is a recent-posts sample plus X's Top ranking, not all posts ever written; keyword rules miss sarcasm and posts in other languages (80.6% were in English); and nothing here measures whether any AI client fetches llms.txt. We did not log client requests, so we make no claim about what Cursor, Claude Code or any other tool reads automatically.
Privacy. The authors are mostly private individuals, so raw search pages stayed on the collection machine and were not published. The published study holds aggregates and de-identified rows only (no post ids, URLs, text, handles or author ids); only organization and documentation hostnames are reported from links. Files, script and results are in the repository under data/studies/llms-txt/.
"""llms_txt_on_x.py - sample X posts that mention "llms.txt" and measure what they talk about.
Uses one twitterapi.io endpoint: /twitter/tweet/advanced_search (15 credits per tweet returned,
$0.15 per 1,000 tweets; an empty page costs the 15-credit minimum).
Prints aggregates only: share of questions, links to llms.txt files, tools and formats named.
"""
import os
import re
import time
from collections import Counter
import requests
BASE = "https://api.twitterapi.io"
HEADERS = {"X-API-Key": os.environ["TWITTERAPI_IO_API_KEY"]}
MAX_PAGES = 45 # about 900 posts at 20 per page
TOOLS = {"Cursor": r"\bcursor\b", "Claude": r"\bclaude\b", "ChatGPT/OpenAI": r"\bchatgpt\b|\bopenai\b",
"Gemini": r"\bgemini\b", "Perplexity": r"\bperplexity\b", "Grok": r"\bgrok\b"}
FORMATS = {"MCP": r"\bmcp\b", "llms-full.txt": r"llms-full", "SKILL.md": r"skills?\.md",
"AGENTS.md": r"agents?\.md", "robots.txt": r"robots\.txt"}
def search(query, query_type="Latest", max_pages=MAX_PAGES):
cursor, seen = "", set()
for _ in range(max_pages):
params = {"query": query, "queryType": query_type}
if cursor:
params["cursor"] = cursor
r = requests.get(f"{BASE}/twitter/tweet/advanced_search", headers=HEADERS, params=params, timeout=30)
if r.status_code == 429:
time.sleep(5)
continue
r.raise_for_status()
body = r.json()
for t in body.get("tweets", []):
if t["id"] not in seen:
seen.add(t["id"])
yield t
if not body.get("has_next_page"):
break
cursor = body.get("next_cursor", "")
posts = list(search('"llms.txt" -filter:retweets'))
n = len(posts) or 1
questions = links_to_file = 0
tools, formats = Counter(), Counter()
for t in posts:
text = (t.get("text") or "").lower()
questions += "?" in text
urls = [u.get("expanded_url", "") for u in (t.get("entities") or {}).get("urls", [])]
links_to_file += any(u.split("?")[0].rstrip("/").endswith(("llms.txt", "llms-full.txt")) for u in urls)
tools.update(k for k, rx in TOOLS.items() if re.search(rx, text))
formats.update(k for k, rx in FORMATS.items() if re.search(rx, text))
print(f"{len(posts)} posts; questions {questions / n:.1%}; link to an llms file {links_to_file / n:.1%}")
print("tools named:", {k: f"{v / n:.1%}" for k, v in tools.most_common()})
print("formats named:", {k: f"{v / n:.1%}" for k, v in formats.most_common()})
print(f"cost: {len(posts)} tweets x $0.00015 = ${len(posts) * 0.00015:.3f}")
Questions readers ask
What is llms.txt?
A Markdown file at a site's root or any subpath that gives AI agents a curated map of the content worth reading: a required H1 with the project name, a blockquote summary, optional notes, and H2 sections listing [name](url): note links, ideally to Markdown copies of each page. The proposal is published at llmstxt.org (v1 September 2024, v2 August 2026).
What is the difference between llms.txt and llms-full.txt?
llms.txt is an index of links; llms-full.txt is the full documentation concatenated into one file. Only llms.txt is defined by the llmstxt.org spec; llms-full.txt is a convention from docs platforms. On X's docs the index is about 864 tokens and the full file about 1.23 million tokens across 1,148 pages, so agents should read the index and fetch individual pages.
Does X (Twitter) have an llms.txt for its API docs?
Yes. docs.x.com/llms.txt is a small root index that links to section indexes for the X API v2 (/x-api/llms.txt, 384 links), Enterprise APIs, the Ads API and the Python and TypeScript XDKs, plus docs.x.com/llms-full.txt, a skill.md and two hosted MCP servers. Every docs page also has a .md copy.
Should I use llms.txt or an MCP server for my API?
Both, for different jobs. llms.txt helps an agent find and read the right docs page so it writes correct code; it is a static file with no install or credentials. An MCP server lets the agent call the API directly from a chat client, which needs a compatible client, credentials and billed calls. A SKILL.md adds task instructions on top.
How do I use llms.txt with Cursor or Claude Code?
X's own docs suggest adding https://docs.x.com/llms-full.txt as documentation context in Cursor or Windsurf, or pasting it into Grok. For a large doc set, pointing the tool at the root llms.txt and letting it fetch the .md pages it needs keeps context small. We did not measure which files these clients fetch on their own, so test with your tool.
Does llms.txt help SEO or rankings in AI search?
The spec makes no ranking claim: it describes llms.txt as used mainly at inference time, when an agent is helping a user. In our sample of 959 X posts, 23.4% framed llms.txt as an SEO or AI-search tactic, but no source we checked shows a ranking effect. Chrome's Lighthouse checks for the file in its agentic-browsing audits, failing only on a server error.
Where should llms.txt live, root or /.well-known/?
The spec puts it at /llms.txt or any subpath like /docs/llms.txt, where it covers the pages under that path. It argues against /.well-known/ as the only location because path owners cannot write there. Some sites, including docs.x.com, serve a copy at /.well-known/llms.txt as well.
How big should an llms.txt file be?
Small enough to read whole. X's root is 3.4 KB (about 864 tokens) and its largest section index about 13.4k tokens. If a single index would list many hundreds of links, split it into section files under a short root, as X does, and keep a one-line note on every link.
Continue
- llmstxt.org: The /llms.txt file, v2 (format, location, link relations, Optional section)
- X developer docs: llms.txt and llms-full.txt (Agent Resources)
- X developer docs: root llms.txt with nested section indexes
- X developer docs: hosted X MCP (api.x.com/mcp via xurl) and Docs MCP
- Chrome for Developers: Lighthouse agentic-browsing llms.txt audit
- twitterapi.io docs: generated llms.txt index
- X API hub: endpoints, prices and guides
- SKILL.md explained: X's skill and the twitterapi.io skill compared
- Use Cursor with the Twitter (X) API through MCP
- Add an MCP server to ChatGPT
- Use Claude with the Twitter (X) API
- Twitter (X) MCP server: the same reads as tools
- xurl, X's CLI that bridges its hosted MCP server
- twitterapi.io API documentation
- Pay-per-call pricing for the twitterapi.io API
- 01X Radar Explained: Twitter (X) Keyword Monitoring, Tiers, Limits and an API Alternative
- 02TweetDeck (X Pro) Alternatives in 2026: Columns, Monitoring and Scheduling for Twitter (X)
- 03Twitonomy Review: What the Twitter (X) Analytics Tool Still Does, and an API Alternative
- 04Botometer in 2026: What It Scored, Its Status, and a Live Alternative for Twitter (X)
- 05Grok Connectors: Add a Custom MCP Connector for Live Twitter (X) Data
- 06How to Quote a Tweet (Quote Post) on X
- 07Twitter (X) Analytics Tools — Side-by-Side Comparison
- 08Twitter (X) User Search: Find Accounts by Name, Keyword or Bio
Stop reading. Start building.
Starter credits cover real testing on real data. Google sign-in, no card, no application queue.
Get an API key