llms.txt and robots.txt for AI: what each file does and how to set them up

In September 2025 we ran a micro-training with Francisco A. Kemeny, co-founder of Kemeny Studio, a generative AI consultancy, who has been doing SEO since 2002. Halfway through he opened the robots.txt of his own site, kemeny.studio, and walked through it bot by bot. That demo is the backbone of this guide, which replaces the two posts we had on llms.txt and AI bot policies.

Our position up front: publish an llms.txt if it takes you less than an afternoon, because it's cheap and forces you to organize your content, but don't expect it to move your citations. The file that actually decides whether an AI assistant can read you is robots.txt, and that's where we keep finding expensive mistakes.

How your page gets into an AI answer

Francisco opened with a technical point that changes how you frame the problem.

"The chatbot itself doesn't have an index the way we know it from Google." — Francisco A. Kemeny, co-founder of Kemeny Studio [15:23]

A language model predicts words from what it learned. According to Francisco, your content reaches it through three routes (minute 11:29):

1. Training: whatever a crawler collected before the model version was frozen. It's what ChatGPT "knows" without searching.
2. RAG (retrieval augmented generation): the system pulls text into the context window and the model answers from that plus the prompt.
3. Tool calling: the assistant queries a search engine, reads the results page and, for Perplexity, ChatGPT search or Gemini, visits your page and extracts the text.

Route 1 is governed by the training crawler. Routes 2 and 3 are governed by search crawlers and by agents that act when a user asks for something. Those are different user-agents, allowed or blocked separately.

A test you can run today

During the session Francisco asked ChatGPT, in instant mode, who the president of Chile is: it went and searched on its own. Then he asked about the president of El Salvador and told it not to search, and it answered from training, with no sources. When he asked about El Salvador's elections with search on, the sources came back in an order very close to Google's (minute 25:13).

Try it with a question from your own category. An answer without sources is training talking. An answer with sources is search, and your robots.txt decides whether your site can be on that list.

What llms.txt is, per the original proposal

Jeremy Howard, co-founder of Answer.AI, proposed llms.txt in September 2024; the spec lives at llmstxt.org. It's a markdown file at the root of your domain (yourdomain.com/llms.txt) that summarizes the site and links the pages a model should read, minus the HTML noise.

The format is short:

• An H1 with the site name. It's the only required part.
• A blockquote with the summary.
• Optional paragraphs or lists, with no headings.
• H2 sections holding link lists in the form - [name](url): notes.
• A section literally named ## Optional, last, with what an agent can skip when it needs a short context.

A minimal example:

# SearchBrand

> Platform that measures whether ChatGPT, Gemini, Perplexity, Claude and AI Overviews mention and cite a brand.

## Pricing

- [Pricing](https://searchbrand.ai/pricing): Pro and Pro+ plans in USD

## Optional

- [About SearchBrand](https://searchbrand.ai/hey-ai-about-us): Company facts written for AI assistants

The llms-full.txt you'll see on documentation sites isn't part of the proposal. It's a convention from docs tooling: one file with all the content in markdown. Anthropic publishes both in its docs, and Francisco showed them as an example (minute 45:25).

"You don't need a plugin, you don't need anything; you just need a clear structure of your site, reflected in markdown and available at the root." — Francisco A. Kemeny, co-founder of Kemeny Studio [45:25]

In the demo, kemeny.studio also references the file with a <link> tag in the <head> and mentions it in robots.txt. Neither is part of the proposal. They do no harm, and there's no evidence any bot reads them.

What we know as of September 2026 about who uses it

Keep publishing and consuming apart.

Plenty of sites publish one: developer docs, SaaS companies, and directories such as llmstxthub.com that collect them. Anthropic has one in its docs. A lab publishing its own llms.txt tells you it helps its users paste documentation into a chat or into Cursor. It doesn't tell you its crawlers use the file to choose what to cite.

Using it to rank or cite, on the other hand, hasn't been confirmed by any major provider. As of this guide, OpenAI, Anthropic, Google, Perplexity and Microsoft have not publicly stated that they use a site's llms.txt to decide whether to surface or cite it. In 2025 Google's John Mueller compared the file to the old keywords meta tag. Francisco said much the same on the webinar: the big labs don't use it "officially", in quotes.

"This is still theory, and we have to start putting it into practice." — Francisco A. Kemeny, co-founder of Kemeny Studio [15:23]

Francisco said that Kemeny Studio clients who index their documentation are seeing "20% more mentions", plus more ChatGPT and Gemini traffic in Analytics once FAQs are tied to product descriptions in the llms.txt. He flagged the confounder himself: overall AI usage is growing. It's a consultancy figure with no published methodology, so treat it as a hypothesis to test on your own site.

robots.txt: the file that actually has an effect

robots.txt has been an IETF standard since 2022 (RFC 9309), and the major AI operators document which user-agent they use for each purpose. This is the list behind the generator below. Check it against each company's docs, because it changes several times a year.

User-agent	Company	What it's for	Honors robots.txt?
GPTBot	OpenAI	Model training	Yes, stated
OAI-SearchBot	OpenAI	ChatGPT search index, to surface and cite sites	Yes, stated
ChatGPT-User	OpenAI	Reads a page when a ChatGPT user or a custom GPT asks	OpenAI notes it may not apply, since a person initiates it
ClaudeBot	Anthropic	Model training	Yes, stated
Claude-SearchBot	Anthropic	Quality of Claude's search results	Yes, stated
Claude-User	Anthropic	Reads a page when a Claude user asks	Yes, stated
PerplexityBot	Perplexity	Perplexity's index to surface and link sites; they say it doesn't train foundation models	Yes, stated
Perplexity-User	Perplexity	Reads a page when a user asks	Perplexity says it generally ignores robots.txt
Google-Extended	Google	Token, doesn't crawl: decides whether Googlebot's crawl trains Gemini and grounds the Gemini app	Yes. Does not control Google Search, AI Overviews or AI Mode
Applebot-Extended	Apple	Token, doesn't crawl: decides whether Applebot's crawl trains Apple Intelligence	Yes. Doesn't remove you from Siri or Spotlight
CCBot	Common Crawl	Open web archive that many labs train on	Yes, stated
Bytespider	ByteDance	Training ByteDance models	CDN operators have reported it doesn't always comply
meta-externalagent	Meta	AI model training and indexing for Meta products	Yes, stated; user-initiated fetches use meta-externalfetcher

Two rows cause most of the confusion we see. Blocking Google-Extended to get out of AI Overviews does nothing, because AI Overviews and AI Mode run on Googlebot; leaving them means leaving Google. And GPTBot and OAI-SearchBot are independent: you can refuse to train OpenAI's models and still get cited in ChatGPT.

"If you block GPTBot from your site, your site won't be indexed in the model's general context." — Francisco A. Kemeny, co-founder of Kemeny Studio [25:13]

How long before you see a change

For training, Francisco estimated six months to a year, since the change only shows up with the next major model release. Hussam Sufan, SearchBrand co-founder and the session's host, added a caveat: if you let the browsing bot in, "that time could be much shorter." Search-driven changes can show up within days or weeks.

The most-specific-group trap

Under RFC 9309, each bot follows only the most specific group that names it. Write User-agent: GPTBot with Allow: / and GPTBot ignores your User-agent: * block, along with its Disallow: /admin and Disallow: /checkout. We see this a lot: someone "opens" the site to AI bots and opens the admin panel to them by accident. That's why the generator repeats private paths in every group.

Your CDN can overrule your robots.txt

Hussam asked about Cloudflare live (minute 24:03). Since July 2025, Cloudflare blocks AI crawlers by default on new domains and offers to charge them per crawl. Your robots.txt can say Allow: / while the CDN returns a 403. If your site sits behind Cloudflare or another CDN with bot protection, check that dashboard before any file.

The policy we recommend

For a B2B brand or an online store that wants to show up in AI answers:

• Always allow search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and user-initiated agents. They're the ones that produce linked citations.
• Allow training bots unless your content is the product, as with a paywalled publisher or a dataset you sell. Then block the seven training rows in the table, GPTBot through meta-externalagent, and leave everything else open.
• Never block search while leaving training open. You hand over the content and lose the citation.

Generate and validate both files

# SearchBrand

> Plataforma chilena que mide si ChatGPT, Gemini, Perplexity, Claude y AI Overviews mencionan y citan a una marca, y qué contenido le falta para aparecer.

## Product and services

- [Cómo lograr que ChatGPT mencione tu marca](https://searchbrand.ai/es/chatgpt-mencione-marca): Qué mira ChatGPT al elegir marcas y cómo medir si te nombra

## Pricing

- [Precios](https://searchbrand.ai/pricing): Planes Pro y Pro+ en USD más IVA, con factura en pesos chilenos

## Guides

- [Medir tráfico de IA en GA4](https://searchbrand.ai/es/blog/medir-trafico-ia-google-analytics-4): Cómo crear un canal de tráfico de IA en Google Analytics 4 paso a paso
- [Auditoría AEO y GEO](https://searchbrand.ai/es/blog/auditoria-aeo-geo-2025): Checklist para auditar la presencia de una marca en motores de respuesta
- [Share of voice en IA](https://searchbrand.ai/es/blog/share-of-voice-en-ias): Cómo calcular qué parte de las respuestas de IA menciona a tu marca

## Company

- [Contacto](https://searchbrand.ai/es/contact): Agenda una demo o escribe al equipo de SearchBrand en Santiago

# robots.txt generated with SearchBrand's AI crawler matrix (list reviewed 2026-06-30)
# Each bot obeys only the most specific group naming it (RFC 9309), so private paths are repeated in every group.

# Model training: allowed
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Bytespider
User-agent: meta-externalagent
Disallow: /admin
Disallow: /app
Disallow: /api
Allow: /

# AI search and index: allowed
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Disallow: /admin
Disallow: /app
Disallow: /api
Allow: /

# User-initiated fetch: allowed
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
Disallow: /admin
Disallow: /app
Disallow: /api
Allow: /

# All other bots (search engines included)
User-agent: *
Disallow: /admin
Disallow: /app
Disallow: /api
Allow: /

Sitemap: https://searchbrand.ai/sitemap.xml

# LLMs: https://searchbrand.ai/llms.txt

The tool builds your llms.txt from your real URLs and your robots.txt from the policy you pick per category. Paste an existing llms.txt and it checks it against the llmstxt.org format and scores it. Everything runs in your browser; your files never reach a server. It can't check your links for 404s, because browsers block reads from other domains, so if you paste your sitemap it flags links that aren't in it.

To see whether those changes show up in AI answers, you can create a SearchBrand account with 10 free credits, no card required and track 10 fixed questions before and after.

Case study: SearchBrand's own llms.txt, step by step

Before publishing this guide we audited ours, and it wasn't pretty. Our internal technical audit found that searchbrand.ai's llms.txt pointed to /es/blog/rss.xml, which doesn't exist, and to /es/contacto/, when the real path is /es/contact. It was last updated on December 1, 2025. Nine months untouched, with two broken links, on the site of a company that sells AI visibility.

Here's how we're fixing it:

1. Check robots.txt against the table above: which bots are allowed in each of the three categories, and whether the CDN blocks any by default.
2. Run the llms.txt through the validator. A partial reconstruction of the old file, with the two broken links, one relative link and a section called "Opcional" instead of "Optional", scores 75 out of 100 with two format errors. The version built with the tool scores 94.
3. Pick 15 to 30 high-value URLs: pricing, the page on getting ChatGPT to mention your brand, our GA4 AI-traffic guide (our best non-brand page by clicks) and the AEO audit. Everything else goes under ## Optional or stays out.
4. Publish, reference the file from robots.txt and log the date.
5. Measure 30 days before and 30 days after: hits from the five bots we care most about (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot) in our Netlify and Cloudflare logs, AI-channel sessions in GA4, and SearchBrand mentions across 10 fixed questions.
6. Publish the numbers on this page, even if they're flat. If llms.txt moves nothing, that tells you how much time it deserves.

If your site has thousands of URLs

An attendee named Fernando asked what a bank or a big retailer with millions of products should do: list everything? Francisco's answer was to highlight what's valuable and leave the rest out (minute 45:25).

"As the model's context window fills up, the model tends to perform worse and worse." — Francisco A. Kemeny, co-founder of Kemeny Studio [45:25]

We tested it with the validator and a 15,000-product sitemap, the figure Francisco used for his e-commerce example. The generator keeps 30 priority URLs and pushes 14,970 into ## Optional. The file goes past 50 KB, loses 15 points for link count and lands at 65. Parking the catalog under Optional doesn't solve anything. For a retailer, the sensible list is categories and buying guides, shipping and returns policies, the FAQ, plus the 20 or 30 products people ask about most.

Francisco also suggested not doing it by hand: ask ChatGPT for a first llms.txt draft, or have Cursor write a script that reads your sitemap (minute 49:11). Run the output through the validator before uploading.

How to tell whether it worked

Nobody will tell you your llms.txt worked. You measure it with four sources, from hardest to softest.

Server logs. Filter by user-agent and count hits per bot per week. If GPTBot downloads your llms.txt you know it read the file, not that it uses it to cite you.

An AI channel in GA4. Francisco recommends a dedicated AI traffic channel you can compare against email or organic search, instead of only eyeballing referrals (minute 40:48).

Temporary chat. To rule out ChatGPT answering from the memory of your own past conversations, repeat the question in a temporary chat (minute 42:01). Francisco called it "the best acid test."

A daily monitor. Francisco has a scheduled task in ChatGPT that checks kemeny.studio every day and reports whether anything changed. For a long time it said "no change"; after some tweaks it started picking up differences. His summary: trial and error.

What we don't recommend

• Buying or installing an "llms.txt plugin" that promises ChatGPT rankings. The file is plain text, and no provider has confirmed it uses it for citations.
• Repeating the "70% better answer quality" stat making the rounds on LinkedIn. It came up in the webinar and we couldn't find a study behind it.
• Blocking Google-Extended to opt out of AI Overviews. That's not how it works.
• Dumping your full catalog into llms.txt.
• Expecting a difference from writing llms.txt in English versus Spanish. An attendee asked; Francisco and Hussam think it doesn't matter because models are multilingual, but nobody has measured it. Write it in your customers' language.
• Treating llms.txt as a substitute for content. Francisco estimates technical SEO does "20% of the work" and the other 80% is authority and relevance, what Google calls E-E-A-T. If your site is a client-rendered SPA with no prerendering, fix that first: most AI crawlers don't execute JavaScript, and no text file makes up for a blank page.

Once both files are live and you want the whole-site picture, move on to the AEO and GEO audit. Then track your 10 questions in SearchBrand: the free account comes with 10 credits, no card required.