What your website tells AI assistants: robots.txt, llms.txt, and who reads what

Try this first. Type your website address into a browser and add /robots.txt to the end, like yourbusiness.com/robots.txt. What comes back is a page of plain text you probably did not know you had: a note pinned to your site’s front door, telling visiting programs which ones may come in.

If you handle marketing for your business, you have heard file names like these in vendor pitches, usually attached to a price. Here is the part the pitches skip. Your website already publishes a small set of plain-text files addressed to machines, and each file is read by a different audience. Once you know who reads what, you can check your own site in about ten minutes and judge any AI-optimization pitch the moment you hear it.

The note on your site’s front door

The file you just opened is robots.txt, a guest policy for visiting programs. Those visitors are bots: programs that read web pages the way a browser does, minus the person. Search engines send them, and so do the companies behind ChatGPT, Claude, and Perplexity. Each section of the file starts with a line naming one visitor (it begins User-agent:, the name a program announces itself with, like caller ID), followed by the rules for that visitor.

Two rules explain almost everything about how the file behaves. First, it can only restrict. A program you never mention is already allowed in, so an empty file and no file at all mean the same thing: everyone is welcome. Second, a named section replaces the general rules rather than adding to them. That quirk is the most common way a site blocks a bot by accident: someone adds a section for one visitor and forgets to repeat the rules everyone else follows.

One company, several bots

The names matter more than they look. OpenAI, the company behind ChatGPT, sends out different bots for different jobs. GPTBot collects pages that may be used to teach future models. OAI-SearchBot instead feeds the search results ChatGPT checks when it answers questions. OpenAI’s own documentation for site owners recommends letting OAI-SearchBot in, and warns that sites which shut it out “will not be shown in ChatGPT search answers.”

That distinction is the highest-stakes line in the whole file. Blocking GPTBot keeps your pages out of AI training material. Blocking OAI-SearchBot removes your business from ChatGPT’s search answers. Blocking the wrong one feels like blocking AI while quietly costing you the answers you wanted to be in.

Anthropic, the company behind Claude, runs its own separately named bots, and Perplexity does the same. Both publish the internet addresses their bots call from, so a web person can confirm a visitor is who it claims to be.

The rules that talk to nobody

Knowing who reads what also catches the opposite mistake: a rule addressed to a reader that does not exist. Brave is the one we check for most often. Brave runs its own search engine, so a robots file will often welcome a visitor named Bravebot. It reads as thorough. It does nothing.

Brave’s own help pages say its search bot deliberately announces no name of its own, partly so sites cannot let Google in while shutting everyone else out. It simply reads whatever Google’s bot may read. There is no Bravebot to welcome, so that block addresses an empty room, and nothing anywhere reports a problem.

That is the thing to know about machine-facing files: they fail silently. A wrong line produces no error, just nothing, which is how a confident-looking file stays wrong for months. This is why we check these against each company’s own documentation rather than against the lists that circulate online, and why a file that looks tidy is not the same as a file that works.

llms.txt, the table of contents for AI tools

The newest file in the pitches is llms.txt: a plain-text list of your important pages with a short description of each, written for AI tools. Three things about it are true at the same time.

First, the companies building the assistants publish one for their own documentation. Anthropic’s runs to hundreds of pages, and OpenAI, Perplexity, and Cloudflare each publish one too. Second, independent checks of website visit records keep finding that AI companies rarely fetch the file from other sites on their own. Third, a site that builds the file automatically from pages that already exist pays nothing to keep it, and it can never fall out of date. It also gets read the moment a person or an AI tool is pointed at it.

Our own site publishes llms.txt and llms-full.txt, generated fresh at every build. We do not expect either file to be the single reason a business gets named in ChatGPT. They earn their place a different way, as one layer among several that stack up: accurate facts, pages an assistant can read, listings that agree with each other, and files pointing at the right things. Alone each is small. Together they decide whether an assistant can describe your business correctly, and that is what puts you in the answer.

The llms.txt proposal has a quieter second half: a clean plain-text version of each important page, which OpenAI publishes for its own documentation. A properly built site gives you both at no extra cost. What deserves a hard look is a vendor selling the file alone, wired into nothing, as the thing that gets you recommended.

The list and the doorbell

Two older files still do the heaviest lifting. A sitemap is a list of every page you want search engines to find. It matters twice over now: AI assistants ask search engines before they answer, so a file that helps search engines helps them too. Your site almost certainly has one; our SEO basics guide covers where it lives and how to submit it.

IndexNow is the newer doorbell: a way to ping search engines the moment a page changes, instead of waiting to be revisited. Submit a change once and participating search engines share it among themselves. For a business that updates hours, prices, or services, that is the difference between the machines finding out today and finding out eventually.

The ten-minute self-check

The whole exercise fits before your next meeting.

1. Open yourbusiness.com/robots.txt. For every section that names a visitor, ask: do we know who this is, and did we mean it? Anything blocking GPTBot or OAI-SearchBot deserves a deliberate decision, not an inherited one.
2. Look near the bottom of the same file for a Sitemap: line, and open the address it points to.
3. Open yourbusiness.com/llms.txt. Missing is normal. Wrong is worth fixing.
4. Ask your web person, or any vendor pitching AI work, three questions: which bots do we allow or block by name, do we publish a sitemap and an llms.txt, and is anything turning away the AI search bots? Anyone doing real work can answer in minutes.

And keep one rule for the pitches themselves: a file is plumbing, not a ranking lever. If someone sells your Houston business a single file as the reason you will show up in AI answers, smile and ask the three questions. Our earlier article on what actually earns an AI recommendation covers the work these files belong to.

Files like these are the checkable layer of generative engine optimization, the part of AI visibility you can inspect yourself before lunch. If your ten minutes turn up a section addressed to nobody, or a bot you never meant to block, bring us the file and we will translate it.

Frequently asked questions

Is llms.txt worth publishing for a small business website?

Yes. The file is a plain-text table of contents for AI tools, listing your important pages. On a well-built site the build generates it with every update, so it stays current on its own, even though studies find AI companies rarely fetch it unprompted. Treat a vendor selling the file by itself, never wired into the site, as a warning sign.

What does a robots.txt file actually do?

It is a note at your site’s front door: a page of plain text listing which visiting programs may read your pages. It can only restrict. A program the file never mentions is already allowed in, so an empty file and no file mean the same thing. A wrong line fails silently, so it is worth knowing what each named section is for.

Will blocking AI bots keep my content out of ChatGPT?

Blocking the wrong bot can remove you from ChatGPT’s answers instead. OpenAI runs GPTBot to gather training material and OAI-SearchBot to power ChatGPT search, and it states that sites which shut out OAI-SearchBot will not be shown in ChatGPT search answers. Decide about each bot by name, not about AI as one thing.

How do I check which AI bots can read my website?

Open your website address followed by /robots.txt and read each section that names a visitor. A bot that is not named and not covered by a general rule is allowed in. Then ask your web person whether anything else on the site is turning away the AI search bots.