robots.txt is a small text file that tells crawlers, the programs search engines and AI companies use
to read the web, which parts of your site they may visit. It's how you let Google in, keep a training crawler out,
or both. Here's how it works and the mistakes that quietly hide a site.
What it looks like
The file lives at the root of your site, at yoursite.com/robots.txt. It's a list of rules, each
starting with the crawler it applies to:
User-agent: *
Disallow: /checkout/
Sitemap: https://meadowbrookbakery.com/sitemap.xml
User-agentnames the crawler.*means every crawler without a rule of its own.Disallowlists a path the crawler should stay out of.Disallow: /means the whole site.Allowopens up a path inside one you've disallowed.Sitemappoints crawlers at the list of your pages.
With no robots.txt at all, crawlers assume they're welcome everywhere. For most small business
websites, that's fine.
Search crawlers and AI crawlers
AI companies run separate crawlers for separate jobs, and each one has its own name you can write a rule for. These are the names the companies publish:
- Google:
Googlebotcrawls for Google Search.Google-Extendedisn't a crawler. It's a name you can disallow to keep your content out of training for Gemini, and Google says it doesn't affect your place in Search. - OpenAI:
OAI-SearchBotfinds sites for ChatGPT's search results.GPTBotcollects content that may be used to train its models.ChatGPT-Uservisits a page when someone asks ChatGPT to. - Anthropic:
Claude-SearchBotimproves Claude's search results.ClaudeBotcollects content that may be used for training.Claude-Uservisits a page when someone asks Claude to. - Perplexity:
PerplexityBotfinds sites for Perplexity's search results, and Perplexity says it isn't used for training.Perplexity-Uservisits a page when someone asks Perplexity to.
The visits people ask for are different. OpenAI and Perplexity both say their user-requested fetchers may not
follow robots.txt, because a person asked for the page.
Staying in search but out of training
If you want AI assistants to find and recommend you, but don't want your words used to train their models, block the training crawlers by name and leave the search crawlers alone:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
Every crawler you don't name, including Googlebot, OAI-SearchBot and
Claude-SearchBot, carries on as before. Crawler names change, so check each company's own page before
you rely on a list, this one included.
What robots.txt can't do
- It doesn't keep a page out of Google. Google says so itself. A
blocked page can still show up in results if other sites link to it. To keep a page out, use a
noindextag or put it behind a password. - It isn't security. The file is public and the rules are requests. Well-behaved crawlers follow them. Anything private belongs behind a login.
- It doesn't describe your business. It only says where crawlers may go. A summary of who you are belongs in llms.txt, and the way your pages look when shared belongs in Open Graph tags.
The mistakes that hide a site
- A leftover
Disallow: /. Sites under construction often block every crawler, and the rule survives the launch. The site works perfectly for visitors and never shows up in search. - Blocking your styles and scripts. Search engines look at your pages the way a visitor does. Block the files that lay the page out and they see a broken page.
- Typos. A crawler ignores a rule it can't understand, so a misspelled line silently does nothing.
How Bloomlight helps
Bloomlight's audits read your robots.txt and flag one that has errors. They also flag any
page that tells search engines not to crawl or list it, so a leftover rule from a rebuild gets caught before it
costs you visitors.