Bloomlight University · Visibility & SEO

What is robots.txt?

How robots.txt tells search and AI crawlers where they may go, how to stay in search but out of AI training, and the mistakes that hide a site.

robots.txt is a small text file that tells crawlers, the programs search engines and AI companies use to read the web, which parts of your site they may visit. It's how you let Google in, keep a training crawler out, or both. Here's how it works and the mistakes that quietly hide a site.

What it looks like

The file lives at the root of your site, at yoursite.com/robots.txt. It's a list of rules, each starting with the crawler it applies to:

User-agent: *
Disallow: /checkout/

Sitemap: https://meadowbrookbakery.com/sitemap.xml
  • User-agent names the crawler. * means every crawler without a rule of its own.
  • Disallow lists a path the crawler should stay out of. Disallow: / means the whole site.
  • Allow opens up a path inside one you've disallowed.
  • Sitemap points crawlers at the list of your pages.

With no robots.txt at all, crawlers assume they're welcome everywhere. For most small business websites, that's fine.

Search crawlers and AI crawlers

AI companies run separate crawlers for separate jobs, and each one has its own name you can write a rule for. These are the names the companies publish:

  • Google: Googlebot crawls for Google Search. Google-Extended isn't a crawler. It's a name you can disallow to keep your content out of training for Gemini, and Google says it doesn't affect your place in Search.
  • OpenAI: OAI-SearchBot finds sites for ChatGPT's search results. GPTBot collects content that may be used to train its models. ChatGPT-User visits a page when someone asks ChatGPT to.
  • Anthropic: Claude-SearchBot improves Claude's search results. ClaudeBot collects content that may be used for training. Claude-User visits a page when someone asks Claude to.
  • Perplexity: PerplexityBot finds sites for Perplexity's search results, and Perplexity says it isn't used for training. Perplexity-User visits a page when someone asks Perplexity to.

The visits people ask for are different. OpenAI and Perplexity both say their user-requested fetchers may not follow robots.txt, because a person asked for the page.

Staying in search but out of training

If you want AI assistants to find and recommend you, but don't want your words used to train their models, block the training crawlers by name and leave the search crawlers alone:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Every crawler you don't name, including Googlebot, OAI-SearchBot and Claude-SearchBot, carries on as before. Crawler names change, so check each company's own page before you rely on a list, this one included.

What robots.txt can't do

  • It doesn't keep a page out of Google. Google says so itself. A blocked page can still show up in results if other sites link to it. To keep a page out, use a noindex tag or put it behind a password.
  • It isn't security. The file is public and the rules are requests. Well-behaved crawlers follow them. Anything private belongs behind a login.
  • It doesn't describe your business. It only says where crawlers may go. A summary of who you are belongs in llms.txt, and the way your pages look when shared belongs in Open Graph tags.

The mistakes that hide a site

  • A leftover Disallow: /. Sites under construction often block every crawler, and the rule survives the launch. The site works perfectly for visitors and never shows up in search.
  • Blocking your styles and scripts. Search engines look at your pages the way a visitor does. Block the files that lay the page out and they see a broken page.
  • Typos. A crawler ignores a rule it can't understand, so a misspelled line silently does nothing.

How Bloomlight helps

Bloomlight's audits read your robots.txt and flag one that has errors. They also flag any page that tells search engines not to crawl or list it, so a leftover rule from a rebuild gets caught before it costs you visitors.

Start watching your site — free, today.

Two minutes to add your domain. First report in your inbox within the hour.