Allow AI crawlers in robots.txt: GPTBot, ClaudeBot and more

If you want AI assistants to find and mention your business, at the very least allow their search crawlers in robots.txt. Whether you also allow crawlers used for training is a separate decision. On top of that, check that your firewall or CDN is not blocking these bots anyway, because that often happens without you knowing.

What is robots.txt?

Robots.txt is a small text file in the root of your website, for example at yourbusiness.com/robots.txt. It tells bots which parts of your site they may visit. Well-behaved bots respect it. It is not security: it is a request.

Training or search: the key difference

AI providers usually run different bots for different purposes:

• Training: the bot collects text to train future models. What you allow here may end up in a model's knowledge later on.
• Search: the bot builds an index so the assistant can find and cite your page when someone asks a question.
• Live fetching: a user asks the assistant something and it fetches a page on the spot.

For visibility in answers, search and live fetching matter most. So blocking training while allowing search is a perfectly sensible choice if you are unsure.

The main bots

At the time of writing, these are the best-known names:

• GPTBot (OpenAI): for training.
• OAI-SearchBot (OpenAI): for search in ChatGPT.
• ChatGPT-User (OpenAI): live fetching on behalf of a user.
• PerplexityBot (Perplexity): for Perplexity's search index.
• Perplexity-User (Perplexity): live fetching on behalf of a user.
• ClaudeBot (Anthropic): for training.
• Claude-SearchBot and Claude-User (Anthropic): for search and live fetching.
• Google-Extended (Google): not a separate crawler, but a setting that tells Google whether it may use your content for Gemini models. According to Google, it does not affect your position in regular search results.

Bot names and what they do can change. Always check the current documentation from OpenAI, Anthropic, Perplexity and Google before you change anything.

An example robots.txt

This example allows search and fetching bots and blocks training bots. If you want to allow training as well, change Disallow: / to Allow: /.

# Search and live fetching: allow
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Claude-SearchBot
User-agent: Claude-User
Allow: /

# Training: block (your choice)
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /

# All other bots
User-agent: *
Disallow: /admin/

Sitemap: https://www.yourbusiness.com/sitemap.xml

Note: a bot usually follows only the group with its own name. If a bot is not listed separately, the User-agent: * group applies.

Check your firewall and CDN too

A good robots.txt is not enough if your hosting still keeps the bots out. Many CDNs, firewalls and security plugins have a setting to block "AI bots" or "unknown bots". Sometimes it is switched on by default.

An online store in Tilburg, for example, had a tidy robots.txt but was never cited as a source. It turned out the security plugin was returning an error to every AI bot. Once that setting was changed, the bots started visiting again.

How to check:

1. Open your robots.txt in a browser and make sure it is correct.
2. Ask your hosting provider or web developer whether bots are being blocked.
3. Look in your server logs for visits from the bots listed above.
4. Review the bot settings in your CDN or firewall.

What robots.txt does not do

Allowing bots does not automatically get you mentioned. It is the first requirement. After that it comes down to clear content, reviews and mentions on other sites. Also read 7 website changes to get mentioned by AI more often and our article on llms.txt.

Can you see the effect?

After a change you want to know whether it pays off. With GEO tracking you see every week whether ChatGPT, Gemini, Perplexity and Claude mention your business. Prefer to start small? The free AI visibility check gives you a first picture in a single report.