Free tools · Free tool
robots.txt checker for AI crawlers
Your robots.txt says which bots may read your website. If you accidentally block the crawlers of ChatGPT, Claude or Perplexity, they cannot use your pages as a source.
What does this mean?
Search and citations. These crawlers decide whether an AI assistant can find your pages and cite them as a source. You almost always want to allow them.
Training. These crawlers collect text to train new models. Blocking them is fine and has no direct effect on search in ChatGPT or Claude.
Partly blocked means some folders or pages are excluded. Check that no important pages are among them.
We only look at your robots.txt. A firewall or security service can also stop bots; you will not see that here.
Want to know whether AI mentions your business?
With the free AI check we ask ChatGPT, Gemini, Perplexity and Claude five customer questions.