Join Postly
Start free trial
SEOFree · no sign-up

Block AI crawlers with robots.txt

Choose which AI bots may read your site. This generator starts with a preset that blocks AI training crawlers such as GPTBot, ClaudeBot, CCBot and Google-Extended while keeping AI search bots allowed, or blocks every AI crawler. Copy the rules into your robots.txt.

Loading the tool…

How to use the Robots.txt Tester & Generator

  1. 1Enter your site to fetch its robots.txt, or switch to Paste and paste a file you're working on.
  2. 2Choose a crawler, or type any user-agent token, and enter the path or full URL to test.
  3. 3Read the verdict and the deciding rule; the table shows the same path for every common crawler.
  4. 4Edit the text to test a change before you upload it, or build a fresh file with the generator.

Frequently asked questions

Does blocking GPTBot remove my site from ChatGPT?

Blocking GPTBot stops OpenAI from collecting your pages for training. ChatGPT search uses OAI-SearchBot instead, so block that too if you don't want to appear in ChatGPT search results.

Will blocking Google-Extended hurt my Google rankings?

No. Google-Extended only controls whether your content is used for Gemini and other Google AI models. Google Search and AI Overviews use Googlebot, which isn't affected.

Do all AI companies respect robots.txt?

The major ones (OpenAI, Anthropic, Google, Apple, Perplexity, Common Crawl) say their crawlers follow robots.txt. Less careful scrapers may ignore it; firewall rules or bot protection can block those.

Related free tools

All tools

Then schedule it with Join Postly

Plan, publish and track your posts on every network from one calendar. Free plan, no credit card.

Start for free View pricing
No credit card required Free forever plan Cancel anytime