AI crawler access checker
See which AI crawlers may read a page: GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Applebot-Extended, Bytespider and more. Each is tested against your robots.txt rules, robots meta tags and X-Robots-Tag header, with search and training bots told apart.
How to use the AI Crawler Access Checker
- 1Enter a page address; the check uses that site's robots.txt and the page's own robots tags.
- 2Read each crawler's verdict and the robots.txt line that decides it.
- 3Decide what you want: most sites allow AI search bots (citations) and choose freely on training bots.
- 4Update robots.txt with the generator's AI presets, then check again.
Frequently asked questions
What is the difference between AI search and AI training crawlers?
Search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot fetch pages to answer questions and link to sources. Training crawlers such as GPTBot, ClaudeBot and CCBot collect pages that may be used to train models.
Should I block AI crawlers?
It's a business choice. Blocking training crawlers keeps your content out of future model training. Blocking AI search crawlers also removes you from those assistants' answers and their links to your site.
Can AI crawlers be blocked by my firewall even if robots.txt allows them?
Yes. Bot protection and settings such as Cloudflare's AI bot blocking stop AI crawlers at the network level. This check reads your rules, so also review your CDN or firewall settings.
Related free tools
All toolsThen schedule it with Join Postly
Plan, publish and track your posts on every network from one calendar. Free plan, no credit card.