Robots.txt tester
Fetch any site's robots.txt or paste your own, pick a crawler such as Googlebot, Bingbot or GPTBot, and test a URL. You see whether it's allowed or blocked, the exact rule and line that decided it, and warnings about lines search engines ignore.
How to use the Robots.txt Tester & Generator
- 1Enter your site to fetch its robots.txt, or switch to Paste and paste a file you're working on.
- 2Choose a crawler, or type any user-agent token, and enter the path or full URL to test.
- 3Read the verdict and the deciding rule; the table shows the same path for every common crawler.
- 4Edit the text to test a change before you upload it, or build a fresh file with the generator.
Frequently asked questions
How does a crawler decide which robots.txt rule applies?
It follows the group for its own user-agent name (the most specific match), or the * group if it has none. Within that group the longest matching path wins, and when an Allow and a Disallow are equally long, Allow wins.
Does Disallow in robots.txt remove a page from Google?
No. Disallow stops crawling, not indexing: Google can still list a blocked URL if other pages link to it, just without a description. To keep a page out of search, allow crawling and add a noindex tag.
What do * and $ mean in robots.txt?
* matches any run of characters and $ marks the end of the URL. For example, Disallow: /*.pdf$ blocks every URL ending in .pdf, and Disallow: /*?sort= blocks URLs with a sort parameter.
Is robots.txt case-sensitive?
The paths are: /Admin and /admin are different rules. The field names (User-agent, Disallow) and the bot names are not case-sensitive.
Related free tools
All tools- Robots.txt generatorRobots.txt generator
- Block AI crawlers in robots.txtBlock AI crawlers with robots.txt
- AI Crawler Access CheckerIs your site open to GPTBot, ClaudeBot, PerplexityBot and others?
- XML Sitemap GeneratorCrawl up to 500 pages and download a ready-to-submit sitemap.xml.
- Meta Robots Tag Generatornoindex, nofollow and snippet rules as tags or headers.
- Website SEO ScoreA free mini audit of one page with the Site Audit's checks and fixes.
Then schedule it with Join Postly
Plan, publish and track your posts on every network from one calendar. Free plan, no credit card.