Join Postly
Start free trial
SEOFree · no sign-up

Robots.txt generator

Build a correct robots.txt in a minute. Start from a preset for WordPress, online shops or staging sites, add the paths to block, choose whether AI and SEO-tool crawlers may visit, add your sitemap, then copy or download the file.

Loading the tool…

How to use the Robots.txt Tester & Generator

  1. 1Enter your site to fetch its robots.txt, or switch to Paste and paste a file you're working on.
  2. 2Choose a crawler, or type any user-agent token, and enter the path or full URL to test.
  3. 3Read the verdict and the deciding rule; the table shows the same path for every common crawler.
  4. 4Edit the text to test a change before you upload it, or build a fresh file with the generator.

Frequently asked questions

Where do I put the robots.txt file?

In the root of your domain, so it opens at yoursite.com/robots.txt. Crawlers only look there; a robots.txt in a subfolder is ignored. Each subdomain needs its own file.

What should a basic robots.txt contain?

For most sites: User-agent: * with Disallow lines for admin, cart, internal search and other pages with no search value, plus a Sitemap line with the full URL of your XML sitemap.

Should I block CSS and JavaScript files in robots.txt?

No. Google renders pages with your CSS and JavaScript; blocking them can make pages look broken to Google and hurt rankings. Only block pages and parameters that shouldn't be crawled.

Related free tools

All tools

Then schedule it with Join Postly

Plan, publish and track your posts on every network from one calendar. Free plan, no credit card.

Start for free View pricing
No credit card required Free forever plan Cancel anytime