Join Postly
Start free trial
SEOFree · no sign-up

robots.txt generator for WordPress

Create a robots.txt for a WordPress site: it keeps crawlers out of /wp-admin/ while allowing admin-ajax.php, skips internal search pages, and adds your sitemap. Optionally block AI training crawlers, then test the file against any URL before uploading it.

Loading the tool…

How to use the Robots.txt Tester & Generator

  1. 1Enter your site to fetch its robots.txt, or switch to Paste and paste a file you're working on.
  2. 2Choose a crawler, or type any user-agent token, and enter the path or full URL to test.
  3. 3Read the verdict and the deciding rule; the table shows the same path for every common crawler.
  4. 4Edit the text to test a change before you upload it, or build a fresh file with the generator.

Frequently asked questions

Does WordPress already have a robots.txt?

WordPress serves a virtual robots.txt if there's no real file, usually just blocking /wp-admin/. SEO plugins such as Yoast or Rank Math let you edit it; uploading a robots.txt file to the site root replaces the virtual one.

Should I block /wp-content/ or /wp-includes/ in robots.txt?

No. Google needs your CSS, JavaScript and images, which live there, to render the page properly. Blocking them can hurt how your pages are understood.

Where do I put robots.txt on a WordPress site?

In the site's root folder (the same folder as wp-config.php), so it opens at yoursite.com/robots.txt. Or paste the rules into your SEO plugin's robots.txt editor.

Related free tools

All tools

Then schedule it with Join Postly

Plan, publish and track your posts on every network from one calendar. Free plan, no credit card.

Start for free View pricing
No credit card required Free forever plan Cancel anytime