What Is The Recommended robots.txt for WordPress?

Recommended robots.txt file for WordPress

Hey there! Some links on this page may be affiliate links which means that, if you choose to make a purchase, I may earn a small commission at no extra cost to you. I greatly appreciate your support!

You know that your WordPress website should have a robots.txt file, but what should you put in it? In this quick tutorial, I’ll give you my best recommendation for a standard robots.txt file in WordPress that’s optimized for SEO.

Recommended WordPress Robots.txt

For most WordPress websites, the following robots.txt file should suffice. Simply copy and paste these lines of code into a file named robots.txt in the root of your website.

Note: Please don’t forget to change the URL in the Sitemap lines to the URL of your website, and make sure sitemaps exist at those location. If you are using the Yoast SEO plugin, these sitemaps should exist.

User-Agent: *
Allow: /wp-content/uploads/
Disallow: /wp-content/plugins/
Disallow: /wp-admin/
Disallow: /readme.html
Sitemap: https://tonyteaches.tech/post-sitemap.xml
Sitemap: https://tonyteaches.tech/page-sitemap.xml

For those of you who care, here is an explanation of each setting in the example sitemap above.

  • User-Agent – With an asterisk allows all bots to crawl your website
  • Allow – Bots are explicitly allowed to access URLs with this path
  • Disallow – Bots are not allowed to access URLs with this path
  • Sitemap – The location of a sitemap

For more information on robots.txt standards and best practices, see Google’s documentation on this topic. Otherwise, if you have any question, please let me know in the comments below.


Meet Tony

Tony from Tony Teaches Tech headshot

With a strong software engineering background, Tony is determined to demystify the web. Discover why Tony quit his job to pursue this mission. You can join the Tony Teaches Tech community here.

Leave a Reply

Your email address will not be published. Required fields are marked *