What Is The Recommended robots.txt for WordPress?

by

You know that your WordPress website should have a robots.txt file, but what should you put in it? In this quick tutorial, I’ll give you my best recommendation for a standard robots.txt file in WordPress that’s optimized for SEO.

Recommended WordPress Robots.txt

For most WordPress websites, the following robots.txt file should suffice. Simply copy and paste these lines of code into a file named robots.txt in the root of your website.

Note: Please don’t forget to change the URL in the Sitemap lines to the URL of your website, and make sure sitemaps exist at those location. If you are using the Yoast SEO plugin, these sitemaps should exist.

User-Agent: *
Allow: /wp-content/uploads/
Disallow: /wp-content/plugins/
Disallow: /wp-admin/
Disallow: /readme.html
Sitemap: https://tonyteaches.tech/post-sitemap.xml
Sitemap: https://tonyteaches.tech/page-sitemap.xml

For those of you who care, here is an explanation of each setting in the example sitemap above.

  • User-Agent – With an asterisk allows all bots to crawl your website
  • Allow – Bots are explicitly allowed to access URLs with this path
  • Disallow – Bots are not allowed to access URLs with this path
  • Sitemap – The location of a sitemap

For more information on robots.txt standards and best practices, see Google’s documentation on this topic. Otherwise, if you have any question, please let me know in the comments below.

About The Author

With a strong software engineering background, Tony is determined to help as many people as possible start their online busines. Discover why Tony quit his hedge fund job to pursue this mission. You can send Tony a message here.

Leave a Comment