How to Create a robots.txt file in WordPress | Wordpress Robots txt

How to create Wordpress Robots txt File Step by step guide

How to create Robots.txt file for wordpress? A robots.txt file has an important role for the SEO of a WordPress blog. The robots.txt file decides how the search engine bots will crawl your blog / website.

Today I am going to tell how to create robots.txt file is created and updated for wordpress. Because if you make even a small mistake while editing Robots.txt, then your blog will never be indexed in the search engine.

So let's know what a Robots.txt file is and how to create a perfect robots.txt file for WordPress.


 How to Create a robots.txt file in WordPress | Wordpress Robots txt
 How to Create a robots.txt file in WordPress | Wordpress Robots txt

What is Robots.txt?

Whenever search engine bots come to your blog, they follow all the links on that page and crawl and index them. Only after this, your blog and post, pages show in search engine like Google, Bing, Yahoo, Yandex etc.

As its name suggests, robots.txt is a text file that resides in the root directory of the website. Search engine bots follow the rules laid down in Robots.txt to crawl your blog.

In a blog or website, apart from post, there are many other things like pages, category, tags, comments etc. But all these things are not useful for a search engine. 

Generally traffic comes from the search engine on a blog from its main url (https://www.technowap.com), posts, pages or images, apart from these things like archaive, pagination, wp-admin are not necessary for the search engine. Here robots.txt instructs search engine bots not to crawl such unnecessary pages.

If you ever had a mail of index coverage on your Gmail, then you must have seen this kind of massage;

New issue found: Submitted URL blocked by robots.txt

Robots.txt is not allowed to crawl your URL here, that is why that url is blocked for search engine.

That means, which web pages of your blog will show in Google or Bing and what not, it decides the robots.txt file. In this, any mistake can remove your entire blog from the search engine. So new bloggers are afraid to create it themselves.

If you have not yet updated the robots.txt file in the blog, then firstly you understand some of its basic rules and create a perfect seo optimized robots.txt file for your blog.


How to Create WordPress Robots.txt file?

On any wordpress blog you get a default robots.txt file. But for better performace and seo of your blog, you have to customize this robots.txt according to yourself.

Wordpress default robots.txt

User-agent: *
Disallow: / wp-admin /
Allow: /wp-admin/admin-ajax.php

Sitemap: [Blog URL] /sitemap.xml

As you can see above some code / syntax is used for robots.txt. But most of you bloggers use these syntax in their blog without understanding it. First you understand the meaning of these syntax, after that you can create a proper robots.txt code for your blog by yourself.

User-Agent: It is used to give instructions to Search Engines Crawlers / Bots.

User-agent: * This means all search engine bots (Ex: googlebot, bingbot etc.) can crawl your site.

User-agent: googlebot
Only Google bot is allowed to crawl here.

Allow: This tag allows search engine bots to crawl your web pages and folder.

Disallow: This prevents syntax bots from crawl and indexing, so that no one else can access them.

1. If you want to index all the pages and directories of your site. You may have seen this syntax in the Blogger robots.txt file.

User-agent: *
Disallow:

2. But this code will block all the pages and directories of your site from being indexed.

User-agent: *
Disallow: /

3. If you use Adsense, then only use this code. This is for AdSense robots that manage ads.

User-agent: Mediapartners-Google *
Allow: /

Example: If such a robots.txt file is there, let us know what is the meaning of the rules given in it;

User-agent: *
Allow: / wp-content / uploads /
Disallow: / wp-content / plugins /
Disallow: / wp- admin /
Disallow: / archives /
Disallow: / refer /

Sitemap: https://www.example.com/sitemap.xml

Whatever files you upload images from within Wordpress are saved in / wp-content / uploads /. So this code allows the permision to index all i robots.txt file. It is not necessary that you also do the robots.txt code that I use.

User-agent: *
Disallow: / cgi-bin /
Disallow: / wp-admin /
Disallow: / archives /
Disallow: / *? *
Disallow: / comments / feed /
Disallow: / refer /
Disallow: /index.php
Disallow: / wp-content / plugins /

User-agent: Mediapartners-Google *
Allow: /

User-agent: Googlebot-Image
Allow: / wp-content / uploads /

User-agent: Adsbot-Google
Allow: /

User-agent: Googlebot-Mobile
Allow: /

Sitemap: https://www.example.com/sitemap.xml

Note: replace the sitemap of your blog under Sitemap here.


How to update robots.txt file in wordpress

You can manually update the robots.txt file by going to the root directory under wordpress hosting, or there are many plugins available through which you can add to the dashboard of wordpress.

But today I will tell you the easiest way, how you can update robots.txt code in wordpress blog with the help of Yoast SEO plugin.

Step 1: If you have a blog on wordpress, then you must be using Yoast SEO plugin to be seo. First go to Yoast SEO and click on Tools button. Here three tools will be open in front of you, you have to click on File Editor.

Step 2: After clicking on the file editor, a page will open, click on the Create robots.txt file button under the Robots.txt section here.

Step 3: In the box below Next Robots.txt, you remove the default code, paste the robots.txt code given above and click on Save Changes to Robot.txt.

After updating the robots.txt file in WordPress, it is necessary to check whether there is any error in the Search Console through the robots.txt tester tool. This tool will automatically fetch your website's robots.txt file and will show it if there are errors and warnings.

The main goal of your robots.txt file is to prevent the search engine from crawling pages that are not required to be made public. Hope this guide will help you to create a seo optimized robots.txt code.

Conclusion:-

So in this article i have complete guide you How to Create a robots.txt file in WordPress. Check this robots.txt by robots.txt checker. 

If you like this article then Help other bloggers by sharing this post as much as possible.

Also Read This

What is Instagram Reels how to use Instagram Reels?

Best Chrome Extension 2020 | Top 10 to Install in 2020

Post a Comment

0 Comments