SEO optimized robots txt configuration for blogger



Robots.txt is the configuration text file with couple of commands to take control over web content sharing with web crawlers.

The robots exclusion protocol is created to instruct robots or web crawlers like google-bot and Bingbot for how to access and index pages from particular site.

Generally, for personal website owner(i.e word press) needs to place this file in main or top level directory.

Example:
http://www.blogger.com/robots.txt

Read more about: Best Example ROBOTS.TXT Generated by google.com

For blogger users, By default this files has been placed in root directory. To access it user needs to Enable it manually by change in settings.

Steps to Enable Robots.txt file in blogger

Step 1. Login to your account and go to settings.

blogger setting

Step 2. Click on Edit and configure your commands.

blogger robot setting
Step 3. Click on save changes

blogger setting 1

Optimized ROBOTS.TXT file 

Wrong setting may affect your whole website or even may be ignored by search engines. So it is necessary to know and understand each command and its importance before use.
User-agent: *
Allow: /
Allow: /*?m=*
Allow: /atom.xml?redirect=false&start-index=1&max-results=100000
Allow: /sitemap.xml?page=1
Allow: /sitemap.xml?page=2
Disallow: /search
Disallow: /*?*
Disallow: /*=*
Sitemap: http://www.yourdomain.com/sitemap.xml

Lets Understand each commands

Allow and index everything
User-agent: * 
Allow: /

Allow to index xml sitemap and mobile urls with special characters
Allow: /*?m=*
Allow: /atom.xml?redirect=false&start-index=1&max-results=100000
Allow: /sitemap.xml?page=1
Allow: /sitemap.xml?page=2

Block dynamic urls which contains Special characters "?" and "="
Disallow: /search 
Disallow: /*?* 
Disallow: /*=*

Sitemap information
Sitemap: http://www.yourdomain.com/sitemap.xml

How its SEO optimized
This configuration is specially designed for bloggers only. In blogs there are several urls that can not be changed.

For example: Dynamic comment url structure contains special characters and expires soon. This cause 404 error in webmaster.

Generally 404 Error does not affect overall site ranking but it does make negative impact on users and so on Search engine optimization

Read more about: 5 Custom configuration for SEO in Blogger

Mobile optimized or multi window blogs
Due to increase of smartphone users Google changed policy about optimization. So if you noticed then the google prioritized the search results according to its design.

If any user searching from mobile phone then the search engine will give first priority to multi window optimized webpages.

In case of blocking all Dynamic URL will affect to search engine bots and you may loss mobile users. That is why *m?=* in command will give manual access to search engine even other dynamic urls were blocked!

This will not just make your blog clean but it will decrease the 404 errors and optimize it.

Read more about: Blogger SEO| Does it really exist?

Validation and verification
As mentioned above any mistake will cause your website to be ignored by search engines. Thus it is important to verify it manually.

Login to your google webmaster account and go to Crawl > Robots txt tester

webmaster setting

webmaster setting1

Click on submit to update your settings. To see changes reload the page and verify it.


Webmaster setting2

Place you sitemap and other url then click on Test. Make sure result is ALLOWED


5 comments:
  1. good! this Robots txt file is working.

    ReplyDelete
  2. This is very important trick.. We restrict any web page on our blog from web crawlers so that it can’t get indexed in search engines like blog labels page, demo page or any other pages that are not as important to get indexed. seo companies

    ReplyDelete
  3. I Will Provide SEO Blog Comment Service. & niche relevant blog comment service we provide a quality service with 100% Buyer satisfaction and My team Believed in Quality work not Quantity Contact me Now 24/7.check out this Blog for your business . your business will be gain 200% more profit I can not recommend you guys enough. SEO Service

    ReplyDelete
  4. If you are planning to outsource SEO services from a leading SEO services provider then it is definitely going to cut your expenses with a wide range of SEO services. Adwords Tips

    ReplyDelete