SEO optimized robots txt configuration for blogger



Robots.txt is the configuration text file with couple of commands to take control over web content sharing with web crawlers.

The robots exclusion protocol is created to instruct robots or web crawlers like google-bot and Bingbot for how to access and index pages from particular site.

Generally, for personal website owner(i.e word press) needs to place this file in main or top level directory.

Example:
http://www.blogger.com/robots.txt

Read more about: Best Example ROBOTS.TXT Generated by google.com

For blogger users, By default this files has been placed in root directory. To access it user needs to Enable it manually by change in settings.

Steps to Enable Robots.txt file in blogger

Step 1. Login to your account and go to settings.

blogger setting

Step 2. Click on Edit and configure your commands.

blogger robot setting
Step 3. Click on save changes

blogger setting 1

Optimized ROBOTS.TXT file 

Wrong setting may affect your whole website or even may be ignored by search engines. So it is necessary to know and understand each command and its importance before use.
User-agent: *
Allow: /
Allow: /*?m=*
Allow: /atom.xml?redirect=false&start-index=1&max-results=100000
Allow: /sitemap.xml?page=1
Allow: /sitemap.xml?page=2
Disallow: /search
Disallow: /*?*
Disallow: /*=*
Sitemap: http://www.yourdomain.com/sitemap.xml

Lets Understand each commands

Allow and index everything
User-agent: * 
Allow: /

Allow to index xml sitemap and mobile urls with special characters
Allow: /*?m=*
Allow: /atom.xml?redirect=false&start-index=1&max-results=100000
Allow: /sitemap.xml?page=1
Allow: /sitemap.xml?page=2

Block dynamic urls which contains Special characters "?" and "="
Disallow: /search 
Disallow: /*?* 
Disallow: /*=*

Sitemap information
Sitemap: http://www.yourdomain.com/sitemap.xml

How its SEO optimized
This configuration is specially designed for bloggers only. In blogs there are several urls that can not be changed.

For example: Dynamic comment url structure contains special characters and expires soon. This cause 404 error in webmaster.

Generally 404 Error does not affect overall site ranking but it does make negative impact on users and so on Search engine optimization

Read more about: 5 Custom configuration for SEO in Blogger

Mobile optimized or multi window blogs
Due to increase of smartphone users Google changed policy about optimization. So if you noticed then the google prioritized the search results according to its design.

If any user searching from mobile phone then the search engine will give first priority to multi window optimized webpages.

In case of blocking all Dynamic URL will affect to search engine bots and you may loss mobile users. That is why *m?=* in command will give manual access to search engine even other dynamic urls were blocked!

This will not just make your blog clean but it will decrease the 404 errors and optimize it.

Read more about: Blogger SEO| Does it really exist?

Validation and verification
As mentioned above any mistake will cause your website to be ignored by search engines. Thus it is important to verify it manually.

Login to your google webmaster account and go to Crawl > Robots txt tester

webmaster setting

webmaster setting1

Click on submit to update your settings. To see changes reload the page and verify it.


Webmaster setting2

Place you sitemap and other url then click on Test. Make sure result is ALLOWED


14 comments:
  1. good! this Robots txt file is working.

    ReplyDelete
  2. This is very important trick.. We restrict any web page on our blog from web crawlers so that it can’t get indexed in search engines like blog labels page, demo page or any other pages that are not as important to get indexed. seo companies

    ReplyDelete
    Replies
    1. I have read your blog its very attractive and impressive. I like it your blog.


      SEO Services in India SEO Company in India SEO Company in India

      Guaranteed SEO services Guaranteed SEO

      Delete
  3. I Will Provide SEO Blog Comment Service. & niche relevant blog comment service we provide a quality service with 100% Buyer satisfaction and My team Believed in Quality work not Quantity Contact me Now 24/7.check out this Blog for your business . your business will be gain 200% more profit I can not recommend you guys enough. SEO Service

    ReplyDelete
  4. If you are planning to outsource SEO services from a leading SEO services provider then it is definitely going to cut your expenses with a wide range of SEO services. Adwords Tips

    ReplyDelete
  5. Nice website for learn seo. i like this blog very much. This contant is very amazing

    seo Training in chennai

    ReplyDelete
  6. It is easy for me to understand the issue , if it is clearly-described and contains the main concepts that do not require any questions.

    ReplyDelete
  7. I Will Provide SEO Blog Comment Service . & niche relevant blog comment service we provide a quality service with 100% Buyer satisfaction and My team Believed in Quality work not Quantity Contact me Now 24/7.check out this Blog for your business . your business will be gain 200% more profit I can not recommend you guys enough. SEO Service

    ReplyDelete
  8. All these works are provided by the SEO services in their affordable SEO packages. So, if you want to increase the sales of your company by driving in more traffic to your website, definitely opt for the affordable SEO packages offered by the top-notch SEO companies India.
    what are link pyramids

    ReplyDelete
  9. This content is very useful to me in clarifying the doubts about the different types of strategies should be followed to rank your website on the top of Google search results. This topic is quite interesting and useful to me in promoting my business.
    SEO Training in Chennai | SEO Course in Chennai | SEO Training Chennai

    ReplyDelete
  10. This is very nice blog iam very clear about the stretigies. This will show how the rank will help you to follow your search result.Thanks for sharing.,


    Android Training in Chennai

    ReplyDelete
  11. I Will Provide Seo Blog Comment Service. & niche relevant blog comment service we provide a quality service with 100% Buyer satisfaction and My team Believed in Quality work not Quantity Contact me Now 24/7.

    ReplyDelete