SEO optimized robots txt configuration for blogger



Robots.txt is the configuration text file with couple of commands to take control over web content sharing with web crawlers.

The robots exclusion protocol is created to instruct robots or web crawlers like google-bot and Bingbot for how to access and index pages from particular site.

Generally, for personal website owner(i.e word press) needs to place this file in main or top level directory.

Example:
http://www.blogger.com/robots.txt

Read more about: Best Example ROBOTS.TXT Generated by google.com

For blogger users, By default this files has been placed in root directory. To access it user needs to Enable it manually by change in settings.

Steps to Enable Robots.txt file in blogger

Step 1. Login to your account and go to settings.

blogger setting

Step 2. Click on Edit and configure your commands.

blogger robot setting
Step 3. Click on save changes

blogger setting 1

Optimized ROBOTS.TXT file 

Wrong setting may affect your whole website or even may be ignored by search engines. So it is necessary to know and understand each command and its importance before use.
User-agent: *
Allow: /
Allow: /*?m=*
Allow: /atom.xml?redirect=false&start-index=1&max-results=100000
Allow: /sitemap.xml?page=1
Allow: /sitemap.xml?page=2
Disallow: /search
Disallow: /*?*
Disallow: /*=*
Sitemap: http://www.yourdomain.com/sitemap.xml

Lets Understand each commands

Allow and index everything
User-agent: * 
Allow: /

Allow to index xml sitemap and mobile urls with special characters
Allow: /*?m=*
Allow: /atom.xml?redirect=false&start-index=1&max-results=100000
Allow: /sitemap.xml?page=1
Allow: /sitemap.xml?page=2

Block dynamic urls which contains Special characters "?" and "="
Disallow: /search 
Disallow: /*?* 
Disallow: /*=*

Sitemap information
Sitemap: http://www.yourdomain.com/sitemap.xml

How its SEO optimized
This configuration is specially designed for bloggers only. In blogs there are several urls that can not be changed.

For example: Dynamic comment url structure contains special characters and expires soon. This cause 404 error in webmaster.

Generally 404 Error does not affect overall site ranking but it does make negative impact on users and so on Search engine optimization

Read more about: 5 Custom configuration for SEO in Blogger

Mobile optimized or multi window blogs
Due to increase of smartphone users Google changed policy about optimization. So if you noticed then the google prioritized the search results according to its design.

If any user searching from mobile phone then the search engine will give first priority to multi window optimized webpages.

In case of blocking all Dynamic URL will affect to search engine bots and you may loss mobile users. That is why *m?=* in command will give manual access to search engine even other dynamic urls were blocked!

This will not just make your blog clean but it will decrease the 404 errors and optimize it.

Read more about: Blogger SEO| Does it really exist?

Validation and verification
As mentioned above any mistake will cause your website to be ignored by search engines. Thus it is important to verify it manually.

Login to your google webmaster account and go to Crawl > Robots txt tester

webmaster setting

webmaster setting1

Click on submit to update your settings. To see changes reload the page and verify it.


Webmaster setting2

Place you sitemap and other url then click on Test. Make sure result is ALLOWED


26 comments:
  1. good! this Robots txt file is working.

    ReplyDelete
    Replies
    1. Everyone looking for the best seo service but it is difficult to search on webpages and get your desire date. I truly happy to learn about west palm beach web development which is easy to make your work valuable. Keep it up!

      Delete
  2. This is very important trick.. We restrict any web page on our blog from web crawlers so that it can’t get indexed in search engines like blog labels page, demo page or any other pages that are not as important to get indexed. seo companies

    ReplyDelete
    Replies
    1. I have read your blog its very attractive and impressive. I like it your blog.


      SEO Services in India SEO Company in India SEO Company in India

      Guaranteed SEO services Guaranteed SEO

      Delete
  3. I Will Provide SEO Blog Comment Service. & niche relevant blog comment service we provide a quality service with 100% Buyer satisfaction and My team Believed in Quality work not Quantity Contact me Now 24/7.check out this Blog for your business . your business will be gain 200% more profit I can not recommend you guys enough. SEO Service

    ReplyDelete
  4. If you are planning to outsource SEO services from a leading SEO services provider then it is definitely going to cut your expenses with a wide range of SEO services. Adwords Tips

    ReplyDelete
  5. Nice website for learn seo. i like this blog very much. This contant is very amazing

    seo Training in chennai

    ReplyDelete
  6. It is easy for me to understand the issue , if it is clearly-described and contains the main concepts that do not require any questions.

    ReplyDelete
  7. I Will Provide SEO Blog Comment Service . & niche relevant blog comment service we provide a quality service with 100% Buyer satisfaction and My team Believed in Quality work not Quantity Contact me Now 24/7.check out this Blog for your business . your business will be gain 200% more profit I can not recommend you guys enough. SEO Service

    ReplyDelete
  8. All these works are provided by the SEO services in their affordable SEO packages. So, if you want to increase the sales of your company by driving in more traffic to your website, definitely opt for the affordable SEO packages offered by the top-notch SEO companies India.
    what are link pyramids

    ReplyDelete
  9. This content is very useful to me in clarifying the doubts about the different types of strategies should be followed to rank your website on the top of Google search results. This topic is quite interesting and useful to me in promoting my business.
    SEO Training in Chennai | SEO Course in Chennai | SEO Training Chennai

    ReplyDelete
  10. This is very nice blog iam very clear about the stretigies. This will show how the rank will help you to follow your search result.Thanks for sharing.,


    Android Training in Chennai

    ReplyDelete
  11. I Will Provide Seo Blog Comment Service. & niche relevant blog comment service we provide a quality service with 100% Buyer satisfaction and My team Believed in Quality work not Quantity Contact me Now 24/7.

    ReplyDelete
  12. This is Very Helpful Information. Robot.txt is very important for seo. For Best SEO Service

    ReplyDelete
  13. This comment has been removed by the author.

    ReplyDelete
  14. This comment has been removed by the author.

    ReplyDelete
  15. If any user searching from mobile phone then the search engine will give first priority to multi window optimized webpages.seo services

    ReplyDelete
  16. I went through your blog, such a good and informative things you had shared for us.
    Thank you

    ReplyDelete
  17. Nice post. You pointed on very important facts by this post. This is really very informative and useful information. Thanks for sharing this post.apply digital marketing jobs in hyderabad.

    ReplyDelete
  18. This is very important trick.. We restrict any web page on our blog from web crawlers so that it can’t get indexed in search engines like blog labels page, demo page or any other pages that are not as important to get indexed.

    ReplyDelete
  19. Best SEO Services Company and Agency in Gurgaon. SEO AGENCY IN GURGAON. SEO COMPANY IN GURGAON. BEST SEO COMPANY IN GURGAON. SEO EXPERTS IN GURGAON. SEO PROFESSIONALS IN GURGAON. AFFORDABLE SEO COMPANY and AGENCY IN GURGAON. BEST SEARCH ENGINE OPTIMIZATION COMPANY IN GURGAON. SEARCH ENGINE OPTIMIZATION COMPANY GURGAON. SEO COMPANY GURGAON

    ReplyDelete
  20. Great and nice informative blog post by you. I like your blog post and also visit SEO in Dewas

    ReplyDelete
  21. SEO optimization is a term that most admins know about. The issue is that the vast majority disregard the best SEO advancement strategies in this way bringing about a pointless loss of movement and along these lines income. Read More

    ReplyDelete