SEO optimized robots txt configuration for blogger



Robots.txt is the configuration text file with couple of commands to take control over web content sharing with web crawlers.

The robots exclusion protocol is created to instruct robots or web crawlers like google-bot and Bingbot for how to access and index pages from particular site.

Generally, for personal website owner(i.e word press) needs to place this file in main or top level directory.

Example:
http://www.blogger.com/robots.txt

Read more about: Best Example ROBOTS.TXT Generated by google.com

For blogger users, By default this files has been placed in root directory. To access it user needs to Enable it manually by change in settings.

Steps to Enable Robots.txt file in blogger

Step 1. Login to your account and go to settings.

blogger setting

Step 2. Click on Edit and configure your commands.

blogger robot setting
Step 3. Click on save changes

blogger setting 1

Optimized ROBOTS.TXT file 

Wrong setting may affect your whole website or even may be ignored by search engines. So it is necessary to know and understand each command and its importance before use.
User-agent: *
Allow: /
Allow: /*?m=*
Allow: /atom.xml?redirect=false&start-index=1&max-results=100000
Allow: /sitemap.xml?page=1
Allow: /sitemap.xml?page=2
Disallow: /search
Disallow: /*?*
Disallow: /*=*
Sitemap: http://www.yourdomain.com/sitemap.xml

Lets Understand each commands

Allow and index everything
User-agent: * 
Allow: /

Allow to index xml sitemap and mobile urls with special characters
Allow: /*?m=*
Allow: /atom.xml?redirect=false&start-index=1&max-results=100000
Allow: /sitemap.xml?page=1
Allow: /sitemap.xml?page=2

Block dynamic urls which contains Special characters "?" and "="
Disallow: /search 
Disallow: /*?* 
Disallow: /*=*

Sitemap information
Sitemap: http://www.yourdomain.com/sitemap.xml

How its SEO optimized
This configuration is specially designed for bloggers only. In blogs there are several urls that can not be changed.

For example: Dynamic comment url structure contains special characters and expires soon. This cause 404 error in webmaster.

Generally 404 Error does not affect overall site ranking but it does make negative impact on users and so on Search engine optimization

Read more about: 5 Custom configuration for SEO in Blogger

Mobile optimized or multi window blogs
Due to increase of smartphone users Google changed policy about optimization. So if you noticed then the google prioritized the search results according to its design.

If any user searching from mobile phone then the search engine will give first priority to multi window optimized webpages.

In case of blocking all Dynamic URL will affect to search engine bots and you may loss mobile users. That is why *m?=* in command will give manual access to search engine even other dynamic urls were blocked!

This will not just make your blog clean but it will decrease the 404 errors and optimize it.

Read more about: Blogger SEO| Does it really exist?

Validation and verification
As mentioned above any mistake will cause your website to be ignored by search engines. Thus it is important to verify it manually.

Login to your google webmaster account and go to Crawl > Robots txt tester

webmaster setting

webmaster setting1

Click on submit to update your settings. To see changes reload the page and verify it.


Webmaster setting2

Place you sitemap and other url then click on Test. Make sure result is ALLOWED


46 comments:
  1. good! this Robots txt file is working.

    ReplyDelete
    Replies
    1. Everyone looking for the best seo service but it is difficult to search on webpages and get your desire date. I truly happy to learn about west palm beach web development which is easy to make your work valuable. Keep it up!

      Delete
    2. Gexton are the Best seo company in Hyderabad.
      For Further Detail: www.gexton.com





      Delete
    3. Gexton are the Best seo company in Hyderabad.
      For Further Detail: www.gexton.com





      Delete
  2. This is very important trick.. We restrict any web page on our blog from web crawlers so that it can’t get indexed in search engines like blog labels page, demo page or any other pages that are not as important to get indexed. seo companies

    ReplyDelete
    Replies
    1. I have read your blog its very attractive and impressive. I like it your blog.


      SEO Services in India SEO Company in India SEO Company in India

      Guaranteed SEO services Guaranteed SEO

      Delete
  3. I Will Provide SEO Blog Comment Service. & niche relevant blog comment service we provide a quality service with 100% Buyer satisfaction and My team Believed in Quality work not Quantity Contact me Now 24/7.check out this Blog for your business . your business will be gain 200% more profit I can not recommend you guys enough. SEO Service

    ReplyDelete
  4. If you are planning to outsource SEO services from a leading SEO services provider then it is definitely going to cut your expenses with a wide range of SEO services. Adwords Tips

    ReplyDelete
  5. Nice website for learn seo. i like this blog very much. This contant is very amazing

    seo Training in chennai

    ReplyDelete
  6. It is easy for me to understand the issue , if it is clearly-described and contains the main concepts that do not require any questions.

    ReplyDelete
  7. I Will Provide SEO Blog Comment Service . & niche relevant blog comment service we provide a quality service with 100% Buyer satisfaction and My team Believed in Quality work not Quantity Contact me Now 24/7.check out this Blog for your business . your business will be gain 200% more profit I can not recommend you guys enough. SEO Service

    ReplyDelete
  8. All these works are provided by the SEO services in their affordable SEO packages. So, if you want to increase the sales of your company by driving in more traffic to your website, definitely opt for the affordable SEO packages offered by the top-notch SEO companies India.
    what are link pyramids

    ReplyDelete
  9. This content is very useful to me in clarifying the doubts about the different types of strategies should be followed to rank your website on the top of Google search results. This topic is quite interesting and useful to me in promoting my business.
    SEO Training in Chennai | SEO Course in Chennai | SEO Training Chennai

    ReplyDelete
  10. This is very nice blog iam very clear about the stretigies. This will show how the rank will help you to follow your search result.Thanks for sharing.,


    Android Training in Chennai

    ReplyDelete
  11. I Will Provide Seo Blog Comment Service. & niche relevant blog comment service we provide a quality service with 100% Buyer satisfaction and My team Believed in Quality work not Quantity Contact me Now 24/7.

    ReplyDelete
  12. This is Very Helpful Information. Robot.txt is very important for seo. For Best SEO Service

    ReplyDelete
  13. This comment has been removed by the author.

    ReplyDelete
  14. This comment has been removed by the author.

    ReplyDelete
  15. If any user searching from mobile phone then the search engine will give first priority to multi window optimized webpages.seo services

    ReplyDelete
  16. I went through your blog, such a good and informative things you had shared for us.
    Thank you

    ReplyDelete
  17. Nice post. You pointed on very important facts by this post. This is really very informative and useful information. Thanks for sharing this post.apply digital marketing jobs in hyderabad.

    ReplyDelete
  18. This is very important trick.. We restrict any web page on our blog from web crawlers so that it can’t get indexed in search engines like blog labels page, demo page or any other pages that are not as important to get indexed.

    ReplyDelete
  19. Best SEO Services Company and Agency in Gurgaon. SEO AGENCY IN GURGAON. SEO COMPANY IN GURGAON. BEST SEO COMPANY IN GURGAON. SEO EXPERTS IN GURGAON. SEO PROFESSIONALS IN GURGAON. AFFORDABLE SEO COMPANY and AGENCY IN GURGAON. BEST SEARCH ENGINE OPTIMIZATION COMPANY IN GURGAON. SEARCH ENGINE OPTIMIZATION COMPANY GURGAON. SEO COMPANY GURGAON

    ReplyDelete
  20. Great and nice informative blog post by you. I like your blog post and also visit SEO in Dewas

    ReplyDelete
  21. SEO optimization is a term that most admins know about. The issue is that the vast majority disregard the best SEO advancement strategies in this way bringing about a pointless loss of movement and along these lines income. Read More

    ReplyDelete
  22. Its a fabulous blog i have ever read in past days. search jobs near by me in hyderabad.

    ReplyDelete
  23. Web Honkers is Canada’s leading digital marketing firm, with top rated SEO Experts who are committed to our client’s success with offices in Toronto, Canada and in London, UK.
    Digital Marketing Firm

    ReplyDelete
  24. Its a fabulous blog i have ever read in past days. search jobs near by you in chennai.

    ReplyDelete
  25. Thanks for sharing nice information about gurgaon. search jobs near by you in gurgaon.

    ReplyDelete
  26. The sites that as of now have the best spots in the web crawlers for your coveted catchphrases are your online rivalry,rank tracker

    ReplyDelete
  27. Perpetually logical and an outstanding contribution to the world of bloggers.
    authority backlinks

    ReplyDelete
  28. Informative post! I really like and appreciate your work, thank you for sharing such a useful information about knowledge management system strategies, keep updating the information, hear i prefer some more information about jobs for your career hr jobs in hyderabad .

    ReplyDelete
  29. These articles have got thorough discernment without unclear the readers.internet marketing in Michigan

    ReplyDelete
  30. This comment has been removed by the author.

    ReplyDelete
  31. For any HTML web page being served from CDN, it is good to have canonical headers brought because as tons as google don’t care for canonicals on images, it honours same header on HTML files so, we can use the rel="canonical" header to suggest the original supply of a web page (rel="canonical" can paintings inside HTML tags and as a separate HTTP header). Crawlers that try and index a record from the CDN will see the canonical URL and store that, improving your search engine optimization. dofollow blog commenting services

    ReplyDelete
  32. I cannot explain the feelings I had while reading this blog. I was completely fascinated.Johnny Chen Houston

    ReplyDelete
  33. Amazing!

    Please visit Never Miss Out for the latest news on sports, business, technology, insurance and many more.....

    ReplyDelete
  34. Thnaks For posting such as awesome article which give me more information on Robots.txt file.

    Seo Training

    ReplyDelete
  35. Doubtlessly there are practically unending methods for doing this. In this article, I will impart to you how to get saw by getting related and relative backlinks from a gathering. manual backlinks

    ReplyDelete
  36. I just wanted to let you know that what you do really affects peoples lives and that people - like me - truly appreciate it.
    website traffic generator

    ReplyDelete
  37. A blend wit and momentous words. Exactly what I was looking for.softwaretesting

    ReplyDelete