Monday, August 17, 2009

Urls restricted by robots.txt - Crawl errors in webmaster tools

"Urls restricted by robots.txt" and not 1, 2, 3 all my Labeled Urls restricted by robots.txt means blocking search engine to crawl all my labled url's When is saw it i was amazed, and wondering that i haven't ever created any robots.txt file and how can i do it since my blog is hosted on blogspot where we have lots of limitation however with blogger widget and With the help of articles like "Blogger Hacks" we can over come it but one limitation is there that you can not upload any file(Exclduing Image, Embed video etc) from blogger account. so i searched for about 4 hours in google help, Blogger forums, webmaster forums and i am posting here a short description of all these that i learned. May be you got your all answers for what you searching for mainly for Blogspot bloggers.


First of all "What is Robots.txt" file
Robots.txt file restricts Crawlers Robots access to your site by search engine and blocked those urls listed in robots.txt to listed in search results. This file is mainly for your privacy if you have some pages in your blog which have information which you don't want to listed in search engine you can include url of that pages into robot.txt file.

How To Create Robots.txt file in Bloggger
You can generate robot.txt file in blogger using Google Webmaster Tool Just Sign up with your Google account and you use all services free. you can also allow blocked specific files and directories by Generate robots.txt

To Generate robot.txt using webmaster tool in blogger you must verify your sit or Blog with google. So after sign in Google webmaster tools home page first of all add your site by clicking add a site button and Enter the URL of a site you'd like to manage. for eg www.google.com and click continue On next page choose the Verification method Html or Meta Tag


Create the HTML verification file specified below and upload it to the submitted url

Copy the meta tags Generated below , and paste it into your site's home page. After section, before the first section.
And Click on varify Button after this

To generate a robots.txt file:

  1. On the Webmaster Tools Home page, click the site you want.
  2. Under Site configuration, click Crawler access.
  3. Click the Generate robots.txt tab.
for more information go on following link http://www.google.com/support/webmasters/bin/answer.py?answer=83098&ctx=sibling

Why Google Blocked my url
In blogger we specified some labels for our posts when a particular labels are associated with many posts than crawlers find duplicated results in our blogs like You have 5 post labeled with keyword "Movie Review" then for search result for "Movie Review" instead of listing your 5 post of different movie review Google crawlers list a single labeled URL. So it automatically blocked these urls to access in search result for or benefits.

You can also check Following links to More about it (Copy and paste in the browser)
http://www.webmasterworld.com/google/3468726.htm
http://googlewebmastercentral.blogspot.com/2006/09/debugging-blocked-urls_19.html
http://www.google.com/support/webmasters/bin/answer.py?answer=35237
http://www.google.com/support/webmasters/bin/answer.py?answer=40360&ctx=sibling
http://www.google.com/support/webmasters/bin/answer.py?answer=35303&ctx=sibling
http://www.google.com/support/webmasters/bin/answer.py?answer=83098&ctx=sibling
https://www.google.com/webmasters/tools/home?hl=en

Share on Facebook
Share on Twitter
Share on Google+

Related : Urls restricted by robots.txt - Crawl errors in webmaster tools

3 comments:

  1. Anonymous5.11.09

    thanks buddy for the information

    ReplyDelete
  2. Thanks for the info...I was thinking there might some issue with the HTML changes i had done to make it more SEO friendly.

    ReplyDelete
  3. Why Can't I Even Copy Paste anything Huh

    ReplyDelete

3
"Urls restricted by robots.txt" and not 1, 2, 3 all my Labeled Urls restricted by robots.txt means blocking search engine to crawl all my labled url's When is saw it i was amazed, and wondering that i haven't ever created any robots.txt file and how can i do it since my blog is hosted on blogspot where we have lots of limitation however with blogger widget and With the help of articles like "Blogger Hacks" we can over come it but one limitation is there that you can not upload any file(Exclduing Image, Embed video etc) from blogger account. so i searched for about 4 hours in google help, Blogger forums, webmaster forums and i am posting here a short description of all these that i learned. May be you got your all answers for what you searching for mainly for Blogspot bloggers.


First of all "What is Robots.txt" file
Robots.txt file restricts Crawlers Robots access to your site by search engine and blocked those urls listed in robots.txt to listed in search results. This file is mainly for your privacy if you have some pages in your blog which have information which you don't want to listed in search engine you can include url of that pages into robot.txt file.

How To Create Robots.txt file in Bloggger
You can generate robot.txt file in blogger using Google Webmaster Tool Just Sign up with your Google account and you use all services free. you can also allow blocked specific files and directories by Generate robots.txt

To Generate robot.txt using webmaster tool in blogger you must verify your sit or Blog with google. So after sign in Google webmaster tools home page first of all add your site by clicking add a site button and Enter the URL of a site you'd like to manage. for eg www.google.com and click continue On next page choose the Verification method Html or Meta Tag


Create the HTML verification file specified below and upload it to the submitted url

Copy the meta tags Generated below , and paste it into your site's home page. After section, before the first section.
And Click on varify Button after this

To generate a robots.txt file:

  1. On the Webmaster Tools Home page, click the site you want.
  2. Under Site configuration, click Crawler access.
  3. Click the Generate robots.txt tab.
for more information go on following link http://www.google.com/support/webmasters/bin/answer.py?answer=83098&ctx=sibling

Why Google Blocked my url
In blogger we specified some labels for our posts when a particular labels are associated with many posts than crawlers find duplicated results in our blogs like You have 5 post labeled with keyword "Movie Review" then for search result for "Movie Review" instead of listing your 5 post of different movie review Google crawlers list a single labeled URL. So it automatically blocked these urls to access in search result for or benefits.

You can also check Following links to More about it (Copy and paste in the browser)
http://www.webmasterworld.com/google/3468726.htm
http://googlewebmastercentral.blogspot.com/2006/09/debugging-blocked-urls_19.html
http://www.google.com/support/webmasters/bin/answer.py?answer=35237
http://www.google.com/support/webmasters/bin/answer.py?answer=40360&ctx=sibling
http://www.google.com/support/webmasters/bin/answer.py?answer=35303&ctx=sibling
http://www.google.com/support/webmasters/bin/answer.py?answer=83098&ctx=sibling
https://www.google.com/webmasters/tools/home?hl=en

Post a Comment

Anonymous delete 5.11.09

thanks buddy for the information

Thanks for the info...I was thinking there might some issue with the HTML changes i had done to make it more SEO friendly.

Why Can't I Even Copy Paste anything Huh

Dear readers, after reading the Content please ask for advice and to provide constructive feedback Please Write Relevant Comment with Polite Language.Your comments inspired me to continue blogging. Your opinion much more valuable to me. Thank you.