"Urls restricted by robots.txt" and not 1, 2, 3 all my Labeled Urls restricted by robots.txt means blocking search engine to crawl all my labled url's When is saw it i was amazed, and wondering that i haven't ever created any robots.txt file and how can i do it since my blog is hosted on blogspot where we have lots of limitation however with blogger widget and With the help of articles like "Blogger Hacks" we can over come it but one limitation is there that you can not upload any file(Exclduing Image, Embed video etc) from blogger account. so i searched for about 4 hours in google help, Blogger forums, webmaster forums and i am posting here a short description of all these that i learned. May be you got your all answers for what you searching for mainly for Blogspot bloggers.
First of all "What is Robots.txt" file
Robots.txt file restricts Crawlers Robots access to your site by search engine and blocked those urls listed in robots.txt to listed in search results. This file is mainly for your privacy if you have some pages in your blog which have information which you don't want to listed in search engine you can include url of that pages into robot.txt file.
Robots.txt file restricts Crawlers Robots access to your site by search engine and blocked those urls listed in robots.txt to listed in search results. This file is mainly for your privacy if you have some pages in your blog which have information which you don't want to listed in search engine you can include url of that pages into robot.txt file.
How To Create Robots.txt file in Bloggger
You can generate robot.txt file in blogger using Google Webmaster Tool Just Sign up with your Google account and you use all services free. you can also allow blocked specific files and directories by Generate robots.txt
You can generate robot.txt file in blogger using Google Webmaster Tool Just Sign up with your Google account and you use all services free. you can also allow blocked specific files and directories by Generate robots.txt
To Generate robot.txt using webmaster tool in blogger you must verify your sit or Blog with google. So after sign in Google webmaster tools home page first of all add your site by clicking add a site button and Enter the URL of a site you'd like to manage. for eg www.google.com and click continue On next page choose the Verification method Html or Meta Tag
And Click on varify Button after this
To generate a robots.txt file:
- On the Webmaster Tools Home page, click the site you want.
- Under Site configuration, click Crawler access.
- Click the Generate robots.txt tab.
Why Google Blocked my url
In blogger we specified some labels for our posts when a particular labels are associated with many posts than crawlers find duplicated results in our blogs like You have 5 post labeled with keyword "Movie Review" then for search result for "Movie Review" instead of listing your 5 post of different movie review Google crawlers list a single labeled URL. So it automatically blocked these urls to access in search result for or benefits.
You can also check Following links to More about it (Copy and paste in the browser)
http://www.webmasterworld.com/google/3468726.htm
http://googlewebmastercentral.blogspot.com/2006/09/debugging-blocked-urls_19.html
http://www.google.com/support/webmasters/bin/answer.py?answer=35237
http://www.google.com/support/webmasters/bin/answer.py?answer=40360&ctx=sibling
http://www.google.com/support/webmasters/bin/answer.py?answer=35303&ctx=sibling
http://www.google.com/support/webmasters/bin/answer.py?answer=83098&ctx=sibling
https://www.google.com/webmasters/tools/home?hl=en
thanks buddy for the information
ReplyDeleteThanks for the info...I was thinking there might some issue with the HTML changes i had done to make it more SEO friendly.
ReplyDeleteWhy Can't I Even Copy Paste anything Huh
ReplyDelete