Articles

How to do a robots.txt for MSN

Find out how to fine tune your robots.txt for MSN’s MSNbot and Picbot

It’s no secret that the robots.txt file is an essential element to inform search engine crawlers which content, directories and pages that you want to be crawled and those you do not. MSN provide an informative page with specifics relating to their crawlers; though it can be used as a general guide as well.

Meta or robots.txt to inform the crawlers?

You have a choice of using the meta data, which is in the header code of the html page or a robots.txt file which is the text file you can create with notepad and place it in the root directory of your web server. The neat thing about the meta option is that you may not have access to the root of your web-server so you still have control over the pages you do and do not want crawled.

Further Reading:

• Control which pages of your website are indexed - MSN
• Visit www.robotstxt.org for more detailed advice about the robots.txt syntax

Tuesday, April 1st, 2008 Posted in Articles | Comments Off