It’s no secret that the robots.txt file is an essential element to inform search engine crawlers which content, directories and pages that you want to be crawled and those you do not. MSN provide an informative page with specifics relating to their crawlers; though it can be used as a general guide as well.
You have a choice of using the meta data, which is in the header code of the html page or a robots.txt file which is the text file you can create with notepad and place it in the root directory of your web server. The neat thing about the meta option is that you may not have access to the root of your web-server so you still have control over the pages you do and do not want crawled.
Pagefix Limited, 4 Gull Close, PO13 0RT, GOSPORT, Hampshire, UK
Tel: +44 (0)1329 281386 | Fax: +44 (0)1329 312829 | Email: info@pagefix.co.uk
Copyright © Pagefix Limited 2007 All rights reserved - www.pagefix.co.uk - Search Engine Optimisation
