adamsarticles.com adamsarticles.com
   
Home   About Us   Add URL   Add Article
 
Article Categories
 

  Index Page » Computers & Software » Internet Domain Names
   
 

Harnessing the Power of Robots.txt
   
Once we have a website up and running, we need to make sure that all visiting search engines can access all the pages we want them to look at.

Sometimes, we may want search engines to not index certain parts of the site, or even ban other SE from the site all together.

This is where a simple, little 2 line text file called robots.txt comes in.

Robots.txt resides in your websites main directory (on LINUX systems this is your /public_html/ directory), and looks something like the following:

User-agent: *
Disallow:

The first line controls the 'bot' that will be visiting your site, the second line controls if they are allowed in, or which parts of the site they are not allowed to visit'

If you want to handle multiple 'bots', then simple repeat the above lines.
So an example:

User-agent: googlebot
Disallow:

User-agent: askjeeves
Disallow: /

This will allow Goggle (user-agent name GoogleBot) to visit every page and directory, while at the same time banning Ask Jeeves from the site completely.
To find a 'reasonably' up to date list of robot user names this visit http://www.robotstxt.org/wc/active/html/index.html

Even if you want to allow every robot to index every page of your site, it's still very advisable to put a robots.txt file on your site. It will stop your error logs filling up with entries from search engines trying to access your robots.txt file that doesn't exist.

For more information on robots.txt see, the full list of resources about robots.txt at http://www.websitesecrets101.com/robotstxt-further-reading-resources
Author: Bruce Hearder
 
Author Bio:

Bruce Hearder owns and runs www.online-money101.com Signup for the Online-Money101 newsletter and learn the simple techniques that everyday people like yourself use to make money on the web, every single day. Visit www.online-money101.com today
This article can be searched using: free domain names, register domain names, purchase domain names, buy web domain names
 
 
 

Related Articles
 
Harnessing the Power of Robots.txt
 
Remote Monitoring of your Child
 
Yes You Can! How to Write a Report or E-course
 
Write for Humans, Design for Search Engines
 
The Important Tips of Creating Useful Links From Your Site
 
Link Trading Myths Uncovered
 
PHP, Learn Something New
 
Leverage Your Way to Profits With Joint Ventures
 
Guide to Internet Business - Website Setup
 
Free Tips To Increase The Number Of Visitors To Your Website
 
 
 

Related Links
(If your site is closely related to this article, our editors would be happy to add it in this section for free.)
 
Robot.txt checker
Analyze robot.txt file and check the syntax errors in few seconds. Use Siteopsys Robot.txt checker and do your website analysis with ease. Register today its free!!!
 
 
Copyright © 2009, www.adamsarticles.com, All rights reserved. Home | Terms and Conditions | Privacy Policy