Robots.txt is a plain text file that Search
Engines use to see if there are areas within
your Website which should not be indexed.
This file
must be placed in your 'www' directory in order
for a search engine to see it.
The
sample below tells search engines not to index
pages in the specified folders.
---- robots.txt (contents follow)
User-agent: *
Disallow: /cgi-bin/
Disallow: /images
Disallow: /secret
Disallow: /members
|