 |
What does a bad request for "robots.txt" mean?
Search engines have indexing robots which look for the robots.txt file located at the root of your
web file system. This indicates to robots which files to exclude from the search by the robot. This is not a
bad sign. It just means that a search engine robot has attempted to crawl your site.
For further information, see http://www.robotstxt.org/
|
 |