View Full Version : Robots

07-16-2007, 03:23 PM
For my website I have included the code for robots and googlebot and I would like to know if there are any other robots?

07-16-2007, 03:32 PM
There are a lot of robots. Refer to Wikipediaís robots.txt (http://en.wikipedia.org/robots.txt) to see how many of them are adressed. However, many of them also have no impact on your site.

Also the meta tags for robots donít have any impact on your siteís indexing or rating. Use a robots.txt file to exclude robots but otherwise your page will be indexed anyway. And ranking depends on the quality of your content and how many links to your site from other trusted sites there are. Those meta tags are nearly irrelevant.

07-16-2007, 04:58 PM
be careful when you use robots.txt files. Some websites would recommend that you include directories and/or files that you don't want web crawlers from accessing.....which obviously informs script kiddies and wanna be hackers...that these folders/files may contain information that might help them in hacking into your website.

Personally, i don't use them at all....as I find them to be a cause for concern more than a helpful solution that supposedly should affect website ranking...which it doesn't. All it suppose to do is to tell web crawlers which files are supposed to be indexed and which are supposed to remove whilst crawling your website etc.