View Full Version : Robots
07-16-2007, 03:23 PM
For my website I have included the code for robots and googlebot and I would like to know if there are any other robots?
07-16-2007, 03:32 PM
There are a lot of robots. Refer to Wikipedia’s robots.txt (http://en.wikipedia.org/robots.txt) to see how many of them are adressed. However, many of them also have no impact on your site.
Also the meta tags for robots don’t have any impact on your site’s indexing or rating. Use a robots.txt file to exclude robots but otherwise your page will be indexed anyway. And ranking depends on the quality of your content and how many links to your site from other trusted sites there are. Those meta tags are nearly irrelevant.
be careful when you use robots.txt files. Some websites would recommend that you include directories and/or files that you don't want web crawlers from accessing.....which obviously informs script kiddies and wanna be hackers...that these folders/files may contain information that might help them in hacking into your website.
Personally, i don't use them at all....as I find them to be a cause for concern more than a helpful solution that supposedly should affect website ranking...which it doesn't. All it suppose to do is to tell web crawlers which files are supposed to be indexed and which are supposed to remove whilst crawling your website etc.
Powered by vBulletin® Version 4.2.2 Copyright © 2016 vBulletin Solutions, Inc. All rights reserved.