Jedi Knight
01-23-2010, 09:08 PM
I'm trying to list all of the spiders that visit my site.
I wrote this array, but don't know if it works or not.
Either it doesn't work, spiders haven't visited my site, or I have the user_agent handles incorrect.
By looking at the code, can anyone see if I've missed anything.
I searched for a list of common crawlers, but haven't found a proper one, yet.
And if anyone knows of a decent list, please point me towards it.
$bot = array("Googlebot","Google Feedfetcher","Google","AdsBot-Google","Baidu","Exabot","Scooter","FAST-WebCrawler","Ask Jeeves/Teoma","Slurp","HenryTheMiragorobot","Lycos_Spider_(T-Rex)","Yahoo","Yahoo!","Yahoo! Slurp China","Yahoo! Slurp","Majestic-12","W3C","SEO Crawler","MSNbot Media","MSN","MSNbot","Gigabot","Alexa");
foreach ($bot as $v) {
if(eregi("$v",$HTTP_USER_AGENT)) {
echo "<span class='botcolor'>".$v."[BOT]</span> ";
}
}
I wrote this array, but don't know if it works or not.
Either it doesn't work, spiders haven't visited my site, or I have the user_agent handles incorrect.
By looking at the code, can anyone see if I've missed anything.
I searched for a list of common crawlers, but haven't found a proper one, yet.
And if anyone knows of a decent list, please point me towards it.
$bot = array("Googlebot","Google Feedfetcher","Google","AdsBot-Google","Baidu","Exabot","Scooter","FAST-WebCrawler","Ask Jeeves/Teoma","Slurp","HenryTheMiragorobot","Lycos_Spider_(T-Rex)","Yahoo","Yahoo!","Yahoo! Slurp China","Yahoo! Slurp","Majestic-12","W3C","SEO Crawler","MSNbot Media","MSN","MSNbot","Gigabot","Alexa");
foreach ($bot as $v) {
if(eregi("$v",$HTTP_USER_AGENT)) {
echo "<span class='botcolor'>".$v."[BOT]</span> ";
}
}