View Full Version : Bots / Crawlers / Spiders
01-15-2003, 10:42 AM
Does anyone know where I can find a list of bots spiders and crawlers, and their hostnames?
This may be a starting place...
01-15-2003, 02:23 PM
Here's one: http://www.psychedelix.com/agents.html
Another interesting page: http://www.searchtools.com/robots/robots-txt.html
And there's BotSpot: http://botspot.com/
01-15-2003, 02:34 PM
pb&j - didn't see your post. Cool list :cool:
01-15-2003, 08:15 PM
Thanks for your replies.
Some of those pages haven't been updated since 1998 or 2000 so not very reliable really :(
01-16-2003, 12:13 AM
Have a look through this site. All the info you need, ever - and up-to-date! :cool:
::] krycek [::
01-16-2003, 04:16 AM
What's really nice about the major SE bots is that they change their alog every 3 months. Enjoy! :D
01-18-2003, 12:01 AM
wat is a bot, spider, and crawler?
Powered by vBulletin® Version 4.2.2 Copyright © 2017 vBulletin Solutions, Inc. All rights reserved.