...

View Full Version : Bots / Crawlers / Spiders



Richard
01-15-2003, 11:42 AM
Does anyone know where I can find a list of bots spiders and crawlers, and their hostnames?

pb&j
01-15-2003, 03:21 PM
This may be a starting place...

http://www.philsearch.de/watch.htm

MCookie
01-15-2003, 03:23 PM
Here's one: http://www.psychedelix.com/agents.html
Another interesting page: http://www.searchtools.com/robots/robots-txt.html
And there's BotSpot: http://botspot.com/

MCookie
01-15-2003, 03:34 PM
pb&j - didn't see your post. Cool list :cool:

Richard
01-15-2003, 09:15 PM
Thanks for your replies.

Some of those pages haven't been updated since 1998 or 2000 so not very reliable really :(

krycek
01-16-2003, 01:13 AM
Have a look through this site. All the info you need, ever - and up-to-date! :cool:

http://www.webmasterworld.com/home.htm

::] krycek [::

zoobie
01-16-2003, 05:16 AM
What's really nice about the major SE bots is that they change their alog every 3 months. Enjoy! :D

Stealth=banned

Antoniohawk
01-18-2003, 01:01 AM
wat is a bot, spider, and crawler?



EZ Archive Ads Plugin for vBulletin Copyright 2006 Computer Help Forum