Spiders

Richard

01-15-2003, 10:42 AM

Does anyone know where I can find a list of bots spiders and crawlers, and their hostnames?

pb&j

01-15-2003, 02:21 PM

This may be a starting place...

http://www.philsearch.de/watch.htm

MCookie

01-15-2003, 02:23 PM

Here's one: http://www.psychedelix.com/agents.html
Another interesting page: http://www.searchtools.com/robots/robots-txt.html
And there's BotSpot: http://botspot.com/

MCookie

01-15-2003, 02:34 PM

pb&j - didn't see your post. Cool list :cool:

Richard

01-15-2003, 08:15 PM

Thanks for your replies.

Some of those pages haven't been updated since 1998 or 2000 so not very reliable really :(

krycek

01-16-2003, 12:13 AM

Have a look through this site. All the info you need, ever - and up-to-date! :cool:

http://www.webmasterworld.com/home.htm

::] krycek [::

zoobie

01-16-2003, 04:16 AM

What's really nice about the major SE bots is that they change their alog every 3 months. Enjoy! :D

Stealth=banned

Antoniohawk

01-18-2003, 12:01 AM

wat is a bot, spider, and crawler?