Bots / Crawlers / Spiders

[Richard]

Does anyone know where I can find a list of bots spiders and crawlers, and their hostnames?

[pb&j]

This may be a starting place...

http://www.philsearch.de/watch.htm

[MCookie]

Here's one: http://www.psychedelix.com/agents.html
Another interesting page: http://www.searchtools.com/robots/robots-txt.html
And there's BotSpot: http://botspot.com/

[MCookie]

pb&j - didn't see your post. Cool list :cool:

[Richard]

Thanks for your replies.

Some of those pages haven't been updated since 1998 or 2000 so not very reliable really :(

[krycek]

Have a look through this site. All the info you need, ever - and up-to-date! :cool:

http://www.webmasterworld.com/home.htm

::] krycek [::

[zoobie]

What's really nice about the major SE bots is that they change their alog every 3 months. Enjoy! :D

Stealth=banned

[Antoniohawk]

wat is a bot, spider, and crawler?