PDA

View Full Version : do all spyders have 'bot' in their userAgent data

bazz
01-31-2009, 02:26 PM
Pretty much my whole question there.

prompted to ask after reading another thread, I wonder is there a common word or phrase in each bots userAgent data. I want to filter them out of my list / numbers of unique visitors.

bazz
demtron
01-31-2009, 07:43 PM
'Fraid not. How about this for a list to start you off:

http://www.botsvsbrowsers.com/category/1/index.html

I wish it were easier than this. There are like a gazillion bots list here that I've never heard of and some I've found that are not on the list.

Good luck!
bazz
01-31-2009, 07:55 PM
OK, I shall consider finding a way to add those (and more), to a table and make my query check against them, before adding the page visit to my stats or not.

Thanks for the link :thumbsup:

bazz
demtron
02-01-2009, 11:06 PM
Maybe there is a common thread among many of these that you can code so you don't have to suck the whole list into a lookup table. I know many bots embed "Mozilla" and other words in that make them look sort of like human visitors.

You're welcome!
bazz
02-02-2009, 12:18 AM
yep; workin' on it. :)
demtron
02-06-2009, 03:28 PM
I just came across a list of known bot IP addresses at http://iplists.com/ and wanted to let you know about it. Hope your project is coming along.
bazz
02-06-2009, 04:40 PM
Demtron, thank you very much. :thumbsup:

Got sidetracked lately with another thing I am working on but I will get back to it and see how I can make it.

bazz
gsnedders
02-07-2009, 02:16 PM
Maybe there is a common thread among many of these that you can code so you don't have to suck the whole list into a lookup table. I know many bots embed "Mozilla" and other words in that make them look sort of like human visitors.

Mozilla tends to be included in UA strings not to make them look human, but because of sites from the days of the browser wars doing UA sniffing. This led to almost all major UAs claiming to be some version of Mozilla.