View Full Version : Search engine bots

11-12-2002, 05:02 PM
I've been creating a search engine for a while, doint bits and bobs on it when I get time, but I'm thinking of creating a search engine bot for it. The thing is, I don't really know what a bot is. I understand it crawls pages, gets the info and then puts all the info into the db, but how? What is a bot created from? How does it read the text? How much info does it take in from each page?

I don't really know which forum this belongs in either, is it server side programming or some kind of program which runs on the server?

Can you help find tutorials on how to create a bot?

So many questions, so little time :p Wish I'd stop thinking of new ideas and just finish the prgram off

11-12-2002, 08:03 PM