I'm trying to develop my own search engine, which will autimatically crawl the web and index the sites. Does anyone have any idea the main idea behind this method and how it can be achieved? Any software which I can start from etc.