I have been using Google Custom Search to build search engines (MP3/AVI) but i'm tired of using them, so i want to build my own search engine but i always fail in the crawler/spider part. Can anyone point some directions? Good PHP/PERL search engines.
I know it isn't dificult to achieve but i lack of knowledge / logic.
1) visit a site,
2) get the html text,
3) extract the links,
4) If on DB, exclude or delay the index
Thanks for all the help provided