Search Engine for Web

[anshul]

I'm a PHP programmer. I'm very much exited and impressed by Google. There lightning speed! I intend to build a search engine just for my study and impressing employers. Simple concept I've is to file_get_contents() a Web page and interpret, use, assimilate htmlentities() of it.

I've questions:
If we scan a Web site per monthly using PHP scripting, how long it takes to analyse and dump results in db ?
What's the approach of Web robots?
Why they've specific user-agent?
If I use a Web browser for my PHP program, Webmasters will see different agents, if I change my browser?
Are they (Google) using C programs?
Google is supposedly using special algorithms; in what language? as I see they ask (employ) for programmers who've specialization in algorithm design.
What language is goog for a well praised demo search engine: C, C++, Perl of PHP or other?

Please discuss this thread.

[anshul]

Friends, I'm surprised why you don't wanna talk about Google :rolleyes:

[mlseim]

I intend to build a search engine just for my study and impressing employers.

1) Search engines are "been there, done that" already. Google, MSN, Ask-Jeeves ... etc. etc ... no need to beat our heads re-designing a search engine.

2) Hard to impress employers when you tell them that the Perl Programmers on CodingForums wrote the script for you. I certainly couldn't write a search engine, but I know others on here could ... but why would they spend the time on this?

3) All of the questions you asked could be answered with a little more research. ;)


Here's one thing ... about API's:
http://www.oetrends.com/news.php?action=view_record&idnum=71

Another Google FAQ:
http://www.geocities.com/googlepubsupgenfaq/

Another:
http://www.sauria.com/~twl/conferences/pycon2005/20050325/Python%20at%20Google.html



.

[anshul]

Hard to impress employers when you tell them that the Perl Programmers on CodingForums wrote the script for you.
No, I just wanted to pick ideas! concepts are the key, I believe ;) Google is supposedly using special algorithms; in what language? as I see they ask (employ) for programmers who've specialization in algorithm design.
What language is goog for a well praised demo search engine: C, C++, Perl of PHP or other? I intend to build a search engine just for my study and impressing employers. Ya, I may be wrong, but, what may I do to let they consider me for employment :D ( pardon! this goes off-topic ./ ).

[netroact]

I have ideas about a specific search engine, but I have put that on the back burner for awhile.

IMO you need to have a specific purpose or aim for your search engine. Occasionally I hear about someone developing a search engine for some purpose, but it never seems to develop into anything. For example, awhile back I read about a new search engine for programmers. They were going to list snippets of code for different languages. I tried it a few times with Perl and there was no results.

Google is still the most relevant, but that's not saying much. I know when I search for Perl snippets they are out there on the web somewhere, but Google doesn't display them in the SERPS. Sometimes, if you dig really deep, you can find an answer.

With the advent of artificial intelligence they are making search engines more user friendly, but IMHO they are all still very much lacking. Maybe MS will have something to offer with their new OS.

Good luck with your engine. I hope you find a more noble purpose than impressing others.

[anshul]

I hope you find a more noble purpose than impressing others. I suppose, neither of above.
Perhaps by making a very simplest example, I hope learn Perl, add it to my skill, and develop some code for others' ( Beginners ) to learn same, I dream ;)

Thank you for your reply and support.

I'm though very small; still I'm working here: Projects for all (http://www.mediasworks.com/project/)