VB.Net Web Crawler Application
 

I am looking for an app to be created that systematically would go through a list of 50000 domains/website records from a SQL server table.

For each record, determine if live site (record in the SQL table status, and date checked), if so, then web crawl up to three levels deep, recording all Urls/Pages that contain certain keywords/phrases (about 10 different ones to search for, from another SQL table).

This list of all qualifying urls to be recorded in another SQL table, along with the originating domain name and what keywords triggered the page.

Probably will need to use something like the webrequest class, and definitely not mshtml, as I believe it downloads all images. Open to suggestion for other class to use.

Must be Multithreaded (10 threads)

Also will have up to 10 PCs using the program, all accessing the same list of records, so must be able to work in multi user type scenario, with no conflicts.

In terms of the form side, just basic display of current domain searching, maybe how many have been done, Urls found. Nothing fancy required though.

Payment Amount: To be agreed

Payment by Paypal/Credit Card

Hi, Please inform if you were able to get this work done ? PM me if you are still looking for a solution to this, I would be able to help you with it.