Hello and welcome to our community! Is this your first visit?
Register
Enjoy an ad free experience by logging in. Not a member yet? Register.
Results 1 to 2 of 2
  1. #1
    New to the CF scene
    Join Date
    Aug 2010
    Posts
    1
    Thanks
    0
    Thanked 0 Times in 0 Posts

    VB.Net Web Crawler Application

    I am looking for an app to be created that systematically would go through a list of 50000 domains/website records from a SQL server table.

    For each record, determine if live site (record in the SQL table status, and date checked), if so, then web crawl up to three levels deep, recording all Urls/Pages that contain certain keywords/phrases (about 10 different ones to search for, from another SQL table).

    This list of all qualifying urls to be recorded in another SQL table, along with the originating domain name and what keywords triggered the page.

    Probably will need to use something like the webrequest class, and definitely not mshtml, as I believe it downloads all images. Open to suggestion for other class to use.

    Must be Multithreaded (10 threads)

    Also will have up to 10 PCs using the program, all accessing the same list of records, so must be able to work in multi user type scenario, with no conflicts.

    In terms of the form side, just basic display of current domain searching, maybe how many have been done, Urls found. Nothing fancy required though.

    Payment Amount: To be agreed

    Payment by Paypal/Credit Card

  • #2
    New to the CF scene
    Join Date
    Oct 2010
    Posts
    1
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Hi, Please inform if you were able to get this work done ? PM me if you are still looking for a solution to this, I would be able to help you with it.


  •  

    Posting Permissions

    • You may not post new threads
    • You may not post replies
    • You may not post attachments
    • You may not edit your posts
    •