Hello and welcome to our community! Is this your first visit?
Enjoy an ad free experience by logging in. Not a member yet? Register.
Results 1 to 3 of 3
  1. #1
    New to the CF scene
    Join Date
    Jun 2009
    Thanked 0 Times in 0 Posts

    Search Engine URL Scraper

    1) Project Details: (be as specific as possible):
    I am looking for a URL scraper similar to the one mentioned on: http://www.codingforums.com/archive/.../t-156958.html

    Specifically, I need to be able to upload a list of keywords. This can be done via a csv file or just copy and paste into a text field. I need to be able to process multiple keywords at a time, I don't want to do this one by one.

    I need it to scrape the top 100, 250, 500 or 1,000 results (let me pick) in Google, Yahoo, MSN, Bing, Ask (let me pick which ones or all). I see that on the scraper mentioned above, it also included the country type for the search engine (.com, .ca, .co.uk, .cn, .com.au, etc) I'd like this option also for each search engine if possible.

    The top 100 results should be written to the screen, and all results to a csv file that I can download.

    For the output here is what I need:

    1) Keyword phrase used to find the site (since multiple keywords can be searched at once via the upload, I'll need to know which keyword found the site)
    2) Search Engine
    3) Title of website
    4) URL of website
    5) Root URL of website (not sure how hard this would be)

    2) Payment Amount:
    Quote Please

    3) Payment method/ details (Paypal, check? Timeline?):

    As soon as possible

    4) Additional Info (about project or potential bidders):

    I'm new to this forum, but you can see on www.WickedFire.com that I pay on time for projects.

  2. #2
    Join Date
    Jun 2007
    Web Designer
    Thanked 6 Times in 6 Posts
    Are you sure it is not illegal ?

  3. #3
    Senior Coder gnomeontherun's Avatar
    Join Date
    Sep 2007
    Thanked 238 Times in 229 Posts
    Well Google provides an API, I'm sure its not too illegal but might be abusive, depending on what scale this will be implemented.
    jeremy - gnomeontherun
    Educated questions often get educated answers, and simple questions often get simple answers.


Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts