electrify77
01-28-2007, 11:35 PM
I was not sure exactly where is the most appropriate to post this but since I'm doing it in Perl I thought I'd ask here...
OK so Google Search API is no longer available but I still need to use Google's search results and the AJAX API they have is totally not going to do the job. So I was thinking of scrapping the search results (it's for my personal use). Now I know GG doesn't like this and they can ban my IP so I was wondering if anyone has any particular observations as to how often I can safely scrap their pages without getting on their radar? Also does it analyze the user agent of where the query came from - in other words should I fake that to be the same as a real browser or I can just leave it as is.
OK so Google Search API is no longer available but I still need to use Google's search results and the AJAX API they have is totally not going to do the job. So I was thinking of scrapping the search results (it's for my personal use). Now I know GG doesn't like this and they can ban my IP so I was wondering if anyone has any particular observations as to how often I can safely scrap their pages without getting on their radar? Also does it analyze the user agent of where the query came from - in other words should I fake that to be the same as a real browser or I can just leave it as is.