View Full Version : Help! with scraping form results (with cURL?)

09-13-2005, 04:40 PM
I'm not very familiar with cURL, and not sure if it's the best way to do this, but I've been trying it based on what I've read.

Basically, there's a form on a government website that calculates your legislative district that I need to spoof. (perfectly legal, the organization i'm writing this for has permission). You can submit fields with a url, because it's a GET submission. (Something like www.governmentwebsite.com/report?zipcode=55555&city=Austin), so it's easy to submit the form from my site.
But, when you use this URL then the site takes the form info into a CGI program, and somehow redirects you to a page where it spits out the results. The results page has the information I need to get both in the page content AND in the URL, so if I could scrape either info from the page I would have what I need. But how do I do this???

I've read everything I could find about screen scraping, but if I use cURL to scrape this:
it only pulls information from the first page, before I'm redirected to the results page. How do I follow the form submission through a redirect and then pull information from the page I end up at?? Or even, based on the above URL, get the results page URL that I will end up at after getting redirected around?

I really appreciate any help anyone can give me!!! I've been pulling my hair out for days over this problem!!