Enjoy an ad free experience by logging in. Not a member yet? Register.
Results 1 to 1 of 1
09-13-2005, 03:40 PM #1
- Join Date
- Sep 2005
- Thanked 0 Times in 0 Posts
Help! with scraping form results (with cURL?)
I'm not very familiar with cURL, and not sure if it's the best way to do this, but I've been trying it based on what I've read.
Basically, there's a form on a government website that calculates your legislative district that I need to spoof. (perfectly legal, the organization i'm writing this for has permission). You can submit fields with a url, because it's a GET submission. (Something like http://www.governmentwebsite.com/rep...5&city=Austin), so it's easy to submit the form from my site.
But, when you use this URL then the site takes the form info into a CGI program, and somehow redirects you to a page where it spits out the results. The results page has the information I need to get both in the page content AND in the URL, so if I could scrape either info from the page I would have what I need. But how do I do this???
I've read everything I could find about screen scraping, but if I use cURL to scrape this:
it only pulls information from the first page, before I'm redirected to the results page. How do I follow the form submission through a redirect and then pull information from the page I end up at?? Or even, based on the above URL, get the results page URL that I will end up at after getting redirected around?
I really appreciate any help anyone can give me!!! I've been pulling my hair out for days over this problem!!