View Full Version : Imitating a browser on a server

11-11-2008, 05:56 AM
We're trying to access a URL with a script (currently PHP). Literally every browser is able to access this URL without trouble, but they are trying to protect themselves from spiders and bots, and when we try to read the URL from our script we get "Bad User Agent". The people who own the URL don't seem to be able to figure out how to get us listed as "good user agents"- they are using code provided by an outside vendor and are not the most technically acute people around.

It's critical that we work around this problem. One approach we're batting around is to find a way to mimic a browser and just read the contents of the page we need to read through a browser window. Can anybody suggest where we might find some example code that would show us how to do this?

11-11-2008, 08:34 AM
Hi -

I'd recommend using PHP's cURL Library (A lot of great documentation and user-examples available @ php.net). You will need to "spoof" your user agent to look like Internet Explorer or Firefox (or any supported browser)

// spoofing FireFox 2.0
$useragent="Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv: Gecko/20061204 Firefox/";

// Get curl handle
$ch = curl_init();

// set user agent
curl_setopt($ch, CURLOPT_USERAGENT, $useragent);

Some sites may use more advanced methods of detection to tell if it's an actual request, including the referral header.

Let me know if it worked out for you!

11-11-2008, 06:56 PM
Thanks- we'll give it a try!