Hello and welcome to our community! Is this your first visit?
Register
Enjoy an ad free experience by logging in. Not a member yet? Register.
Results 1 to 3 of 3
  1. #1
    New to the CF scene
    Join Date
    Nov 2008
    Posts
    6
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Question Imitating a browser on a server

    We're trying to access a URL with a script (currently PHP). Literally every browser is able to access this URL without trouble, but they are trying to protect themselves from spiders and bots, and when we try to read the URL from our script we get "Bad User Agent". The people who own the URL don't seem to be able to figure out how to get us listed as "good user agents"- they are using code provided by an outside vendor and are not the most technically acute people around.

    It's critical that we work around this problem. One approach we're batting around is to find a way to mimic a browser and just read the contents of the page we need to read through a browser window. Can anybody suggest where we might find some example code that would show us how to do this?

  • #2
    New to the CF scene
    Join Date
    Nov 2008
    Location
    Orange County, California
    Posts
    4
    Thanks
    0
    Thanked 1 Time in 1 Post
    Hi -

    I'd recommend using PHP's cURL Library (A lot of great documentation and user-examples available @ php.net). You will need to "spoof" your user agent to look like Internet Explorer or Firefox (or any supported browser)

    Code:
    // spoofing FireFox 2.0
    $useragent="Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1";
    
    // Get curl handle
    $ch = curl_init();
    
    // set user agent
    curl_setopt($ch, CURLOPT_USERAGENT, $useragent);
    Some sites may use more advanced methods of detection to tell if it's an actual request, including the referral header.

    Let me know if it worked out for you!

  • #3
    New to the CF scene
    Join Date
    Nov 2008
    Posts
    6
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Thanks- we'll give it a try!


  •  

    Tags for this Thread

    Posting Permissions

    • You may not post new threads
    • You may not post replies
    • You may not post attachments
    • You may not edit your posts
    •