Hello and welcome to our community! Is this your first visit?
Register
Enjoy an ad free experience by logging in. Not a member yet? Register.
Results 1 to 10 of 10
  1. #1
    Regular Coder
    Join Date
    May 2011
    Location
    new york
    Posts
    118
    Thanks
    4
    Thanked 0 Times in 0 Posts

    Authentication using PHP Curl

    I'm trying to access AngiesList using curl. From what I understand, you login to the site and it saves a cookie on your computer. With that cookie, the site gives you access to the contents.

    i've done the following.


    PHP Code:
    <?php

    //create a file called cookie.txt and place in site root with perms at 777 first. 


    curl_login("https://my.angieslist.com/AngiesList/Login.aspx""ctl00_AjaxScriptManager_HiddenField=&__LASTFOCUS=&__EVENTTARGET=&__EVENTARGUMENT=&__VIEWSTATE_KEY=85220a6e-a7e4-4172-babb-9ddd5f88da62&ctl00%24ContentPlaceHolderMainContent%24LoginControl%24UserNameTextbox=email%40gmail.com&ctl00%24ContentPlaceHolderMainContent%24LoginControl%24UserPasswordTextbox=comeonin&ctl00%24ContentPlaceHolderMainContent%24LoginControl%24LoginButton=Sign+In&ctl00%24ContentPlaceHolderMainContent%24LoginControl%24userClientTime=Sun%2C+08+Sep+2013+17%3A56%3A51+GMT&ctl00%24ContentPlaceHolderMainContent%24LoginControl%24userClientTimeZoneOffset=420""""false"); 
    echo 
    curl_grab_page("http://deals.angieslist.com/""""false"); 

    function 
    curl_login($url,$data,$proxy,$proxystatus){
        
    $fp fopen("cookie.txt""w"); 
        
    fclose($fp); 
        
    $login curl_init(); 
        
    curl_setopt($loginCURLOPT_COOKIEJAR"cookie.txt"); 
        
    curl_setopt($loginCURLOPT_COOKIEFILE"cookie.txt"); 
        
    curl_setopt($loginCURLOPT_USERAGENT"Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)"); 
        
    curl_setopt($loginCURLOPT_TIMEOUT40); 
        
    curl_setopt($loginCURLOPT_RETURNTRANSFERTRUE); 
        if (
    $proxystatus == 'true') { 
            
    curl_setopt($loginCURLOPT_SSL_VERIFYHOSTFALSE); 
            
    curl_setopt($loginCURLOPT_HTTPPROXYTUNNELTRUE); 
            
    curl_setopt($loginCURLOPT_PROXY$proxy); 
        } 
        
    curl_setopt($loginCURLOPT_URL$url); 
        
    curl_setopt($loginCURLOPT_HEADERTRUE); 
        
    curl_setopt($loginCURLOPT_USERAGENT$SERVER['HTTP_USER_AGENT']); 
        
    curl_setopt($loginCURLOPT_FOLLOWLOCATIONTRUE); 
        
    curl_setopt($loginCURLOPT_POSTTRUE); 
        
    curl_setopt($loginCURLOPT_POSTFIELDS$data); 
        
    ob_start(); //prevent any output 
        
    return curl_exec ($login); //execute the curl command 
        
    ob_end_clean(); //stop preventing output 
        
    curl_close ($login); 
        unset(
    $login); 
    }


    function 
    curl_grab_page($site,$proxy,$proxystatus){ 
        
    $ch curl_init(); 
        
    curl_setopt($chCURLOPT_RETURNTRANSFERTRUE); 
        if (
    $proxystatus == 'true') { 
            
    curl_setopt($loginCURLOPT_SSL_VERIFYHOSTFALSE); 
            
    curl_setopt($loginCURLOPT_HTTPPROXYTUNNELTRUE); 
            
    curl_setopt($loginCURLOPT_PROXY$proxy); 
        } 
        
    curl_setopt($chCURLOPT_COOKIEFILE"cookie.txt"); 
        
    curl_setopt($chCURLOPT_URL$site); 
        
    ob_start(); //prevent any output 
        
    return curl_exec ($ch); //execute the curl command 
        
    ob_end_clean(); //stop preventing output 
        
    curl_close ($ch); 
    }  

    ?>

    but when running the command, I'm not getting the contents of the page in the echo.

    The post data in the first function I got from Tamper, I've attached a photo of what I'm seeing in tamper.

    Anyone have any suggestions?
    Attached Thumbnails Attached Thumbnails Authentication using PHP Curl-tamperang.jpg  

  • #2
    Regular Coder
    Join Date
    May 2011
    Location
    new york
    Posts
    118
    Thanks
    4
    Thanked 0 Times in 0 Posts
    When i run the command

    "php phplogin.php"

    I get

    Code:
    Content-type: text/html
    Is there a way to at least see the html or error that is resulting from the failed curl attempt?

  • #3
    Regular Coder
    Join Date
    May 2011
    Location
    new york
    Posts
    118
    Thanks
    4
    Thanked 0 Times in 0 Posts
    PHP Code:
    <?php
    $ch 
    curl_init ("http://deals.angieslist.com/");
    curl_setopt ($chCURLOPT_COOKIEFILE'cookiecustom.txt');
    curl_setopt ($chCURLOPT_RETURNTRANSFERtrue);
    $output curl_exec ($ch);
    return 
    $output;
    ?>

    I'm having absolutely no luck trying to even just get CURL to output the page contents regardless of whether it successfully logs in.

    for example, I'm trying the above (I've downloaded the cookie file by hand and am attempting to use the cookie to get a successful login) to output the page contents but when I run, I only get

    Code:
     [~/angies-list]# php usingacookie.php
    Content-type: text/html

  • #4
    God Emperor Fou-Lu's Avatar
    Join Date
    Sep 2002
    Location
    Saskatoon, Saskatchewan
    Posts
    16,987
    Thanks
    4
    Thanked 2,660 Times in 2,629 Posts
    That's all I get with that. I don't even get the text/html since I'm just executing directly on the cli.
    You'll need to follow it as well. The output is a 302, not a 200, so you have to follow the link to the next in the chain. Your cookies need to be available in the cookies file, are you sure they're there? Format won't necessarily be the same either, so make sure you have that proper. Add your cookiejar and it should create the file automagically for you.
    PHP Code:
    header('HTTP/1.1 420 Enhance Your Calm'); 

  • #5
    Regular Coder
    Join Date
    May 2011
    Location
    new york
    Posts
    118
    Thanks
    4
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by Fou-Lu View Post
    That's all I get with that. I don't even get the text/html since I'm just executing directly on the cli.
    You'll need to follow it as well. The output is a 302, not a 200, so you have to follow the link to the next in the chain. Your cookies need to be available in the cookies file, are you sure they're there? Format won't necessarily be the same either, so make sure you have that proper. Add your cookiejar and it should create the file automagically for you.
    I'm not exactly sure of what you've just suggested to do.

  • #6
    God Emperor Fou-Lu's Avatar
    Join Date
    Sep 2002
    Location
    Saskatoon, Saskatchewan
    Posts
    16,987
    Thanks
    4
    Thanked 2,660 Times in 2,629 Posts
    You'll need to read the header results, and if its a 300 version status code (302 probably being the most common), you'll need to follow the provided link and do it all over again.
    I have no problem connecting to the site. I have no output since there is none to provide; the http result is a 302 so it's telling me to go elsewhere to get data. You can see this in a browser by simply going to the link, and seeing the link change. That is what the redirection is all about.
    After an exec you can get information using the curl_getinfo function. You should probably be able to bypass that as well and add the redirection curl opts to follow. I don't know for 100% certainty if the curl will automatically follow into a secure domain or not.
    PHP Code:
    header('HTTP/1.1 420 Enhance Your Calm'); 

  • #7
    Regular Coder
    Join Date
    May 2011
    Location
    new york
    Posts
    118
    Thanks
    4
    Thanked 0 Times in 0 Posts
    I've modified the code and am able to get at least an echo of the page contents. however it does not look like it is logging in properly.

    The resulting source is as if the user is still unauthenticated.


    PHP Code:
    <?php 
    $hiddenfield 
    ='';
    $lastfocus='';
    $eventtarget='';
    $eventargument='';
    $viewstate='';
    $login_email 'useremail@gmail.com';
    $login_pass 'passwordgoeshere';
    $loginbutton='';
    $clienttime='';
    $clientoffset='';
     
    $ch curl_init();
    curl_setopt($chCURLOPT_URL'https://my.angieslist.com/AngiesList/Login.aspx');
    curl_setopt($chCURLOPT_POSTFIELDS,'ctl00_AjaxScriptManager_HiddenField=&__LASTFOCUS=&__EVENTTARGET=&__EVENTARGUMENT=&__VIEWSTATE_KEY=85220a6e-a7e4-4172-babb-9ddd5f88da62'.'&ctl00%24ContentPlaceHolderMainContent%24LoginControl%24UserNameTextbox='.urlencode($login_email).'&ctl00%24ContentPlaceHolderMainContent%24LoginControl%24UserPasswordTextbox='.urlencode($login_pass).'&ctl00%24ContentPlaceHolderMainContent%24LoginControl%24LoginButton=Sign+In&ctl00%24ContentPlaceHolderMainContent%24LoginControl%24userClientTime=Wed%2C+11+Sep+2013+18%3A10%3A21+GMT&ctl00%24ContentPlaceHolderMainContent%24LoginControl%24userClientTimeZoneOffset=420');
    curl_setopt($chCURLOPT_POST1);
    curl_setopt($chCURLOPT_HEADER0);
    curl_setopt($chCURLOPT_FOLLOWLOCATION1);
    curl_setopt($chCURLOPT_COOKIEJAR"/home5/usernameic/angies-list/my_cookies.txt");
    curl_setopt($chCURLOPT_COOKIEFILE"/home5/usernameic/angies-list/my_cookies.txt");
    curl_setopt($chCURLOPT_RETURNTRANSFER1);
    curl_setopt($chCURLOPT_USERAGENT"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.3) Gecko/20070309 Firefox/2.0.0.3");
    curl_setopt($chCURLOPT_REFERER"http://www.angieslist.com");
    $page curl_exec($ch);
    echo 
    $page;
    ?>
    I've looked at the past data using tamper data (screenshot is above) and i believe the way I have formatted the post data is correct unless I've done something wrong?

  • #8
    God Emperor Fou-Lu's Avatar
    Join Date
    Sep 2002
    Location
    Saskatoon, Saskatchewan
    Posts
    16,987
    Thanks
    4
    Thanked 2,660 Times in 2,629 Posts
    Your also assuming that:
    1. They don't change their fields per request
    2. They don't use JS to alter fields

    You need to make sure as well that the Login.aspx is the correct processing script. Whatever generates the form itself is of no value, only where it goes. Check your cookies as well to see if they're sticking.
    But all and all, if the post fields are all correct and the processing script is correct, than there is no reason to block the authentication.

    Personally I hate the curl library and avoid it like the plague. Its far more work to do proper at least when you start, but using sockets are far more reliable IMO than using curl.
    PHP Code:
    header('HTTP/1.1 420 Enhance Your Calm'); 

  • #9
    Regular Coder
    Join Date
    May 2011
    Location
    new york
    Posts
    118
    Thanks
    4
    Thanked 0 Times in 0 Posts
    So I was actually finally able to get it working, but after the fact, I'm realizing that the page I'm trying to capture is populated using Javascript. So even if you get the source code of the page, it doesn't include any of the information.

    I'm doing more research and it seems that cURL doesnt handle javascript.

    Does this mean I have to find another way? If so what is the recommendation?
    If not, what do I have to research in order to get this working as is?

  • #10
    God Emperor Fou-Lu's Avatar
    Join Date
    Sep 2002
    Location
    Saskatoon, Saskatchewan
    Posts
    16,987
    Thanks
    4
    Thanked 2,660 Times in 2,629 Posts
    Nope, you'll need to write an ECMA parser if you plan on dealing with the JS. Curl doesn't deal with JS, that's one of the reasons why browsers are so popular :P
    You can ask the owner of the site if they have a developer api instead.
    PHP Code:
    header('HTTP/1.1 420 Enhance Your Calm'); 


  •  

    Posting Permissions

    • You may not post new threads
    • You may not post replies
    • You may not post attachments
    • You may not edit your posts
    •