Hello and welcome to our community! Is this your first visit?
Register
Enjoy an ad free experience by logging in. Not a member yet? Register.
Results 1 to 3 of 3
  1. #1
    Senior Coder
    Join Date
    May 2006
    Posts
    1,673
    Thanks
    28
    Thanked 4 Times in 4 Posts

    Should I put this code into a function ?

    Hi,

    I am looping through a table of about 11,000 items
    and for each one I am doing a curl operation.

    It takes just under 4 hours to run.
    ( which equates to one every 1.26 second)

    Would it be more effiient if I put it inside a function
    and called it - or does it not make any difference ?

    This is my code:

    PHP Code:
    $sql_url "SELECT id FROM my_temp ORDER BY id";
    $result_url mysql_query($sql_url)    or write_error("Could not SELECT id FROM my_temp ".mysql_error()." \r\n");     

    $ctr 1;
     
    while(
    $row_url mysql_fetch_assoc($result_url)){

       
    $my_code $row_url['id'];
       
    $target_url "http://zzzzz.$my_code.example.com/";  
       
       
    $ch curl_init();
       
    curl_setopt($chCURLOPT_USERAGENT$userAgent);
       
    curl_setopt($chCURLOPT_URL,$target_url);
       
    curl_setopt($chCURLOPT_FAILONERRORtrue);
       
    curl_setopt($chCURLOPT_STDERR$ceh);        
       
    curl_setopt($chCURLOPT_VERBOSE1);
       
    curl_setopt($chCURLOPT_HEADER,1); 
       
    curl_setopt($chCURLOPT_FOLLOWLOCATIONtrue);
       
    curl_setopt($chCURLOPT_AUTOREFERERtrue);
       
    curl_setopt($chCURLOPT_RETURNTRANSFERtrue);
       
    curl_setopt($chCURLOPT_TIMEOUT20);

       
    $output curl_exec($ch);
       
    $info curl_getinfo($ch);

       if (
    $output === FALSE ) {
          
    write_log("No cURL data returned for $target_url ["$info['http_code']. "]\r\n ");
                        
          if (
    curl_error($ch))  {
             
    write_log($output."CURL error number: curl_errno($ch)\r\n CURL error: curl_error($ch)\r\n");
            }
          }        
       else {
        
    $new_url curl_getinfo($ch,CURLINFO_EFFECTIVE_URL);
        
    $new_url mysql_real_escape_string($cb_url); 
                        
        
    $new_dom GetDomain($new_url);
        
    $new_dom mysql_real_escape_string($new_dom);
     
        
    write_log("LOOK UP URL: $ctr) $target_url: $new_url, $new_dom\r\n");

        
    $sql_temp_url "UPDATE my_temp SET    
        url = '$new_url',    
        dom = '$new_dom'
        WHERE id = '$newcode' "
    ;

        
    $result_temp_url mysql_query($sql_temp_url)
               or 
    write_error("Could not UPDATE my_temp url # $ctr.".mysql_error()." \r\n");         
        
    write_log("WRITEN URL: $ctr) $cb_code: $new_url, $new_dom\r\n");
        }

        
    curl_close($ch);
     
    $ctr++; 
     } 
    If the curl should go into a function to make the while loop faster,
    I am not sure how much should go in the function.

    Would it be just this:

    PHP Code:
    function do_curl($target_url) {
       
    $ch curl_init();
       
    curl_setopt($chCURLOPT_USERAGENT$userAgent);
       
    curl_setopt($chCURLOPT_URL,$target_url);
       
    curl_setopt($chCURLOPT_FAILONERRORtrue);
       
    curl_setopt($chCURLOPT_STDERR$ceh);        
       
    curl_setopt($chCURLOPT_VERBOSE1);
       
    curl_setopt($chCURLOPT_HEADER,1); 
       
    curl_setopt($chCURLOPT_FOLLOWLOCATIONtrue);
       
    curl_setopt($chCURLOPT_AUTOREFERERtrue);
       
    curl_setopt($chCURLOPT_RETURNTRANSFERtrue);
       
    curl_setopt($chCURLOPT_TIMEOUT20);

       
    $output curl_exec($ch);
       return(
    $output);

    Then call do_curl($target_url); ?

    Or should the checking if's go up in the function a well ?

    Would do you recommend to make it most efficient ?

    If doing this won't make any difference to the speed of execution, then
    for readability, I will leave it as it is.

    Would appreciate any input.



    Thanks.
    Last edited by jeddi; 02-22-2010 at 07:15 AM.
    If you want to attract and keep more clients, then offer great customer support.

    Support-Focus.com. automates the process and gives you a trust seal to place on your website.
    I recommend that you at least take the 30 day free trial.

  • #2
    Senior Coder
    Join Date
    Dec 2005
    Location
    Slovenia
    Posts
    1,960
    Thanks
    120
    Thanked 76 Times in 76 Posts
    I'm not PHP,this should not take 4 hours, unless

    Code:
    mysql_fetch_assoc($result_url))
    does look directly into database each time !?

    which would be very bad.

    more likely doing this 11000 times is slowing things down
    Code:
        $sql_temp_url = "UPDATE my_temp SET    
        url = '$new_url',    
        dom = '$new_dom'
        WHERE id = '$newcode' ";
    Last edited by BubikolRamios; 02-22-2010 at 07:58 AM.
    Found a flower or bug and don't know what it is ?
    agrozoo.net galery
    if you don't spot search button at once, there is search form:
    agrozoo.net galery search

  • #3
    Senior Coder
    Join Date
    May 2006
    Posts
    1,673
    Thanks
    28
    Thanked 4 Times in 4 Posts
    Yes,

    Good point. I should be building the array with the loop and then
    doing the insert in one hit.

    What about the cUrl ?

    So You think I should leave the cUrl as it is ?

    No advantage to put it in a function ?



    .
    If you want to attract and keep more clients, then offer great customer support.

    Support-Focus.com. automates the process and gives you a trust seal to place on your website.
    I recommend that you at least take the 30 day free trial.


  •  

    Posting Permissions

    • You may not post new threads
    • You may not post replies
    • You may not post attachments
    • You may not edit your posts
    •