...

View Full Version : Should I put this code into a function ?



jeddi
02-22-2010, 07:54 AM
Hi,

I am looping through a table of about 11,000 items
and for each one I am doing a curl operation.

It takes just under 4 hours to run.
( which equates to one every 1.26 second)

Would it be more effiient if I put it inside a function
and called it - or does it not make any difference ?

This is my code:


$sql_url = "SELECT id FROM my_temp ORDER BY id";
$result_url = mysql_query($sql_url) or write_error("Could not SELECT id FROM my_temp ".mysql_error()." \r\n");

$ctr = 1;

while($row_url = mysql_fetch_assoc($result_url)){

$my_code = $row_url['id'];
$target_url = "http://zzzzz.$my_code.example.com/";

$ch = curl_init();
curl_setopt($ch, CURLOPT_USERAGENT, $userAgent);
curl_setopt($ch, CURLOPT_URL,$target_url);
curl_setopt($ch, CURLOPT_FAILONERROR, true);
curl_setopt($ch, CURLOPT_STDERR, $ceh);
curl_setopt($ch, CURLOPT_VERBOSE, 1);
curl_setopt($ch, CURLOPT_HEADER,1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_AUTOREFERER, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_TIMEOUT, 20);

$output = curl_exec($ch);
$info = curl_getinfo($ch);

if ($output === FALSE ) {
write_log("No cURL data returned for $target_url [". $info['http_code']. "]\r\n ");

if (curl_error($ch)) {
write_log($output."CURL error number: curl_errno($ch)\r\n CURL error: curl_error($ch)\r\n");
}
}
else {
$new_url = curl_getinfo($ch,CURLINFO_EFFECTIVE_URL);
$new_url = mysql_real_escape_string($cb_url);

$new_dom = GetDomain($new_url);
$new_dom = mysql_real_escape_string($new_dom);

write_log("LOOK UP URL: $ctr) $target_url: $new_url, $new_dom\r\n");

$sql_temp_url = "UPDATE my_temp SET
url = '$new_url',
dom = '$new_dom'
WHERE id = '$newcode' ";

$result_temp_url = mysql_query($sql_temp_url)
or write_error("Could not UPDATE my_temp url # $ctr.".mysql_error()." \r\n");
write_log("WRITEN URL: $ctr) $cb_code: $new_url, $new_dom\r\n");
}

curl_close($ch);
$ctr++;
}


If the curl should go into a function to make the while loop faster,
I am not sure how much should go in the function.

Would it be just this:


function do_curl($target_url) {
$ch = curl_init();
curl_setopt($ch, CURLOPT_USERAGENT, $userAgent);
curl_setopt($ch, CURLOPT_URL,$target_url);
curl_setopt($ch, CURLOPT_FAILONERROR, true);
curl_setopt($ch, CURLOPT_STDERR, $ceh);
curl_setopt($ch, CURLOPT_VERBOSE, 1);
curl_setopt($ch, CURLOPT_HEADER,1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_AUTOREFERER, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_TIMEOUT, 20);

$output = curl_exec($ch);
return($output);
}

Then call do_curl($target_url); ?

Or should the checking if's go up in the function a well ? :confused:

Would do you recommend to make it most efficient ?

If doing this won't make any difference to the speed of execution, then
for readability, I will leave it as it is. :)

Would appreciate any input.



Thanks.

BubikolRamios
02-22-2010, 08:52 AM
I'm not PHP,this should not take 4 hours, unless



mysql_fetch_assoc($result_url))


does look directly into database each time !?

which would be very bad.

more likely doing this 11000 times is slowing things down


$sql_temp_url = "UPDATE my_temp SET
url = '$new_url',
dom = '$new_dom'
WHERE id = '$newcode' ";

jeddi
02-22-2010, 09:58 AM
Yes,

Good point. I should be building the array with the loop and then
doing the insert in one hit.

What about the cUrl ?

So You think I should leave the cUrl as it is ?

No advantage to put it in a function ?



.



EZ Archive Ads Plugin for vBulletin Copyright 2006 Computer Help Forum