...

View Full Version : Fatal Error when downloading file that is too big. Is there a work around?



GameOn
12-19-2008, 02:00 PM
I have a cron job scheduled to download a file from someone else's web server each morning. The job points to a php script with this code

#!/usr/bin/php
<?php
$file = file_get_contents("http://www.ZZZZ.com/path/info.txt");
file_put_contents("/myPath/myInfo.txt", $file);
?>

It has worked for smaller files, so I think the error was because this file is so large:
PHP Fatal error: Allowed memory size of 25165824 bytes exhausted (tried to allocate 18721732 bytes) in /home/myPath/myScript.php on line 4

Is there a better way to do this?

Apache version 2.0.63
PHP version 5.2.5

Fou-Lu
12-19-2008, 02:09 PM
Try using standard fread and fwrite techniques for the larger files.
As long as you're unsetting and overwriting the original variable, you shouldn't have a problem with memory usage.
Sadly thats about the only idea I can come up with (at least at this time in the morning ;)) that may work for php, and I can't actually confirm that it will.

Give it a shot, post back results.


Oh yes, up you're timelimt to 10 minutes or so max. 30-60 seconds may not be quite enough when you need to use fread and fwrite.

GameOn
12-19-2008, 05:15 PM
Your idea made reminded me of fputcsv and fgetcsv since I am downloading a csv file. This worked great several times with a file >15MB.


#!/usr/bin/php
<?php
$readfile = fopen("http://www.ZZZZ.com/path/info.csv","r") or die("can't open read file");
$writefile = fopen("myPath/myFile.csv","w") or die("can't open write file");
$line=fgetcsv($readfile);
while(! feof($readfile))
{
fputcsv($writefile,$line,","," ");
$line=fgetcsv($readfile);
}

fclose($readfile);
fclose($writefile);
?>


Oh yes, up you're timelimt to 10 minutes or so max. 30-60 seconds may not be quite enough when you need to use fread and fwrite.
I am not sure how to do this but it seems like a good safety measure so I will look it up and see if it is something I can add.

GameOn
12-19-2008, 05:49 PM
My phpinfo() page is telling me the "max_execution_time = 30" "and safe_mode = off", so I added this line directly below the <?php opening tag...


<?php
set_time_limit(600);

The script still works, but of course I cannot confirm the time limit line because it is not taking that long.

Thanks for the help.

Fou-Lu
12-20-2008, 02:00 AM
You're welcome.
You can test you're timelimit by creating a log file for it. This can actually be done really cheaply (which I won't show you how to do), but its better to use once again an fwrite method:


<?php
$fStart = microtime(true); // PHP 5

// You're stuff.

$fEnd = microtime(true);
if ($fh = fopen('logfile.log', 'a+'))
{
$sEntry = sprintf("%s took %0.5f seconds\n", date("F j Y H:i:s"), ($fEnd - $fStart));
fwrite($fh, $sEntry);
fclose($fh);
}

Something like this. If you're not on PHP5, microtime will need splitting and combining:


function mtime()
{
list($sec, $usec) = explode(' ' , microtime());
return ((float)$sec + (float)$usec);
}

simple as that!
Of course, you can always just echo out the time it took, but on a cron job you can either send that to you're email or log it (which I prefer).



EZ Archive Ads Plugin for vBulletin Copyright 2006 Computer Help Forum