...

View Full Version : How do I download a file and unzip it with curl ?



jeddi
10-26-2009, 07:34 AM
Hello,

What I want to do is download file like this:
www.support-focus.com/test_map.zip

This is my sitemap zipped up and I want to use this as a test.

I am going to run the php script on a different server.

test_map is an xml file so what I want to do is download
the test_map.zip, unzip it and rename it to sitemap.xml

When I googled this, I found reference to a Pclzip class
But I was thinking there might be something in the curl
library that does it.

Since I want to grab the file with curl, does anyone know if curl can
unzip the file ?

If so how do I do it with the above file.

many thanks

oracleguy
10-26-2009, 05:09 PM
CURL cannot unzip files, you would need to use a different library for that. You can still use CURL to download the file first though.

jeddi
10-27-2009, 07:44 PM
Thanks, I also just noticed this in the php manual:

About using the Zip functions:

PHP 5.2.0 and later Linux systems

In order to use these functions you must compile PHP with zip support by using the --enable-zip configure option.

When using the pecl install it should automatically download the latest version and install it right? So to use the pecl command do I just login with putty and at the command line type:

$ pecl install zip

No yum install ? Is that dollar sign supposed to be there ?

Would much appreciate the clarification :)

Lamped
10-27-2009, 07:57 PM
If it's your own server, or you have access to it, you can always:



exec('unzip '.$myzipfile.' /home/mysite/temp');

oesxyl
10-27-2009, 10:20 PM
Thanks, I also just noticed this in the php manual:

About using the Zip functions:

PHP 5.2.0 and later Linux systems

In order to use these functions you must compile PHP with zip support by using the --enable-zip configure option.

When using the pecl install it should automatically download the latest version and install it right? So to use the pecl command do I just login with putty and at the command line type:

$ pecl install zip

No yum install ? Is that dollar sign supposed to be there ?

Would much appreciate the clarification :)
in manual say how you can do in general, for a given linux distro you must use the repository of the package manager you use. Find what rpm you need and then use yum to install it.

best regards

jeddi
10-29-2009, 05:01 PM
OK, after a lot of updating of PEAR and stuff I now
have the ZipArchive working ... a bit :o

This is my code:



<?php

if (!extension_loaded('zip')) {
dl('zip.so');
}

$func_path = "/home/guru54gt5/public_html/im/my_functions.php";
require_once("$func_path");

$target_url = "http://www.support-focus.com/test_map.zip";
$userAgent = 'Googlebot/2.1 (http://www.googlebot.com/bot.html)';

echo "<br>Starting<br>Target_url: $target_url<br><br>";

// make the cURL request to $target_url
$ch = curl_init();
curl_setopt($ch, CURLOPT_USERAGENT, $userAgent);
curl_setopt($ch, CURLOPT_URL,$target_url);
curl_setopt($ch, CURLOPT_FAILONERROR, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_AUTOREFERER, true);
curl_setopt($ch, CURLOPT_BINARYTRANSFER,true);
curl_setopt($ch, CURLOPT_TIMEOUT, 10);
$page = curl_exec($ch);
if (!$page) {
echo "<br />cURL error number:" .curl_errno($ch);
echo "<br />cURL error:" . curl_error($ch);
exit;
}
curl_close($ch);

// Un zip the file

$zip = new ZipArchive;
if (! $zip) {
echo "Could not make ZipArchive object.";
exit;
}
$zip->open("$page");
$zip->extractTo('./');
$zip->close();
echo "Ok!";
?>



The errors I get are in fact warnings:

Warning: ZipArchive::extractTo() [function.ZipArchive-extractTo]: Invalid or unitialized Zip object in /home/guru54gt5/public_html/sys/convert_xml_no1.php on line 40

Warning: ZipArchive::close() [function.ZipArchive-close]: Invalid or unitialized Zip object in /home/guru54gt5/public_html/sys/convert_xml_no1.php on line 41
Ok!

Line 40 and 41 are:


$zip->extractTo('./');
$zip->close();


Is the problem because my $page is not a file ?

I am a bit confused about this so would appreciate some help.

Thanks.

jeddi
10-30-2009, 09:20 AM
I have moved forward with my script
but I am still getting errors.

This is what I have:


<?php

if (!extension_loaded('zip')) {
dl('zip.so');
}

$func_path = "/home/guru54gt5/public_html/im/my_functions.php";
require_once("$func_path");

$target_url = "http://www.support-focus.com/test_map.zip";
$userAgent = 'Googlebot/2.1 (http://www.googlebot.com/bot.html)';
$file_zip = "new.zip";
$file_txt = "new.txt";

echo "<br>Starting<br>Target_url: $target_url<br><br>";


// make the cURL request to $target_url
$ch = curl_init();
$fp = fopen("$file_zip", "w");
curl_setopt($ch, CURLOPT_USERAGENT, $userAgent);
curl_setopt($ch, CURLOPT_URL,$target_url);
curl_setopt($ch, CURLOPT_FAILONERROR, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_AUTOREFERER, true);
curl_setopt($ch, CURLOPT_BINARYTRANSFER,true);
curl_setopt($ch, CURLOPT_TIMEOUT, 10);
curl_setopt($ch, CURLOPT_FILE, $fp);

$page = curl_exec($ch);


if (!$page) {
echo "<br />cURL error number:" .curl_errno($ch);
echo "<br />cURL error:" . curl_error($ch);
exit;
}
curl_close($ch);

echo "<br>Downloaded file: $target_url";
echo "<br>Saved as file: $file_zip";
echo "<br>About to unzip ...";

// Un zip the file

$zip = new ZipArchive;
if (! $zip) {
echo "Could not make ZipArchive object.";
exit;
}
$zip->open("$file_zip");
$zip->extractTo("$file_txt");
$zip->close();

echo "<br>Unzipped file to: $file_txt<br><br>";

?>


The result:


Starting
Target_url: http://www.support-focus.com/test_map.zip


Downloaded file: http://www.support-focus.com/test_map.zip
Saved as file: new.zip
About to unzip ...
Warning: ZipArchive::extractTo() [function.ZipArchive-extractTo]: Invalid or unitialized Zip object in /home/guru54gt5/public_html/sys/convert_xml_no2.php on line 52

Warning: ZipArchive::close() [function.ZipArchive-close]: Invalid or unitialized Zip object in /home/guru54gt5/public_html/sys/convert_xml_no2.php on line 53

Unzipped file to: new.txt


Even though it says that it has unzipped the file,
it doesn't. :confused:

Anyone know what I am doing wrong ?

djm0219
10-30-2009, 11:40 AM
Your call to cURL does not appear to be suppressing headers so what you're getting back is likely the headers of the page as well as the ZIP file itself. Adding the headers is likely the reason why your ZIP file ends up being invalid. Try adding:


curl_setopt ($ch, CURLOPT_HEADER,0);

to your cURL options.

You could probably see this by looking at the ZIP file that gets downloaded with a text editor. It will likely be the headers followed by the binary data that is the actual ZIP file.

jeddi
10-30-2009, 12:25 PM
Thanks for advice :)

I have incorporated 2 changes:

1)The headers have been set to 0 so as not
to put them in the file

2) I have checked to see if open() is working
( It isn't )

Here is my script:



if (!extension_loaded('zip')) {
dl('zip.so');
}

$func_path = "/home/guru54gt5/public_html/im/my_functions.php";
require_once("$func_path");

$target_url = "http://www.support-focus.com/test_map.zip";
$userAgent = 'Googlebot/2.1 (http://www.googlebot.com/bot.html)';
$file_zip = "new.zip";
$file_txt = "new.txt";

echo "<br>Starting<br>Target_url: $target_url";
echo "<br>Headers stripped out";

// make the cURL request to $target_url
$ch = curl_init();
$fp = fopen("$file_zip", "w");
curl_setopt($ch, CURLOPT_USERAGENT, $userAgent);
curl_setopt($ch, CURLOPT_URL,$target_url);
curl_setopt($ch, CURLOPT_FAILONERROR, true);
curl_setopt($ch, CURLOPT_HEADER,0);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_AUTOREFERER, true);
curl_setopt($ch, CURLOPT_BINARYTRANSFER,true);
curl_setopt($ch, CURLOPT_TIMEOUT, 10);
curl_setopt($ch, CURLOPT_FILE, $fp);

$page = curl_exec($ch);


if (!$page) {
echo "<br />cURL error number:" .curl_errno($ch);
echo "<br />cURL error:" . curl_error($ch);
exit;
}
curl_close($ch);

echo "<br>Downloaded file: $target_url";
echo "<br>Saved as file: $file_zip";
echo "<br>About to unzip ...";

// Un zip the file

$zip = new ZipArchive;
if (! $zip) {
echo "<br>Could not make ZipArchive object.";
exit;
}
if($zip->open("$file_zip") != "true") {
echo "<br>Could not open $file_zip";
}
$zip->extractTo("$file_txt");
$zip->close();

echo "<br>Unzipped file to: $file_txt<br><br>";

?>



And the result:


Starting
Target_url: http://www.support-focus.com/test_map.zip
Headers stripped out
Downloaded file: http://www.support-focus.com/test_map.zip
Saved as file: new.zip
About to unzip ...
Could not open new.zip
Warning: ZipArchive::extractTo() [function.ZipArchive-extractTo]: Invalid or unitialized Zip object in /home/guru54gt5/public_html/sys/convert_xml_no2.php on line 55

Warning: ZipArchive::close() [function.ZipArchive-close]: Invalid or unitialized Zip object in /home/guru54gt5/public_html/sys/convert_xml_no2.php on line 56

Unzipped file to: new.txt

So the ope() is not working - but I do not know why :confused:

The file is only a small sitemap file zipped up.

So I opened the downloaded file new.zip with notepad

This is what it contains:



PK  ZO;hmM 2 test_map.xmlWr0}+\^;MnM az{ʨ5$c IMi+X2:g/^W번Vh"}nZRn>U֦:Q{ܹjMrX,c+sx0Nz}ڪ(Iׯf*?(m}*l~ER(N^O{X6 "iP8̢F<VJc4FSh#y~kxTF i4eVÂdO*2$<ě\%. ~M30{
*(I90~vq JgUy'I0 ]%@0|[T/I;!]0JSW+
a]IY] Oy |=gMVVBn0kgBM0.8% W :( JI-\# c!4hNjC9Dp8]m4B%P sâTu]j,O^Hz]"|.`Lz7U#< ^ktOG3}"t8j0&TI#myE8wy^-LvL漱| أ^]<賷dQj薓%5>Y-"662*{xL!&Ӽs!LPK   ZO;hmM 2  test_map.xmlPK   : 

Any ideas why things are not working ?

djm0219
10-30-2009, 12:40 PM
Are you able to open the downloaded ZIP file without errors (not from your program)?

jeddi
10-30-2009, 04:35 PM
Good question.

The answer is YES

I ftpped the downloaded zip file over to my pc
and then unzipped it.

No problem with unzipping the file



EZ Archive Ads Plugin for vBulletin Copyright 2006 Computer Help Forum