Getting HTML from other webistes - fopen/fsockopen

[Fatman]

I am trying to retrieve information (HTML) from other sites using fopen and fscokopen.

in the past, its all worked fine but i have found a couple of sites that I am having problem reading information from. eg:

http://www.google.com/search?hl=en&lr=&ie=UTF-8&oe=UTF-8&safe=off&q=here

or

http://www.amazon.co.uk/exec/obidos/registry/wishlist/202-0277180-4903802

I either get blank information or just random rubbish

I can get this all to work in ASP using a similar method...

any ideas? just looking for example code that willread either of those links

Thanks
Fatman

[firepages]

my experience with amazon is that they dont want data grabbing , e.g. if they make it hard then you probably should not be doing it, many sites now require a cookie or use sessions to prevent remote data grabbers , in saying that this worked for me for google.


<?
$fp = fsockopen ("www.google.com", 80, $errno, $errstr, 30);
if (!$fp) {
echo "$errstr ($errno)<br>\n";
} else {
fputs ($fp, "GET /search?hl=en&lr=&ie=UTF-8&oe=UTF-8&safe=off&q=here HTTP/1.0\r\nHost: www.google.com\r\n");
fputs ($fp, "\r\n");
while (!feof($fp)) {
echo fgets ($fp,128);
}
fclose ($fp);
}
?>

[bcarl314]

Not to mention copyright issues.

Be very careful if your pulling in content from other sites without their permission. If it's for your personal site, you're probably safe with the "Fair Use" clause, but IANAL, so just be careful. It's also a really good idea to at least give credit to where you get the information.

[Fatman]

yep - im fine about things like that! :D

its just the actual process and purely for personal use - but the amazon one is the one i want to work most

any easy way to get it?

[bcarl314]

Amazon's got all sorts of neat tools you can use including XML feeds and the like. Go and sign up as an "amazon Associate" to get more info.

[Fatman]

i already am an associate!

but cant see anything for wish lists....

[sad69]

There are two PHP technologies that I am somewhat familiar with, namely: cURL and Snoopy.

Do a search for those online and you'll probably be able to use one of them to solve your problem.

Good luck,
Sadiq.

[Fatman]

tried snoopy - that did the same ?

need to install curl - would any kind soul be willing to try this as i dont wanna ask my host to install it if it does nothing,,,,,