...

View Full Version : Why use curl rather than fopen() and fread() ?



jeddi
10-21-2009, 04:18 PM
Sorry - I am a bit of a newbie with this curl stuff.

Funny thing is, I have been using curl but now I come to
want to code up something from scratch I have to ask:
why is curl better than using fopen() and fread() ?

I have got the Snoopy class as well if I want to use it
and I just got the "elHttpClient" with is a cUrl addon
class to supposedly make things easier.

Now I don't know what to use and why :(

Can someone please help explaing what the differences are ?

Thanks.

bcarl314
10-21-2009, 06:06 PM
I may not be 100% right here, but in general fread, fopen are going to send HTTP GET requests, where as curl can be used to send either HTTP GET or HTTP POST requests.

jeddi
10-21-2009, 06:47 PM
Thanks for your answer.

I also now that with cUrl you can select your agent
and referer, but I don't really see how important that is
- does that mean that you can "cloak" your identity from the
server you visit ?

jeddi
10-21-2009, 08:10 PM
If anyone else can offer some insight
it would be appreciated.

Maybe the main difference is being able to submit forms using cUrl ?

(But then Snoopy does that without cUrl, doesn't it ? )

Please help as I really want to grasp this difference and use the best method.

thanks

kbluhm
10-21-2009, 09:33 PM
It's also much faster, and does not rely on allow_url_fopen.

jeddi
10-22-2009, 07:55 AM
Well that is an important point.

So how much faster is cUrl over fopen() and fread() ?

Are we talking about marginally faster ( like 10 % )
or factorially faster ( like 3 x faster ) ?

I want to chose the best one for crawling the web and developing my own index. I will open a thread on that subject today.

Any one know more about this ?

kbluhm
10-22-2009, 01:52 PM
Well why not research this yourself?

Create a file for each option -- curl, fopen, file_get_contents, or fsockopen -- and just benchmark dead requests within a for-loop, using microtime() to benchmark the execution time.

Fou-Lu
10-23-2009, 12:03 AM
Well why not research this yourself?

Create a file for each option -- curl, fopen, file_get_contents, or fsockopen -- and just benchmark dead requests within a for-loop, using microtime() to benchmark the execution time.

And post you're results too, I'm a little curious. Given that these are all stream handlers, I'm a little curious what beats what (my suspicion: fsockopen > curl > fopen >>>>>> file_get_contents [since this one I expect to be the slowest])

FWDrew
10-23-2009, 10:35 AM
And post you're results too, I'm a little curious. Given that these are all stream handlers, I'm a little curious what beats what (my suspicion: fsockopen > curl > fopen >>>>>> file_get_contents [since this one I expect to be the slowest])
Here is a little data for you all, from what I have seen, cURL is much faster:

http://stackoverflow.com/questions/555523/filegetcontents-vs-curl-what-has-better-performance

jeddi
10-23-2009, 03:11 PM
That was very useful.

I read a lot of the surrounding links and learnt a lot.

ricorine
12-18-2009, 09:20 PM
About performance, I've just read this from http://forums.winamp.com/showthread.php?postid=2338006 :
http://ca3.php.net/manual/en/ref.curl.php#75126
"I have created a script that tests all 4 methods on 20 different websites 10 times each. The results are clear, here is one set of the results:

Curl time: 43.02 seconds
FGC time: 86.48 seconds
Fopen time: 86.34 seconds
Socket time: 44.91 seconds

As you can see, there is little performance difference between file_get_contents and fopen/fread. Likewise, there is little difference between curl and sockets."

Anyway, apart from the speed difference and the ability for cURL to make GET and POST requests, are there other differences?
(And in addition, what would be the IP passed, the server one?)

(Please excuse me if I made some mistakes, I'm not english ^^)

Lamped
12-19-2009, 11:05 AM
It doesn't matter which is faster if you want to write your own crawler. You should specify your own User-Agent and manage your own Referrer fields, which requires either cURL or your own totally custom socket handling functions.



EZ Archive Ads Plugin for vBulletin Copyright 2006 Computer Help Forum