I am looking for a tutorial that teaches you how to build a web crawler in php, one that follows links looks at the robots.txt etc. But I can't find one can someone point me to one.

Thanks