View Full Version : setting up crawler to login to sites

03-25-2010, 05:56 PM
I have a crawler which crawls for links and i want to make it login to sites that require you to login to view them. my crawler has this function in the phpcrawler.class.php file

// Method adds an authentication-login for special URLs given in expression
// (PCRE)
function addBasicAuthentication($expression, $username, $password)
$check = PHPCrawlerUtils::checkExpressionPattern($expression); // Check pattern

if ($check == true)
$c = count($this->pageRequest->basic_authentications);
$this->pageRequest->basic_authentications[$c]["match"] = $expression;
$this->pageRequest->basic_authentications[$c]["username"] = $username;
$this->pageRequest->basic_authentications[$c]["password"] = $password;
return true;
else return false;

and here is checkExpressionPattern from PHPCrawlerUtils.class.php

// Function checks if a given preg_pattern is a valid one.

function checkExpressionPattern($pattern)
$check = @preg_match($pattern, "anything"); // thats the easy way to check a pattern ;)
if (is_integer($check) == false) return false;
else return true;

with this info is it possible to create something like that? if so can somebody help me? i would appreciate it. Thanks.

03-26-2010, 04:27 AM
so no one can help me with this? Its not a big deal but if anyone could help me it would be very much appreciated.