...

View Full Version : setting up crawler to login to sites



cosmicsea
03-25-2010, 04:56 PM
I have a crawler which crawls for links and i want to make it login to sites that require you to login to view them. my crawler has this function in the phpcrawler.class.php file

// Method adds an authentication-login for special URLs given in expression
// (PCRE)
function addBasicAuthentication($expression, $username, $password)
{
$this->initCrawler();
$check = PHPCrawlerUtils::checkExpressionPattern($expression); // Check pattern

if ($check == true)
{
$c = count($this->pageRequest->basic_authentications);
$this->pageRequest->basic_authentications[$c]["match"] = $expression;
$this->pageRequest->basic_authentications[$c]["username"] = $username;
$this->pageRequest->basic_authentications[$c]["password"] = $password;
return true;
}
else return false;
}

and here is checkExpressionPattern from PHPCrawlerUtils.class.php

// Function checks if a given preg_pattern is a valid one.

function checkExpressionPattern($pattern)
{
$check = @preg_match($pattern, "anything"); // thats the easy way to check a pattern ;)
if (is_integer($check) == false) return false;
else return true;
}

}
with this info is it possible to create something like that? if so can somebody help me? i would appreciate it. Thanks.

cosmicsea
03-26-2010, 03:27 AM
so no one can help me with this? Its not a big deal but if anyone could help me it would be very much appreciated.



EZ Archive Ads Plugin for vBulletin Copyright 2006 Computer Help Forum