Hello and welcome to our community! Is this your first visit?
Register
Enjoy an ad free experience by logging in. Not a member yet? Register.
Results 1 to 2 of 2
  1. #1
    Regular Coder
    Join Date
    Jan 2010
    Location
    Washington
    Posts
    223
    Thanks
    34
    Thanked 0 Times in 0 Posts

    Question setting up crawler to login to sites

    I have a crawler which crawls for links and i want to make it login to sites that require you to login to view them. my crawler has this function in the phpcrawler.class.php file
    Code:
      // Method adds an authentication-login for special URLs given in expression
      // (PCRE)
      function addBasicAuthentication($expression, $username, $password)
      {
        $this->initCrawler();
        $check = PHPCrawlerUtils::checkExpressionPattern($expression); // Check pattern
        
        if ($check == true)
        {
          $c = count($this->pageRequest->basic_authentications);
          $this->pageRequest->basic_authentications[$c]["match"] = $expression;
          $this->pageRequest->basic_authentications[$c]["username"] = $username;
          $this->pageRequest->basic_authentications[$c]["password"] = $password;
          return true;
        }
        else return false; 
      }
    and here is checkExpressionPattern from PHPCrawlerUtils.class.php
    PHP Code:
      // Function checks if a given preg_pattern is a valid one.
      
      
    function checkExpressionPattern($pattern)
      {
        
    $check = @preg_match($pattern"anything"); // thats the easy way to check a pattern ;)
        
    if (is_integer($check) == false) return false;
        else return 
    true;
      }
      

    with this info is it possible to create something like that? if so can somebody help me? i would appreciate it. Thanks.

  • #2
    Regular Coder
    Join Date
    Jan 2010
    Location
    Washington
    Posts
    223
    Thanks
    34
    Thanked 0 Times in 0 Posts
    so no one can help me with this? Its not a big deal but if anyone could help me it would be very much appreciated.


  •  

    Posting Permissions

    • You may not post new threads
    • You may not post replies
    • You may not post attachments
    • You may not edit your posts
    •