Hello and welcome to our community! Is this your first visit?
Register
Enjoy an ad free experience by logging in. Not a member yet? Register.
Results 1 to 3 of 3
  1. #1
    New to the CF scene
    Join Date
    Dec 2010
    Posts
    1
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Twitter's robots.txt question:

    Twitter's robots.txt, It shows everything is disallowed, but surprisingly search engines are crawling and indexing everybody's profiles pages, Why?

  • #2
    Senior Coder
    Join Date
    Jul 2009
    Location
    South Yorkshire, England
    Posts
    2,318
    Thanks
    6
    Thanked 304 Times in 303 Posts
    Is there a robots meta tag on the profile pages?

  • #3
    Regular Coder adarshakb's Avatar
    Join Date
    Jun 2009
    Location
    Silicon valley of india
    Posts
    247
    Thanks
    11
    Thanked 1 Time in 1 Post
    nope i 2 saw and am confused...
    here is the link http://twitter.com/robots.txt
    here is the actual content in it...
    Code:
    #Google Search Engine Robot
    User-agent: Googlebot
    # Crawl-delay: 10 -- Googlebot ignores crawl-delay ftl
    Disallow: /*?
    Disallow: /*/with_friends
    
    #Yahoo! Search Engine Robot
    User-Agent: Slurp
    Crawl-delay: 1
    Disallow: /*?
    Disallow: /*/with_friends
    
    #Microsoft Search Engine Robot
    User-Agent: msnbot
    Disallow: /*?
    Disallow: /*/with_friends
    
    # Every bot that might possibly read and respect this file.
    User-agent: *
    Disallow: /*?
    Disallow: /*/with_friends
    Disallow: /oauth
    Disallow: /1/oauth
    Two things are infinite: the universe and human stupidity; and I'm not sure about the universe.

    Albert Einstein
    -----------------------------------------------------
    My Blog songs


  •  

    Posting Permissions

    • You may not post new threads
    • You may not post replies
    • You may not post attachments
    • You may not edit your posts
    •