Hello and welcome to our community! Is this your first visit?
Register
Enjoy an ad free experience by logging in. Not a member yet? Register.
Results 1 to 4 of 4
  1. #1
    Regular Coder saeed's Avatar
    Join Date
    Oct 2002
    Location
    West Yorkshire
    Posts
    343
    Thanks
    32
    Thanked 0 Times in 0 Posts

    stop crawling your website

    Hello,

    I don't want to search engine bots crawling my website at all.

    I'll appreciate your suggestions, tips ... please share your experience.

    Thank you,
    Don't click on this!

    #!/usr/bin/saeed

  • #2
    Supreme Master coder! abduraooft's Avatar
    Join Date
    Mar 2007
    Location
    N/A
    Posts
    14,854
    Thanks
    160
    Thanked 2,223 Times in 2,210 Posts
    Blog Entries
    1
    Add a file named robots.txt in the root having
    Code:
    user-agent:*
    Disallow:/
    Also you could remove the google cache via www.google.com/webmasters/tools/
    The Dream is not what you see in sleep; Dream is the thing which doesn't let you sleep. --(Dr. APJ. Abdul Kalam)

  • #3
    Regular Coder saeed's Avatar
    Join Date
    Oct 2002
    Location
    West Yorkshire
    Posts
    343
    Thanks
    32
    Thanked 0 Times in 0 Posts
    thanks mate,

    I have about 2 domains and domain pointer are set ... for e.g.

    /
    /site1
    /site2

    site1.com --> /site1

    as u said create a file in root directory ... and if i want to stop crawler from site1.com ... u want me to create robot.txt in site1 folder ?
    Don't click on this!

    #!/usr/bin/saeed

  • #4
    Supreme Master coder! abduraooft's Avatar
    Join Date
    Mar 2007
    Location
    N/A
    Posts
    14,854
    Thanks
    160
    Thanked 2,223 Times in 2,210 Posts
    Blog Entries
    1
    Exactly! All search engines respect a file located at http://yourdomain.com/robots.txt
    The Dream is not what you see in sleep; Dream is the thing which doesn't let you sleep. --(Dr. APJ. Abdul Kalam)

  • Users who have thanked abduraooft for this post:

    saeed (04-21-2008)


  •  

    Posting Permissions

    • You may not post new threads
    • You may not post replies
    • You may not post attachments
    • You may not edit your posts
    •