Hello and welcome to our community! Is this your first visit?
Register
Enjoy an ad free experience by logging in. Not a member yet? Register.
Results 1 to 5 of 5
  1. #1
    Regular Coder dan-dan's Avatar
    Join Date
    Aug 2009
    Location
    England
    Posts
    483
    Thanks
    22
    Thanked 79 Times in 78 Posts

    Disallow folder indexing

    I'm creating a site locally, though I want to test it live.

    I don't have any hosting at the moment, though I'm able to host it on a friends site I created for her some time ago.

    However, I don't want the folder and its contents to get indexed by any search engines.

    I read that I can use the below rule within robots.txt
    Code:
    User-agent: *
    Disallow: /
    My question is, will this work being placed within a specific folder, or is it something that only works from root for the entire site.

    If the later, is there a way to do it?

    Thank you.

  • #2
    Codeasaurus Rex
    Join Date
    Jun 2008
    Location
    Redmond, WA
    Posts
    659
    Thanks
    31
    Thanked 100 Times in 94 Posts
    Some search engines will look for it in each directory, but not all. As such, in order for it to be effective, you need to put the file in the "root" directory (i.e: the top level directory, http://example.org/robots.txt).

    Having said this, you can tell robots to simply not index your directory. So, instead of doing Disallow: /, which would tell search engines to not index the entire site, replace that with Disallow: /my_directory/ which will restrict no indexing to your directory (and sub-directories thereof).

    For more information on this, check out this site.
    Unless otherwise stated, any code posted is most likely untested and may contain syntax errors.
    My posts, comments, code, and suggestions reflect only my personal views.
    Web Portfolio and Code Snippets: http://shanechism.com

  • Users who have thanked ShaneC for this post:

    dan-dan (04-14-2012)

  • #3
    Regular Coder dan-dan's Avatar
    Join Date
    Aug 2009
    Location
    England
    Posts
    483
    Thanks
    22
    Thanked 79 Times in 78 Posts
    Thanks for the explanation and link ShaneC, much appreciated.

  • #4
    New to the CF scene
    Join Date
    Apr 2012
    Posts
    3
    Thanks
    0
    Thanked 0 Times in 0 Posts
    does this .htaccess file go in the folder of the "Disallowed" listing or in the root of the site?

  • #5
    New Coder
    Join Date
    Sep 2011
    Posts
    80
    Thanks
    0
    Thanked 13 Times in 12 Posts
    I prefer to just use .htaccess and add a password to the directory.

    That way search engines can't access it (some shady web crawlers ignore robots.txt rules) and it means people can't access it by accident.


  •  

    Posting Permissions

    • You may not post new threads
    • You may not post replies
    • You may not post attachments
    • You may not edit your posts
    •