Hello and welcome to our community! Is this your first visit?
Register
Enjoy an ad free experience by logging in. Not a member yet? Register.
Results 1 to 4 of 4

Thread: robots txt file

  1. #1
    Regular Coder
    Join Date
    Jul 2009
    Posts
    195
    Thanks
    26
    Thanked 0 Times in 0 Posts

    robots txt file

    I placed the contents of the old version of my site into a folder under my site. I have no link whatsoever to that folder but still search bots might enter there i guess...

    How can I prevent search bots to enter to that folder? a robot.txt file or something? what do i write in that file?

    the reason is i dont want to be penalized by search engines for duplicate content

  • #2
    $object->toCD-R(LP); vinyl-junkie's Avatar
    Join Date
    Jun 2003
    Posts
    3,088
    Thanks
    2
    Thanked 23 Times in 23 Posts
    You need to have something like this:
    Code:
    User-agent: *
    Disallow: /yourfoldername
    The "User-agent: *" means this section applies to all robots. The "Disallow: /yourfoldername" tells the robot that it should not visit any pages on the site within that folder.
    Music Around The World - Collecting tips, trade
    and want lists, album reviews, & more
    SNAP to it!

  • #3
    Supreme Master coder! abduraooft's Avatar
    Join Date
    Mar 2007
    Location
    N/A
    Posts
    14,852
    Thanks
    160
    Thanked 2,223 Times in 2,210 Posts
    Blog Entries
    1
    It'd better to protect that folder by an HTTP authentication, so that you can prevent users also from any "unauthorised" access. (You host's control panel might have an easy interface to add this)
    The Dream is not what you see in sleep; Dream is the thing which doesn't let you sleep. --(Dr. APJ. Abdul Kalam)

  • #4
    $object->toCD-R(LP); vinyl-junkie's Avatar
    Join Date
    Jun 2003
    Posts
    3,088
    Thanks
    2
    Thanked 23 Times in 23 Posts
    If I read this person's question correctly, they're asking how to keep the search engines from indexing content, not how to keep an unauthorized person out of those pages.
    Music Around The World - Collecting tips, trade
    and want lists, album reviews, & more
    SNAP to it!


  •  

    Posting Permissions

    • You may not post new threads
    • You may not post replies
    • You may not post attachments
    • You may not edit your posts
    •