I placed the contents of the old version of my site into a folder under my site. I have no link whatsoever to that folder but still search bots might enter there i guess...
How can I prevent search bots to enter to that folder? a robot.txt file or something? what do i write in that file?
the reason is i dont want to be penalized by search engines for duplicate content
05-14-2011, 05:08 PM
You need to have something like this:
The "User-agent: *" means this section applies to all robots. The "Disallow: /yourfoldername" tells the robot that it should not visit any pages on the site within that folder.
05-14-2011, 05:19 PM
It'd better to protect that folder by an HTTP authentication (http://www.google.com/search?q=htaccess+prtection), so that you can prevent users also from any "unauthorised" access. (You host's control panel might have an easy interface to add this)
05-14-2011, 05:28 PM
If I read this person's question correctly, they're asking how to keep the search engines from indexing content, not how to keep an unauthorized person out of those pages.