View Full Version : robots txt file

05-14-2011, 02:32 PM
I placed the contents of the old version of my site into a folder under my site. I have no link whatsoever to that folder but still search bots might enter there i guess...

How can I prevent search bots to enter to that folder? a robot.txt file or something? what do i write in that file?

the reason is i dont want to be penalized by search engines for duplicate content

05-14-2011, 04:08 PM
You need to have something like this:

User-agent: *
Disallow: /yourfoldername
The "User-agent: *" means this section applies to all robots. The "Disallow: /yourfoldername" tells the robot that it should not visit any pages on the site within that folder.

05-14-2011, 04:19 PM
It'd better to protect that folder by an HTTP authentication (http://www.google.com/search?q=htaccess+prtection), so that you can prevent users also from any "unauthorised" access. (You host's control panel might have an easy interface to add this)

05-14-2011, 04:28 PM
If I read this person's question correctly, they're asking how to keep the search engines from indexing content, not how to keep an unauthorized person out of those pages.

EZ Archive Ads Plugin for vBulletin Copyright 2006 Computer Help Forum