View Full Version : No robots in session secured area... please

01-29-2010, 10:59 AM
So I have an area secured by sessions where people login, nothing within this area can be used unless you have have this session.

I have only recently started to become concerned about robots accessing this area - whats the advise on this?

The area is accessed via a post form so login variables are not sent with the url, I have the usual meta in all php files within this section:

<meta name="googlebot" content="noarchive" />
<meta name="googlebot" content="nosnippet" />
<meta name="googlebot" content="noindex" />
<meta name="googlebot" content="nofollow"/>

and a:

User-agent: *
Disallow: /

in my robots.txt

Allthough I feel there maybe a way to do this with .htaccess

Any advice would be very welcome on stoping any robots caching anything in this area, thanks in advance.

Thinking about it can robots get in here without a password?

So a bit more research I could use in my .htaccess:

RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} ^bot
RewriteRule ^(.*)$ http://www.robotstxt.org/

or in my virtual hosts file:

<Directory />
BrowserMatch ^bot deny_host
Deny from env=deny_host

hmmm is there a better way to do this? (without involving placing and updating all bots in these files)