...

View Full Version : prevent Google/Yahoo!/MSN spidering my webpage



crmpicco
04-05-2009, 11:17 PM
Hi,

Am i safe enough with the following meta tags to prevent Google/Yahoo!/MSN Search from spidering my webpage?



<meta name="robots" content="noindex" />
<meta name="robots" content="nofollow" />
<meta name="robots" content="noarchive" />
<meta name="robots" content="noodp" />
<meta name="robots" content="noimageindex,nomediaindex" />
<meta name="robots" content="unavailable_after: 05-Apr-2009 22:00:00 CET" />
<meta name="googlebot" content="nosnippet" />
<meta name="slurp" content="noydir">


Cheers,
Picco

oesxyl
04-05-2009, 11:38 PM
you can use disallow and the name of the bot in robots.txt.
noindex is only to not index your page.

best regards

twodayslate
04-05-2009, 11:48 PM
http://www.google.com/support/webmasters/bin/answer.py?answer=35302

crmpicco
04-06-2009, 12:19 AM
Nice one, thanks.

I have added a robots.txt to my server root, disallowing the one page that I don't want to appear in search results.



User-agent: *
Disallow: /picco.html


Does anyone know of a way to test this, short of doing a search on Google over the next couple of days.......

Picco

timgolding
04-06-2009, 12:36 AM
No need to use the robots.txt in the way you have and use the meta tags you have provided. Prolley just use one or the other.

In answere to your question



Am i safe enough with the following meta tags to prevent Google/Yahoo!/MSN Search from spidering my webpage?


Yes you are.

crmpicco
04-06-2009, 11:45 PM
cheers, is there a test site out there where I can check if a search engine will be able to spider my page?

Thanks.

sea4me
04-07-2009, 12:01 AM
cheers, is there a test site out there where I can check if a search engine will be able to spider my page?

Thanks.

Try this:
http://tools.seobook.com/general/spider-test/

oesxyl
04-07-2009, 12:23 AM
Try this:
http://tools.seobook.com/general/spider-test/
this will show how op site is seen by a spider not if it can be indexed or not.

best regards

Fisher
04-07-2009, 07:25 PM
cheers, is there a test site out there where I can check if a search engine will be able to spider my page?

You can test it from within Google Webmaster Tools itself if you have the site listed in your dashboard (verified owner).

RomanTaylor
04-08-2009, 09:31 AM
Try to type this in the search box: site:www.yoursite.com
This will work for some search engines to check what pages of your web site are there.



EZ Archive Ads Plugin for vBulletin Copyright 2006 Computer Help Forum