Hello and welcome to our community! Is this your first visit?
Register
Enjoy an ad free experience by logging in. Not a member yet? Register.
Results 1 to 10 of 10
  1. #1
    Senior Coder crmpicco's Avatar
    Join Date
    Jan 2005
    Location
    Mauchline, Scotland
    Posts
    1,096
    Thanks
    15
    Thanked 1 Time in 1 Post

    prevent Google/Yahoo!/MSN spidering my webpage

    Hi,

    Am i safe enough with the following meta tags to prevent Google/Yahoo!/MSN Search from spidering my webpage?

    Code:
    <meta name="robots" content="noindex" />
    <meta name="robots" content="nofollow" />
    <meta name="robots" content="noarchive" />
    <meta name="robots" content="noodp" />
    <meta name="robots" content="noimageindex,nomediaindex" />
    <meta name="robots" content="unavailable_after: 05-Apr-2009 22:00:00 CET" />
    <meta name="googlebot" content="nosnippet" />
    <meta name="slurp" content="noydir">
    Cheers,
    Picco

  • #2
    Master Coder
    Join Date
    Dec 2007
    Posts
    6,682
    Thanks
    436
    Thanked 890 Times in 879 Posts
    you can use disallow and the name of the bot in robots.txt.
    noindex is only to not index your page.

    best regards

  • #3
    Senior Coder twodayslate's Avatar
    Join Date
    Mar 2007
    Location
    VA
    Posts
    1,042
    Thanks
    67
    Thanked 39 Times in 39 Posts
    twitter | Quality Hosting - $5.95/mo*
    Feel free to PM me!

  • #4
    Senior Coder crmpicco's Avatar
    Join Date
    Jan 2005
    Location
    Mauchline, Scotland
    Posts
    1,096
    Thanks
    15
    Thanked 1 Time in 1 Post
    Nice one, thanks.

    I have added a robots.txt to my server root, disallowing the one page that I don't want to appear in search results.

    Code:
    User-agent: *
    Disallow: /picco.html
    Does anyone know of a way to test this, short of doing a search on Google over the next couple of days.......

    Picco
    Last edited by crmpicco; 04-05-2009 at 11:25 PM. Reason: mis-spelt URL

  • #5
    Senior Coder timgolding's Avatar
    Join Date
    Aug 2006
    Location
    Southampton
    Posts
    1,519
    Thanks
    114
    Thanked 110 Times in 109 Posts
    No need to use the robots.txt in the way you have and use the meta tags you have provided. Prolley just use one or the other.

    In answere to your question

    Am i safe enough with the following meta tags to prevent Google/Yahoo!/MSN Search from spidering my webpage?
    Yes you are.
    Last edited by timgolding; 04-05-2009 at 11:40 PM.
    You can not say you know how to do something, until you can teach it to someone else.

  • #6
    Senior Coder crmpicco's Avatar
    Join Date
    Jan 2005
    Location
    Mauchline, Scotland
    Posts
    1,096
    Thanks
    15
    Thanked 1 Time in 1 Post
    cheers, is there a test site out there where I can check if a search engine will be able to spider my page?

    Thanks.

  • #7
    Regular Coder sea4me's Avatar
    Join Date
    Jan 2009
    Location
    Damn, I don't know...
    Posts
    390
    Thanks
    11
    Thanked 28 Times in 27 Posts
    Quote Originally Posted by crmpicco View Post
    cheers, is there a test site out there where I can check if a search engine will be able to spider my page?

    Thanks.
    Try this:
    http://tools.seobook.com/general/spider-test/

  • #8
    Master Coder
    Join Date
    Dec 2007
    Posts
    6,682
    Thanks
    436
    Thanked 890 Times in 879 Posts
    Quote Originally Posted by sea4me View Post
    this will show how op site is seen by a spider not if it can be indexed or not.

    best regards

  • #9
    Regular Coder
    Join Date
    Jan 2009
    Posts
    316
    Thanks
    7
    Thanked 92 Times in 91 Posts
    Quote Originally Posted by crmpicco View Post
    cheers, is there a test site out there where I can check if a search engine will be able to spider my page?
    You can test it from within Google Webmaster Tools itself if you have the site listed in your dashboard (verified owner).

  • #10
    Regular Coder
    Join Date
    Mar 2009
    Location
    Troutdale, OR
    Posts
    145
    Thanks
    12
    Thanked 6 Times in 6 Posts
    Try to type this in the search box: site:www.yoursite.com
    This will work for some search engines to check what pages of your web site are there.


  •  

    Posting Permissions

    • You may not post new threads
    • You may not post replies
    • You may not post attachments
    • You may not edit your posts
    •