View Full Version : stop crawling your website
04-21-2008, 08:46 AM
I don't want to search engine bots crawling my website at all.
I'll appreciate your suggestions, tips ... please share your experience.
04-21-2008, 09:26 AM
Add a file named robots.txt in the root having
Disallow:/Also you could remove the google cache via www.google.com/webmasters/tools/
04-21-2008, 09:35 AM
I have about 2 domains and domain pointer are set ... for e.g.
site1.com --> /site1
as u said create a file in root directory ... and if i want to stop crawler from site1.com ... u want me to create robot.txt in site1 folder ?
04-21-2008, 09:45 AM
Exactly! All search engines respect a file located at http://yourdomain.com/robots.txt
Powered by vBulletin® Version 4.2.2 Copyright © 2017 vBulletin Solutions, Inc. All rights reserved.