View Full Version : stop crawling your website

04-21-2008, 07:46 AM

I don't want to search engine bots crawling my website at all.

I'll appreciate your suggestions, tips ... please share your experience.

Thank you,

04-21-2008, 08:26 AM
Add a file named robots.txt in the root having
Disallow:/Also you could remove the google cache via www.google.com/webmasters/tools/

04-21-2008, 08:35 AM
thanks mate,

I have about 2 domains and domain pointer are set ... for e.g.


site1.com --> /site1

as u said create a file in root directory ... and if i want to stop crawler from site1.com ... u want me to create robot.txt in site1 folder ?

04-21-2008, 08:45 AM
Exactly! All search engines respect a file located at http://yourdomain.com/robots.txt

EZ Archive Ads Plugin for vBulletin Copyright 2006 Computer Help Forum