...

View Full Version : stop crawling your website



saeed
04-21-2008, 07:46 AM
Hello,

I don't want to search engine bots crawling my website at all.

I'll appreciate your suggestions, tips ... please share your experience.

Thank you,

abduraooft
04-21-2008, 08:26 AM
Add a file named robots.txt in the root having
user-agent:*
Disallow:/Also you could remove the google cache via www.google.com/webmasters/tools/

saeed
04-21-2008, 08:35 AM
thanks mate,

I have about 2 domains and domain pointer are set ... for e.g.

/
/site1
/site2

site1.com --> /site1

as u said create a file in root directory ... and if i want to stop crawler from site1.com ... u want me to create robot.txt in site1 folder ?

abduraooft
04-21-2008, 08:45 AM
Exactly! All search engines respect a file located at http://yourdomain.com/robots.txt



EZ Archive Ads Plugin for vBulletin Copyright 2006 Computer Help Forum