...

View Full Version : Hiding pages from SearchBots



Rocket Boy
11-02-2005, 11:49 PM
Does anybody know how I can exclude certain pages being submitted to search engines whilst still allowing the majority of my pages to be submitted.

In a nutshell, how do tell bots to only record particular pages and ignore others?

Thanks in advance.

gsnedders
11-03-2005, 12:19 AM
Google for robots.txt (sorry, I don't have the time to give any specific info).

Rocket Boy
11-03-2005, 02:22 AM
Ta for that :thumbsup:

vinyl-junkie
11-03-2005, 05:13 AM
What you're talking about is called robots exclusion. Here (http://www.robotstxt.org/wc/exclusion.html) is one of the best sites I've found to explain how that works. Hope it helps.

iota
11-03-2005, 05:25 AM
http://forum.flashband.net/viewtopic.php?t=320

Rocket Boy
11-03-2005, 07:25 AM
I have added a Robots.txt file to the root directory. I guess only time will tell if it works or not?

gsnedders
11-03-2005, 03:41 PM
I have added a Robots.txt file to the root directory. I guess only time will tell if it works or not?
Yeah, I don't think there are any other ways of testing it.

Rocket Boy
11-03-2005, 05:15 PM
I can't make up my mind wether or not their was a hint of sarcasm in their so I'd better ask, is there a test?

gsnedders
11-03-2005, 06:10 PM
I can't make up my mind wether or not their was a hint of sarcasm in their so I'd better ask, is there a test?
Probably. Don't know where though.

SteelValor
11-03-2005, 06:51 PM
The robots will stop visiting once they see the disallow command in the robots.txt file. However, the search engine might still keep a copy of those pages on file. It would be best to also use the meta tags for the search engines, robots, googlebot, etc., to tell them that the content has expired and not to re-index that page/site.

http://www.searchenginewatch.com is a pretty usefull site.

* Security Note * Never put ANY file/directory references to secured information in your robots.txt file. That's one of the first places a hacker will hit up when looking for exploitable files.

iota
11-11-2005, 05:59 AM
Both robots.txt and meta tags only suggest bots for stopping.
They don't force to prevent them.

Rocket Boy
11-15-2005, 01:06 PM
In light of opinion here and advice from Google etc, robots.txt is out and meta tags are in

vinyl-junkie
11-15-2005, 01:19 PM
In light of opinion here and advice from Google etc, robots.txt is out and meta tags are inBots can (and some do) ignore both.

gsnedders
11-15-2005, 03:24 PM
and meta tags are more often ignored than robots.txt

SteelValor
11-15-2005, 04:51 PM
Bottom Line is turn off directory browsing rename the files and don't like to those files from other web pages. If they are listed they should disappear in a few weeks.



EZ Archive Ads Plugin for vBulletin Copyright 2006 Computer Help Forum