...

View Full Version : Subdomains and restricted access Web pages



anshul
04-06-2005, 07:36 AM
This topic is related to SEO; I am not sure to post it here, but i am doing that.
I wanna ask, if some Web pages are set user login permission ( may be HTTP authentication on Apache/Linux ); then how'll SE robots/spider crawl/index the keywords? ( afterall, only valid cookie/session values will have the access! )

Secondly, how'll SE crawl subdomain, take e.g., 'abc.def.domain.com'
Still, webmasters use forums.domain.com mostly.
Actually, I can't see if any robots, visits my subdomains, when I see 'Awstats log' for my Web site.
( SEs ask/allow for domain submission, not subdomain submission )

So if domain.com/forums is better than forums.domain.com, they what is the need of subdomains, unless ftp access ( and main domain's images/css need relative-linking protection ) limitations are desired.

Please discuss.

Noodles24
04-07-2005, 12:24 PM
This topic is related to SEO; I am not sure to post it here, but i am doing that.
I wanna ask, if some Web pages are set user login permission ( may be HTTP authentication on Apache/Linux ); then how'll SE robots/spider crawl/index the keywords? ( afterall, only valid cookie/session values will have the access! )


If your site is passworded using .htaccess or something similar, then robots will not be able to spider the site, unless you write an exception into your authentication (i.e. allow all users who have the useragent of "GoogleBot", although that creates a bit of a security hole)



Secondly, how'll SE crawl subdomain, take e.g., 'abc.def.domain.com'
Still, webmasters use forums.domain.com mostly.
Actually, I can't see if any robots, visits my subdomains, when I see 'Awstats log' for my Web site.
( SEs ask/allow for domain submission, not subdomain submission )

So if domain.com/forums is better than forums.domain.com, they what is the need of subdomains, unless ftp access ( and main domain's images/css need relative-linking protection ) limitations are desired.

Please discuss.

Subdomains are just other domains, there's no difference (to a browser/spider etc) between forums.mydomain.com and www.mydomain.com. To get spiders to index your sub domain, just link to it like you do for any other domain. The more incoming links you have to a domain, the more pages you're going to get spidered/the better you're going to do in search engines for the pages that do get spidered.



EZ Archive Ads Plugin for vBulletin Copyright 2006 Computer Help Forum