...

View Full Version : crawler - pointing it where to go



BubikolRamios
02-13-2010, 10:27 AM
for some reason all boots are repeatedly looking here:
http://agrozoo.net/jsp/Tools.jsp
but rarely to other of my pages on that site

So I included this into 'Tools.jsp' page:


<a style ="visibility:hidden" href="../jsp/Galery_list_of_specieses.jsp">dfgsdfg</a>

which points to:
http://agrozoo.net/jsp/Galery_list_of_specieses.jsp

Hope this will make crawlers pick that list and links. m'I right ?

BubikolRamios
02-13-2010, 11:22 AM
in another words:

How are things done on coding forums , that in a time period of one hour after submiting this post, this post becomes third hit in google, typing in "agrozoo.net" ?

Pennimus
02-16-2010, 12:53 PM
You have a nofollow directive in the page source:



<meta name="robots" content="nofollow" />


This prevents search engines from following any of the links (including your new hidden one), which is why nothing else gets crawled.

You should remove that hidden link, by the way - it's against search engine guidelines.

Codingforums gets crawled and indexed quickly because it's a very popular site that lots of people link to on a regular basis - and because it doesn't block crawlers from accessing pages with rogue meta tags!

BubikolRamios
02-16-2010, 03:32 PM
jeah I have this nofollow and boot stops in robots txt for a reason. Forinstance:
http://www.google.si/search?hl=sl&rls=com.microsoft%3Asl&q=phaonia+subventa+agrozoo&meta=&aq=f&oq=

What does this find ? A page where "phaonia+subventa" was once, and crawler picked it up there at the time, it is still on site but not there, pushed forward.

hence I created
http://agrozoo.net/jsp/Galery_list_of_specieses.jsp
which has no robots stops and presents things to crawlers in clear way.

using this: http://www.auditmypc.com/free-sitemap-generator.asp
to check if google will see the pages I want, and it does according to that tool. Submited pagemap.xml to google a couple of days ago, no hits to ".../jsp/Galery_list_of_specieses" jet ....

BubikolRamios
02-16-2010, 06:17 PM
just checked what google tool for web masters did with my sitemap.xml , and it says:

670 submited links
23 indexed links

Doest this menan that this is it, or will it process the rest allso in time ?



EZ Archive Ads Plugin for vBulletin Copyright 2006 Computer Help Forum