View Full Version : crawler - pointing it where to go

02-13-2010, 10:27 AM
for some reason all boots are repeatedly looking here:
but rarely to other of my pages on that site

So I included this into 'Tools.jsp' page:

<a style ="visibility:hidden" href="../jsp/Galery_list_of_specieses.jsp">dfgsdfg</a>

which points to:

Hope this will make crawlers pick that list and links. m'I right ?

02-13-2010, 11:22 AM
in another words:

How are things done on coding forums , that in a time period of one hour after submiting this post, this post becomes third hit in google, typing in "agrozoo.net" ?

02-16-2010, 12:53 PM
You have a nofollow directive in the page source:

<meta name="robots" content="nofollow" />

This prevents search engines from following any of the links (including your new hidden one), which is why nothing else gets crawled.

You should remove that hidden link, by the way - it's against search engine guidelines.

Codingforums gets crawled and indexed quickly because it's a very popular site that lots of people link to on a regular basis - and because it doesn't block crawlers from accessing pages with rogue meta tags!

02-16-2010, 03:32 PM
jeah I have this nofollow and boot stops in robots txt for a reason. Forinstance:

What does this find ? A page where "phaonia+subventa" was once, and crawler picked it up there at the time, it is still on site but not there, pushed forward.

hence I created
which has no robots stops and presents things to crawlers in clear way.

using this: http://www.auditmypc.com/free-sitemap-generator.asp
to check if google will see the pages I want, and it does according to that tool. Submited pagemap.xml to google a couple of days ago, no hits to ".../jsp/Galery_list_of_specieses" jet ....

02-16-2010, 06:17 PM
just checked what google tool for web masters did with my sitemap.xml , and it says:

670 submited links
23 indexed links

Doest this menan that this is it, or will it process the rest allso in time ?