Hello and welcome to our community! Is this your first visit?
Register
Enjoy an ad free experience by logging in. Not a member yet? Register.
Results 1 to 7 of 7
  1. #1
    Regular Coder
    Join Date
    Nov 2002
    Location
    Carmel California
    Posts
    471
    Thanks
    0
    Thanked 1 Time in 1 Post

    web crawlers with dynamic content

    How do the web crawlers of the main search engines around today act with dynamically generated content. I have a page were the navigation is dynamically generated with javascript, and I am wondering- do the web crawlers actually pull url's out of scripts, or only out of links in the actual body
    Kris Hubby
    kwhubby site

  • #2
    Kor
    Kor is offline
    Red Devil Mod Kor's Avatar
    Join Date
    Apr 2003
    Location
    Bucharest, ROMANIA
    Posts
    8,478
    Thanks
    58
    Thanked 379 Times in 375 Posts
    How do the web crawlers of the main search engines around today act with dynamically generated content
    ... they don't. As far as I know. If anyone knows otherwise, I'll be glad to hear, I am also interested in subject.
    KOR
    Offshore programming
    -*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*

  • #3
    Regular Coder
    Join Date
    Nov 2002
    Location
    Carmel California
    Posts
    471
    Thanks
    0
    Thanked 1 Time in 1 Post
    if they dont pull urls out of scripts, would a method for having the search engine crawl those urls to also have the urls as links in a hidden div
    Kris Hubby
    kwhubby site

  • #4
    Kor
    Kor is offline
    Red Devil Mod Kor's Avatar
    Join Date
    Apr 2003
    Location
    Bucharest, ROMANIA
    Posts
    8,478
    Thanks
    58
    Thanked 379 Times in 375 Posts
    I don't know... As far as I know, the search engines search for words as text and ignore words in code, so... If have a JavaScript dynamically generated page is somehow the same to have a CGI generated page... I mean a virtual page, from the crowler's point of view... I don't know much...
    KOR
    Offshore programming
    -*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*

  • #5
    Regular Coder
    Join Date
    Jun 2002
    Posts
    185
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Crawlers will look at what the web server returns. In the case of CGI or any server-side scripting like ASP, the web server executes the code and outputs an HTML page. A crawler will see that output, not the actual code that generated it. In other words, it sees what you see when you load a page in a browser and select "View Source."

    In the case of client-side scripts, the crawler will only see the raw code embedded in the HTML. I don't know of any that can actually execute the code as the browser does. In fact, that would be rather difficult as many times code runs off user-driven events like clicking on links.

    A crawler could conceivably scan the javascript code and look for strings that appear to be URLs but I imagine most don't. As an example, I might have plain text on a page like "http://www.abc.net/test.html" which is a URL but is not an actual link. A search engine crawler should ignore that. Instead it should only look at URLs within the href part of A tags.

    If you have links generated in client-side script (that don't appear elsewhere on the page in static HTML), you can place those same links inside a <noscript></noscript> tag pair to make then visible to crawlers. That has the added advantage of displaying the links to any users with scripting turned off on their browser.

  • #6
    Rockstar Coder
    Join Date
    Jun 2002
    Location
    USA
    Posts
    9,074
    Thanks
    1
    Thanked 328 Times in 324 Posts
    Originally posted by BrainJar
    If you have links generated in client-side script (that don't appear elsewhere on the page in static HTML), you can place those same links inside a <noscript></noscript> tag pair to make then visible to crawlers. That has the added advantage of displaying the links to any users with scripting turned off on their browser.
    That is a good solution.
    OracleGuy

  • #7
    Regular Coder
    Join Date
    Nov 2002
    Location
    Carmel California
    Posts
    471
    Thanks
    0
    Thanked 1 Time in 1 Post
    ah, thank you! thats a good idea.... now also..... what about frames--- I know some crawlers dont function at all with them, and some do, but the question is is which do
    Kris Hubby
    kwhubby site


  •  

    Posting Permissions

    • You may not post new threads
    • You may not post replies
    • You may not post attachments
    • You may not edit your posts
    •