Hello and welcome to our community! Is this your first visit?
Register
Enjoy an ad free experience by logging in. Not a member yet? Register.
Results 1 to 5 of 5
  1. #1
    New to the CF scene
    Join Date
    Feb 2018
    Posts
    4
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Scrape a Javascript object from a website in Java

    I am trying to scrape a website in Java, to extract some percentages from a table, which is*this one.

    These percentages are rendered after the HTML source is processed. So we can know these elements are rendered via Javascript, which makes scraping harder (ops, problem)

    So this is the difference between the element BEFORE being rendered:

    Code:
    <div class="user_forecasts" id="57464" />
    and AFTER being rendered:

    Code:
    <div class="user_forecasts" id="57464"> <b>1</b> <div class="percents">61% | 25% | 14%</div> </div>
    Obviously, I wanna get the "61% | 25% | 14%" string, and the rest of percents in the table...

    Well, in fact, yes, it's rendered by*Javascript, and I found the .js file and luckily I found the interesting part:
    Code:
    // ajax user_forecast load - one call if ($('div.user_forecasts').length > 0) { $.ajax({ url: '/vote/percentage', global: false, type: 'GET', data: { a: $('#jornadaq').val() }, success: function(percentages) { perc_obj = eval(percentages); $('div.user_forecasts').each(function(ind, val) { if (ind == 14) { $(this).html("<b>" + perc_obj[ind].value + "</b><div class='percents'>" + perc_obj[ind].porcent + "%" + "</div>"); } else { $(this).html("<b>" + perc_obj[ind].forecast + "</b><div class='percents'>" + perc_obj[ind].local + "% | " + perc_obj[ind].tie + "% | " + perc_obj[ind].visitor + "%" + "</div>"); } }); } }); }

    As you see, it's an AJAX call. I checked if I could get the percentages by pasting this code into the Chrome Developer Virtual Machine, and yes, I got what I wanted: the group of elements which contains the data I need for my program.

    Please look*this ScreenShot (Chrome Developer Virtual Machine)

    The thing is I don't know how should I tell Java to code this XML Http Request and then get this data. What libraries do you recommend for this, and how could I use them especifically for this case?



    Enviado desde mi SM-J530F mediante Tapatalk

  2. #2
    New Coder
    Join Date
    Jan 2016
    Location
    Malaysia
    Posts
    47
    Thanks
    0
    Thanked 0 Times in 0 Posts
    What the red line told you ?

  3. #3
    New to the CF scene
    Join Date
    Feb 2018
    Posts
    4
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by john joe View Post
    What the red line told you ?
    The red line is to make you see the difference between the HTML from Chrome (which is already the actual one, because JavaScript has rendered some elements after the HTML code) and Eclipse Java Console (whose code doesnt include the element I need, because JavaScript code didnt start.

    Enviado desde mi SM-J530F mediante Tapatalk

  4. #4
    New Coder
    Join Date
    Jan 2016
    Location
    Malaysia
    Posts
    47
    Thanks
    0
    Thanked 0 Times in 0 Posts
    No, I mean this line
    Code:
    Scanner s =new Scanner(.........);

  5. #5
    New to the CF scene
    Join Date
    Feb 2018
    Posts
    4
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by john joe View Post
    No, I mean this line
    Code:
    Scanner s =new Scanner(.........);
    It gave me the raw HTML code of the page, with no JavaScript element modifications (So It doesnt include the Object Array I need)

    Enviado desde mi SM-J530F mediante Tapatalk


 

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •