Hello and welcome to our community! Is this your first visit?
Register
Enjoy an ad free experience by logging in. Not a member yet? Register.
Results 1 to 2 of 2
  1. #1
    New to the CF scene
    Join Date
    Apr 2017
    Posts
    1
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Web Scraping Using Python 3.6 and Beautiful Soup

    I am a very new user to Python and could use some help with some code. I am trying to learn how to scrape statistics from various websites using requests, BeautifulSoup, and Python 3.6. My code is listed below.

    The problems I get are as follows:
    1. The first player, Drew Brees, is posted in column A whereas the remainder of players are in column C.
    2. It is printing the position, QB, in column 2, versus after the players name as it is on the website.
    3. I have been unable to get the remainder of statistics to pull in to the spreadsheet - they vary between floats and integers but I would be fine if they were all floats.
    4. There is 2 pages of players - I have no clue how to get it to loop to the next page and grab those players.


    The code is below.

    from bs4 import BeautifulSoup
    import requests

    f = open('C:\\Users\\Schreier Family\\Python\\outfileNFL.csv','w')
    errorFile = open('C:\\Users\\Schreier Family\\Python\\error.csv','w')


    r = requests.get('http://www.nfl.com/stats/categorystats?tabSeq=0&statisticCategory=PASSING&conference=null&season=2016&seasonType=REG&d-447263-s=PASSING_YARDS&d-447263-o=2&d-447263-n=1')
    soup = BeautifulSoup(r.text, "html.parser")
    tableStats = soup.find("table", {"id" : "result"})
    x = 0
    while (x<500):
    for row in tableStats.findAll('tr')[:]:
    col = row.findAll('td')

    try:
    name = col[1].a.string.strip()
    team = col[2].a.string.strip
    position = col[3].string
    stats = col[4]
    f.write(name+'\n')
    f.write(',')
    f.write(position)
    f.write(',')
    f.write(team)

    f.write(str(team))

    f.write(stats)

    except Exception as e:
    errorFile.write(str(x) + '*************' + str(e) + '***************' + str(col) + '\n')
    pass

    x = x + 1

    f.close
    errorFile.close


    Any help would be appreciated as I feel it is somewhat close, at least for the first page.

  2. #2
    New to the CF scene
    Join Date
    Apr 2017
    Posts
    8
    Thanks
    1
    Thanked 0 Times in 0 Posts
    Hello @mtdewrocks,
    I would like to help sent you PN
    Please check.

    Regards,
    Ron Anderson


 

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •