CODEWITHSUNDEEP
pythonpython-2.7web-scrapingbeautifulsouphtml-parsing
ezeagwulae
ezeagwulae

Reputation: 299

scrape data from into dataframe with BeautifulSoup

I'm working on a project to scrape and parse data from California lottery into a dataframe

Here's my code so far, it produces no error but also no output:

import requests
from bs4 import BeautifulSoup as bs4

draw = 'http://www.calottery.com/play/draw-games/superlotto-plus/winning-numbers/?page=1'
page = requests.get(draw) 
soup = bs4(page.text)

drawing_list = []

for table_row in soup.select("table.tag_even_numbers tr"):
    cells = table_row.findAll('td')

    if len(cells) > 0:
        draw_date = cells[0].text.strip()
        numbers = cells[1].text.strip()
        mega = cells[2].text.strip()

        drawings = {'dates': draw_date, 'winning_numbers': numbers, 'mega_number': mega}
        drawing_list.append(drawings)
        print "added {0} {1} {2}, to the list".format(draw_date, numbers, mega)

Expected Output: I'd love to scrape the table rows into a dataframe

draw_date  | numbers        | mega
-----------|----------------|-----
12/06/2017 | 12 24 07 01 02 | 23
12/02/2017 | 33 18 07 42 40 | 7

Thanks for any revision or assistance into the right direction.

Upvotes: 1

Views: 1246

Answers (1)

t.m.adam
t.m.adam

Reputation: 15376

This expression "table.tag_even_numbers tr" selects nothing because the table has no 'tag_even_numbers' class, but has a 'tag_even' class and a 'numbers' class.

So if you change this: 

soup.select("table.tag_even_numbers tr")

to: 

soup.select("table.tag_even.numbers tr")

you should have 20 items in drawing_list.

Also by using .text to select numbers you get all the numbers joined side by side in a string.
If you want a list of numbers you should use .stripped_strings instead, eg: 

numbers = list(cells[1].stripped_strings)

Then you can create a dataframe from drawing_list, eg: 

df = pd.DataFrame(drawing_list)
print(df.head())

                 dates mega_number       winning_numbers
0   Dec 6, 2017 - 3201          23  [12, 24, 07, 01, 02]
1   Dec 2, 2017 - 3200           7  [33, 18, 07, 42, 40]
2  Nov 29, 2017 - 3199           6  [03, 33, 26, 27, 07]
3  Nov 25, 2017 - 3198          19  [21, 46, 13, 25, 17]
4  Nov 22, 2017 - 3197           3  [32, 40, 27, 42, 08]

Upvotes: 1

Related Questions