How can I get the web-scraped data into one dataframe?

Question

I am trying to pull all of the tables together. I can grab the first set of data which I think means that the scraping aspect works, however, I think there is an issue when I'm trying to bring all of it together.

I've tried to declare the dataframe early on and then have the table data fill it in every loop.

names = {'Iron-Man',
        'Incredible-Hulk-The',
        'Thor',
        'Iron-Man-2',
        'Captain-America-The-First-Avenger',
        'Avengers-The-(2012)',
        'Iron-Man-3',
        'Thor-The-Dark-World',
        'Captain-America-The-Winter-Soldier',
        'Guardians-of-the-Galaxy',
        'Avengers-Age-of-Ultron',
        'Ant-Man',
        'Captain-America-Civil-War',
        'Doctor-Strange-(2016)',
        'Guardians-of-the-Galaxy-Vol-2',
        'Spider-Man-Homecoming',
        'Thor-Ragnarok',
        'Black-Panther',
        'Avengers-Infinity-War',
        'Ant-Man-and-the-Wasp',
        'Captain-Marvel-(2019)',
        'Avengers-Endgame-(2019)'
         }

This piece of code works for grabbing the pages table

    data = requests.get('https://www.the-numbers.com/movie/Iron-Man#tab=box- office')
    soup = BeautifulSoup(data.text, 'html.parser')

    data = []

    div = soup.find('div' , {'id': 'box_office_chart'})
    table = div.find('table')
    tbody = table.find('tbody')
    html = table.encode().decode('utf8')
    dfs = pd.read_html(html,header=0)
    df = dfs[0]
    df

This piece of code is where I'm expecting it to loop through everything and grab it.

for name in names:
    print(name)
    data = requests.get('https://www.the-numbers.com/movie/' + name + '#tab=box-office')
    soup = BeautifulSoup(data.text, 'html.parser')
    div = soup.find('div' , {'id': 'box_office_chart'})
    table = div.find('table')
    tbody = table.find('tbody')
    html = table.encode().decode('utf8')
    dfs = pd.read_html(html,header=0)
    df2 = dfs[0]
    df2
    df.append(df2)
    print(name)
df

All of the movies printed out twice so I know that it at least went to each page. Here is the output which doesn't include any of the other movies.

df Output:

    Date            Rank    Gross           % Change    Theaters    Per Theaters    Total Gross     Week    movie
0   May 2, 2008     1       $102,118,668    NaN         4105        $24,877         $102,118,668    1       Iron-Man
1   May 9, 2008     1       $51,190,629     -50%        4111        $12,452         $177,825,024    2       Iron-Man
2   May 16, 2008    2       $31,838,996     -38%        4154        $7,665          $223,124,385    3       Iron-Man
3   May 23, 2008    3       $20,447,253     -36%        3915        $5,223          $252,614,669    4       Iron-Man
4   May 30, 2008    4       $13,541,264     -34%        3650        $3,710          $276,166,336    5       Iron-Man
5   Jun 6, 2008     6       $7,477,439      -45%        2931        $2,551          $288,847,640    6       Iron-Man
6   Jun 13, 2008    7       $5,620,375      -25%        2403        $2,339          $297,918,329    7       Iron-Man
7   Jun 20, 2008    9       $4,030,272      -28%        1912        $2,108          $304,816,141    8       Iron-Man
8   Jun 27, 2008    11      $2,257,113      -44%        1379        $1,637          $309,179,318    9       Iron-Man
9   Jul 4, 2008     12      $1,459,613      -35%        1019        $1,432          $311,708,133    10      Iron-Man
10  Jul 11, 2008    14      $939,134        -36%        710         $1,323          $313,421,025    11      Iron-Man
11  Jul 18, 2008    16      $451,838        -52%        375         $1,205          $314,376,968    12      Iron-Man
12  Jul 25, 2008    22      $310,654        -31%        274         $1,134          $314,925,955    13      Iron-Man
13  Aug 1, 2008     16      $580,179        +87%        407         $1,426          $315,687,768    14      Iron-Man
14  Aug 8, 2008     19      $426,502        -26%        45          $1,236          $316,468,817    15      Iron-Man
15  Aug 15, 2008    23      $341,178        -20%        315         $1,083          $317,058,295    16      Iron-Man
16  Aug 22, 2008    29      $243,342        -29%        257         $947            $317,473,452    17      Iron-Man
17  Aug 29, 2008    33      $223,636        -8%         220         $1,017          $317,794,156    18      Iron-Man
18  Sep 5, 2008     38      $126,734        -43%        205         $618            $318,006,770    19      Iron-Man
19  Sep 12, 2008    39      $94,816         -25%        156         $608            $318,134,740    20      Iron-Man
20  Sep 19, 2008    43      $59,037         -38%        124         $476            $318,219,154    21      Iron-Man
21  Sep 26, 2008    48      $58,364         -1%         121         $482            $318,298,180    22      Iron-Man

I keep expecting to have all of the tables from the other pages added to df. Not sure where I'm going wrong.

EDIT: So I got rid of the first attempt at grabbing data and just used a bunch of elif statements to create all 22 dataframes. Thanks to everyone for the suggestions.

How can I get the web-scraped data into one dataframe?

Answers (1)

Related Questions