How to parse this html structure using BeautifulSoup?

Question

I would like to parse this TABLE line by line and save to a csv file. What I have done so far, return nothing in the csv file:

Django:

data_scrapper makes a request from Yahoo Finance.

def button_clicked(request):
    headers = []
    rows = []
    gen_table = data_scrapper(symbol)
    soup = BeautifulSoup(gen_table)
    table = soup.find_all('table')
    for table in soup.find_all('table'):
        headers.extend([header.text for header in table.find_all('th')])
    for row in soup.find_all('tr'):
        rows.extend([val.text for val in row.find_all('td')])

    response = HttpResponse(content_type='text/csv')
    response['Content-Disposition'] = 'attachment; filename= "{}.csv"'.format(symbol)

    writer = csv.writer(response)
    writer.writerow(headers)
    writer.writerows(row for row in rows if row)

    return response

html:


  
    
      
        
          
            
              Period Ending
            
          
          Dec 31, 2014
          Dec 31, 2013
          Dec 31, 2012
        
        
          
            
              Total Revenue
            
          
          
            
              4,479,648  
            
          
          
            
              3,777,068  
            
          
          
            
              3,209,782  
            
          
        
        
          Cost of Revenue
          3,160,470  
          2,656,189  
          2,284,485  
        
        
          
            
          
        
        
          
            
              Gross Profit
            
          
          
            
              1,319,178  
            
          
          
            
              1,120,879  
            
          
          
            
              925,297  
            
          
        
        
          
            
          
        
        
          
            
          
          Operating Expenses
        
        
          
            
          
          Research Development
          148,458  
          139,193  
          127,361  
        
        
          
            
          
          Selling General and Administrative
          456,030  
          403,772  
          319,511  
        
        
          
            
          
          Non Recurring
          
            -  
          
          
            -  
          
          
            -  
          
        
        
          
            
          
          Others
          
            -  
          
          
            -  
          
          
            -  
          
        
        
          
            
          
          
            
          
        
        
          
            
          
          Total Operating Expenses
          
            -  
          
          
            -  
          
          
            -  
          
        
        
          
            
          
        
        
          
            
          
        
        
          
            
              Operating Income or Loss
            
          
          
            
              714,690  
            
          
          
            
              577,914  
            
          
          
            
              478,425  
            
          
        
        
          
            
          
        
        
          
            
          
          Income from Continuing Operations
        
        
          
            
          
          Total Other Income/Expenses Net
          (10)
          5,139  
          7,529  
        
        
          
            
          
          Earnings Before Interest And Taxes
          710,556  
          580,639  
          485,775  
        
        
          
            
          
          Interest Expense
          11,239  
          6,210  
          5,932  
        
        
          
            
          
          Income Before Tax
          699,317  
          574,429  
          479,843  
        
        
          
            
          
          Income Tax Expense
          245,288  
          193,360  
          167,533  
        
        
          
            
          
          Minority Interest
          
            -  
          
          
            -  
          
          
            -  
          
        
        
          
            
          
          
            
          
        
        
          
            
          
          Net Income From Continuing Ops
          454,029  
          381,069  
          312,310  
        
        
          
            
          
        
        
          
            
          
          Non-recurring Events
        
        
          
            
          
          Discontinued Operations
          
            -  
          
          (3,777)
          
            -  
          
        
        
          
            
          
          Extraordinary Items
          
            -  
          
          
            -  
          
          
            -  
          
        
        
          
            
          
          Effect Of Accounting Changes
          
            -  
          
          
            -  
          
          
            -  
          
        
        
          
            
          
          Other Items
          
            -  
          
          
            -  
          
          
            -  
          
        
        
          
            
          
        
        
          
            
          
        
        
          
            
              Net Income
            
          
          
            
              454,029  
            
          
          
            
              377,292  
            
          
          
            
              312,310  
            
          
        
        
          Preferred Stock And Other Adjustments
          
            -  
          
          
            -  
          
          
            -  
          
        
        
          
            
          
        
        
          
            
              Net Income Applicable To Common Shares
            
          
          
            
              454,029  
            
          
          
            
              377,292  
            
          
          
            
              312,310

How to parse this html structure using BeautifulSoup?

Answers (1)

Related Questions