Extract table from html file using python

Question

I want to extract table from an html file. I have written the following code-snippet to extract the first table:

import urllib2
import os
import time
import traceback
from bs4 import BeautifulSoup

#find('table',{'class':'tbl_with_brdr'})
outfile= open('D:/Dropbox/Python/apelec.txt','wb')
rfile  = open('D:/Dropbox/PRI/Data/AP/195778.html')
rsoup  = BeautifulSoup(rfile)
nodes  = rsoup.find('div',{'class':'frmtext'}).find('table').find('tr')
for node in nodes[1:]:
    x = node.find('th').find('b').get_text().encode("utf-8")
    print x 
    y   = node.find('th').findNext('th').find('b').get_text().encode("utf-8")
    print y
    outfile.write(str(x)+"	"+str(y)+"
")
outfile.close()

Here is the error:

     9 rfile  = open('D:/Dropbox/PRI/Data/AP/195778.html')
     10 rsoup  = BeautifulSoup(rfile)
---> 11 nodes  = rsoup.find('div',{'class':'frmtext'}).find('table').find('tr')
     12 for node in nodes[1:]:
     13     x = node.find('th').find('b').get_text().encode("utf-8")
     AttributeError: 'NoneType' object has no attribute 'find'

And the html file is:







consolidated Election Report





                                      



   
    



 
 
 
 
 
 



       
 
 
                Area Profiler
          
        
    
    
    
      
         
            
              Home 
            

            
          
        
      
        
        
          
            
              Choose Theme :
              
                
                    
                    

              
              
              

            
               
              
               
              
              
              Select Language :
              
                
                
                 
              
              
            
          
      
        










    
      
        

          
                .
             















            
                
                
                   
                       Consolidated Election
                       
                      
              
   
                



                
                        
                            State Name    
                            Local Body Type   
                            Election Term 

                                Local Body Name   

                        
                        
                            ANDHRA PRADESH 
                            Village Panchayat   
                            

                                        02-Aug-2013   To  

                                         01-Aug-2018
                               

                                KODIHALLI    

                        
                    

Consolidated Election
        0
                                   1
                          2
                                 3
                          4
                             5
                           6
                               7
                            8
        
    
        
                        S.No.
                        Name Age Caste Category Gender Qualification Occupation Email Address Ward Name Reservation
    

    

                             

                1
                Kambanna
                36
                OBC
                Male
                Middle or Lower Secondary
                N/A
                


                            N/A

                
                N/A
                

                     Yes  (OBC / Others)

                           
            

                

                2
                Ramesh
                39
                OBC
                Male
                Middle or Lower Secondary
                Workers not reporting any occupations
                

                            N/A

                
                Ward no 1
                

                                Yes (OBC / Others)

                           
            

                             

                3
                S.Manjunath
                29
                OBC
                Male
                Higher Secondary or Intermediate or Pre University or Senior Secondary
                Workers not reporting any occupations
                

                            N/A

                
                Ward no 2
                

                                No (General / Others)

                           
            

                

                4
                Obuleshu
                48
                OBC
                Male
                Below Primary
                Workers not reporting any occupations
                


                            N/A

                
                Ward no 3
                

                                No (General / Others)

                           
            

                             

                5
                Mamatha
                24
                OBC
                Female
                Matriculation or Junior School Certificate or Secondary
                N/A
                

                            N/A

                
                Ward no 4
                

                                Yes (General / Female)

                           
            

                

                6
                Shivamma
                38
                OBC
                Female
                Below Primary
                N/A
                


                            N/A

                
                Ward no 5
                

                                Yes (General / Female)

                           
            

                             

                7
                Hanumantappa
                46
                SC
                Male
                Illiterate
                N/A
                


                            N/A

                
                Ward no 6
                

                                No (General / Others)

                           
            

                

                8
                Malingappa
                45
                SC
                Male
                Illiterate
                N/A
                


                            N/A

                
                Ward no 7
                

                                No (General / Others)

                           
            
                             

                9
                Kamalamma
                52
                OBC
                Female
                Illiterate
                N/A
                


                            N/A

                
                Ward no 8
                

                                Yes (OBC / Female)

                           
            


                


                10
                Muddamma
                48
                OBC
                Female
                Illiterate
                N/A
                


                            N/A

                
                Ward no 9
                

                                Yes (General / Female)

                           
            


                             

                11
                Patta Tayamma
                45
                SC
                Female
                Middle or Lower Secondary
                N/A
                


                            N/A

                
                Ward no 10
                

                                Yes (SC / Female)

                           
            


                


                12
                Sujatha
                35
                OBC
                Female
                Middle or Lower Secondary
                N/A
                


                            N/A

                
                Ward no 11
                

                                Yes (OBC / Female)

                           
            



                             

                13
                Kadurappa
                35
                SC
                Male
                Middle or Lower Secondary
                N/A
                


                            N/A

                
                Ward no 12
                

                                Yes (SC / Others)

                           
            

    





      
          
               


                    

          
      
  




    
    

               


        
            Report generated through Area Profiler (http://areaprofiler.gov.in)Thu Oct 02 22:34:20 IST 2014

        






          
        
      
    
  

 











         
          
        
        

          
          

          Site is designed, hosted
                and maintained by National Informatics Centre
 Contents on
                this website is owned,updated and managed by the Ministry of
                Panchayati Raj

rafalopez79 · Accepted Answer

I paste here an approach, it is not exactly the solution but you can use it as a guide. You have to traverse the DOM tree and extract the values you want. I changed the class of the div you look for from frmtext to frmtxt and in the traversal you have to check if anything is found or not.

import urllib2
import os
import time
import traceback
from bs4 import BeautifulSoup


outfile= open('out.txt','wb')
rfile  = open('195778.html')
rsoup  = BeautifulSoup(rfile)
nodes1  = rsoup.find('div',{'class':'frmtxt'})
nodes = nodes1.find('table').find_all('tr')
for node in nodes:
a = node.find('th') 
x = None   
if a != None:
    x1 = x.find('b')
    if x1 != None:
        x2 = x1.get_text().encode("utf-8")        
        print x2 
        x = x2
y = node.find('th')
if y != None: 
    print 'y',y
    y2  = y.findNext('th')
    if y2 != None:
        print 'y2',y2
        y3 = y2.find('b')
        if y3 != None:
            y = y3.get_text().encode("utf-8")               
            print y
outfile.write(str(x)+"	"+str(y)+"
")
outfile.close()

Extract table from html file using python

Answers (1)

Related Questions