Scraping dynamically generated HTML table without an ID and Class with BeautifulSoup

Question

I generated a table using Python Selenium and the source code of the generated table looks like the following (the code has

        
    
                          
        Day End Summary from Jan 03, 2019 to Jan 01, 2019                    

    
      
        
          
            #
          DATE
          TRADING CODE
            LTP*
          HIGH
          LOW
          OPENP*
          CLOSEP*
          YCP
          TRADE
          VALUE (mn)
          VOLUME                
          

          
             1
           2019-01-03
          ACI
          328.5
          329.9
          323
          325.7
          328.4
          325.7
          256
          5.285
          16,158
          

          
             2
           2019-01-02
          ACI
          325.5
          328.8
          324.3
          326
          325.7
          327.6
          289
          7.443
          22,851
          

          
             3
           2019-01-01
          ACI
          329.1
          329.9
          325.6
          329.9
          327.6
          326.2
          153
          3.234
          9,890
          

        



The link of the page is here Dhaka Stock Exchange Historical Price. 
Now, how can I extract the full table from here? The table seems not to have any ID and Class. I am completely new to HTML and web scraping and need to do it for a project that has a very tight schedule. Any kind of help would be appreciated.  

Update: I could extract the required table with the following code-
def extractdse(fdate, tday, instrument):


# load the driver
driver = webdriver.Chrome()

# load DSE data archive site
driver.get('https://www.dsebd.org/day_end_archive.php')

# deal with the pop up warning
try: 
    alert = driver.switch_to.alert
    alert.accept()
except NoAlertPresentException:
    print ("No alert here")

# from date
fdate=datetime.datetime.strptime(fdate, '%Y-%m-%d')
yr=str(fdate.year)
mn='0'+str(fdate.month)
dy='0'+str(fdate.day)

# to date
tday=datetime.datetime.strptime(tday, '%Y-%m-%d')
bdd='0'+str(tday.day)
bdm='0'+str(tday.month)
bdy=str(tday.year)

# beginning day
x=Select(driver.find_element_by_name("DayEndSumDate1_day"))
x.select_by_value(dy)

# beginning month
y=Select(driver.find_element_by_name("DayEndSumDate1_month"))
y.select_by_value(mn)

# beginning year
z=Select(driver.find_element_by_name("DayEndSumDate1_year"))
z.select_by_value(yr)

# end day
xp=Select(driver.find_element_by_name("DayEndSumDate2_day"))
xp.select_by_value(bdd)

# end month
yp=Select(driver.find_element_by_name("DayEndSumDate2_month"))
yp.select_by_value(bdm)

# end year
zp=Select(driver.find_element_by_name("DayEndSumDate2_year"))
zp.select_by_value(bdy)

# instrument
inst=Select(driver.find_element_by_name("Symbol"))
inst.select_by_visible_text(instrument)

# generate the table
driver.find_element_by_name("ViewDayEndArchive").click()

# extract the table
soup = BeautifulSoup(driver.page_source, 'html.parser')
tables = soup.find_all('table')
tt=tables[374]
#ti=pd.read_html(tt)
#tm=display_html(tt, raw=True)
#tx=pd.read_html(tt)
return(tt)



And now can extract the HTML code for the required table with-


extractdse('2019-01-01', '2019-01-03', 'ACI')



Now, is there any way to convert the HTML code into Pandas Dataframe?

Scraping dynamically generated HTML table without an ID and Class with BeautifulSoup

Answers (1)

Related Questions