CODEWITHSUNDEEP
pythonseleniumxpathweb-scrapingweb-crawler
SIM
SIM

Reputation: 22440

Facing issues while scraping data from a table using python with selenium

I've written a script using python in combination with selenium to parse table from a target page which can be reached out following some steps I've tried to describe below for the clarity. It does reach the destination but at the time of scraping data from that table It throws an error showing in the console "Unable to locate element". I tried with online xpath tester to see if it is wrong but I found that the xpath I've used in my script for "td_data" is right. I suppose, what I'm missing here is beyond my knowledge. Hope there is somebody to take a look into it and provide me with a workaround. Btw, the site link is given in my script.

Link to see the html contents for the table: "https://www.dropbox.com/s/kaom5qzk78xndqn/Partial%20Html%20content%20for%20the%20table.txt?dl=0"

Steps to reach the target page which my script is able to maintain:

  1. Selecting "I've read and understand above"
  2. Putting this keyword "pump" in the inputbox located right below "Select medical devices".
  3. Selecting the checkbox "Devices found for "pump".
  4. Finally, pressing the search button

Script I've tried with so far:

from selenium import webdriver
import time

driver = webdriver.Chrome()
driver.get('http://apps.tga.gov.au/Prod/devices/daen-entry.aspx')

driver.find_element_by_id('disclaimer-accept').click()
time.sleep(5)

driver.find_element_by_id('medicine-name').send_keys('pump')
time.sleep(8)

driver.find_element_by_id('medicines-header-text').click()

driver.find_element_by_id('submit-button').click()
time.sleep(7)

for item in driver.find_elements_by_xpath('//div[@class="table-responsive"]'):
    for tr_data in item.find_elements_by_xpath('.//tr'):
        td_data = tr_data.find_element_by_xpath('.//span[@class="hovertext"]//a')
        print(td_data.text)

driver.close()

Upvotes: 0

Views: 368

Answers (1)

James Schinner
James Schinner

Reputation: 1579

Why don't you just do this:

from selenium import webdriver
import time

driver = webdriver.Chrome()
driver.get('http://apps.tga.gov.au/Prod/devices/daen-entry.aspx')

driver.find_element_by_id('disclaimer-accept').click()
time.sleep(5)

driver.find_element_by_id('medicine-name').send_keys('pump')
time.sleep(8)

driver.find_element_by_id('medicines-header-text').click()

driver.find_element_by_id('submit-button').click()
time.sleep(7)

for item in driver.find_elements_by_xpath(
'//table[@id]/tbody/tr/td[@class]/span[@class]/a[@id]'
):
    print(item.text)

driver.close()

Output:

27233
27283
27288
27289
27390
27413
27441
27520
25445
27816
27866
27970
28033
28238
26999
28264
28407
28448
28437
28509
28524
28553
28647
28677
28646

Maybe you want to think about saving the page with driver.page_source, pull out the table, save it as a html file. Then use pandas from html to open the table into a dataframe

Upvotes: 1

Related Questions