CODEWITHSUNDEEP
pythonseleniumselenium-webdriver
Srikanth
Srikanth

Reputation: 29

Python Selenium Webdriver - Retrieving Sequential Data from a Website

I'm trying to retrieve a list of names of doctors, their addresses and other fields from https://www.bcbsga.com/health-insurance/provider-directory/searchcriteria. I've attached my code below. I'm not sure what is wrong in the 'for' loop at the end, I get something like the output below rather than the data presented on the site.

selenium.webdriver.remote.webelement.WebElement (session="ba8c85ad6e398939063d5b43ee513668", element="0.6436841330730552-19")> selenium.webdriver.remote.webelement.WebElement (session="ba8c85ad6e398939063d5b43ee513668", element="0.6436841330730552-20")>

import time
from selenium import webdriver
from selenium.webdriver.common.keys import Keys

divID = 'ctl00_MainContent_maincontent_PFPlanQuestionnaire_ddlQuestionnaireInsurance_chosen'
inputID = 'ctl00_MainContent_maincontent_PFPlanQuestionnaire_ddlQuestionnaireInsurance_chosen_input'
inputValue = 'I buy it myself (or plan to buy it myself)'

driver = webdriver.Chrome()

driver.get("https://www.bcbsga.com/health-insurance/provider-directory/searchcriteria")
driver.find_element_by_id("btnGuestContinue").click()
driver.implicitly_wait(10)

driver.find_element_by_id('ctl00_MainContent_maincontent_PFPlanQuestionnaire_ddlQuestionnaireInsurance_chosen').click()
driver.find_element_by_id('ctl00_MainContent_maincontent_PFPlanQuestionnaire_ddlQuestionnaireInsurance_chosen_input').send_keys('I buy it myself (or plan to buy it myself)')
driver.find_element_by_id('ctl00_MainContent_maincontent_PFPlanQuestionnaire_ddlQuestionnaireInsurance_chosen_input').send_keys(Keys.RETURN)

driver.find_element_by_id("ctl00_MainContent_maincontent_PFPlanQuestionnaire_ddlQuestionnaireWhobought_chosen").click()
driver.find_element_by_id("ctl00_MainContent_maincontent_PFPlanQuestionnaire_ddlQuestionnaireWhobought_chosen_input").send_keys("Marketplace/exchanges")
driver.find_element_by_id("ctl00_MainContent_maincontent_PFPlanQuestionnaire_ddlQuestionnaireWhobought_chosen_input").send_keys(Keys.RETURN)

driver.find_element_by_id("ctl00_MainContent_maincontent_PFPlanQuestionnaire_ddlQuestionnaireState_chosen").click()
driver.find_element_by_id("ctl00_MainContent_maincontent_PFPlanQuestionnaire_ddlQuestionnaireState_chosen_input").send_keys("Georgia")
driver.find_element_by_id("ctl00_MainContent_maincontent_PFPlanQuestionnaire_ddlQuestionnaireState_chosen_input").send_keys(Keys.RETURN)

driver.find_element_by_id("ctl00_MainContent_maincontent_PFPlanQuestionnaire_ddlCareTypes_chosen").click()
driver.find_element_by_id("ctl00_MainContent_maincontent_PFPlanQuestionnaire_ddlCareTypes_chosen_input").send_keys("Medical")
driver.find_element_by_id("ctl00_MainContent_maincontent_PFPlanQuestionnaire_ddlCareTypes_chosen_input").send_keys(Keys.RETURN)

driver.find_element_by_id("ctl00_MainContent_maincontent_PFPlanQuestionnaire_ddlquestionnairePlanNetworks_chosen").click()
driver.find_element_by_id("ctl00_MainContent_maincontent_PFPlanQuestionnaire_ddlquestionnairePlanNetworks_chosen_input").send_keys("All plans/networks selection")
driver.find_element_by_id("ctl00_MainContent_maincontent_PFPlanQuestionnaire_ddlquestionnairePlanNetworks_chosen_input").send_keys(Keys.RETURN)

driver.find_element_by_id("btnGuestQuestionnaireContinue").click()

#'located near' field
driver.find_element_by_id("ctl00_MainContent_maincontent_pfSearchMain_pfLocationSearch_txtZipCityState").send_keys("30303")

#within a distance of
driver.find_element_by_id("ctl00_MainContent_maincontent_pfSearchMain_pfLocationSearch_ddlRadius_chosen").click()
driver.find_element_by_id("ctl00_MainContent_maincontent_pfSearchMain_pfLocationSearch_ddlRadius_chosen_input").send_keys("10 miles")
driver.find_element_by_id("ctl00_MainContent_maincontent_pfSearchMain_pfLocationSearch_ddlRadius_chosen_input").send_keys(Keys.RETURN)
#speciality
driver.find_element_by_id("ctl00_MainContent_maincontent_pfSearchMain_ddlSpeciality_chosen").click()
driver.find_element_by_id("ctl00_MainContent_maincontent_pfSearchMain_ddlSpeciality_chosen_input").send_keys("Cardiology")
driver.find_element_by_id("ctl00_MainContent_maincontent_pfSearchMain_ddlSpeciality_chosen_input").send_keys(Keys.RETURN)
#click the search button
driver.find_element_by_id("btnSearch").click()
time.sleep(10)

for i in range(10):
        name = driver.find_element_by_id("ctl00_ResultsList_ctrl" + str(i) + "_lnkName")
        address = driver.find_element_by_id("ctl00_ResultsList_ctrl" + str(i) + "_lblAddress")
        print(name, address)

driver.close()

Upvotes: 0

Views: 135

Answers (2)

Srikanth
Srikanth

Reputation: 29

I guess I left out the .text field in the 'for' loops at the end. Adding them retrieves the names of the elements! 

for i in range(10):
        name = driver.find_element_by_id("ctl00_ResultsList_ctrl" + str(i) + "_lnkName").text
        address = driver.find_element_by_id("ctl00_ResultsList_ctrl" + str(i) + "_lblAddress").text
        print(name, address)

Upvotes: 1

Andersson
Andersson

Reputation: 52675

The problem is that find_element_by_id should return web-element, but not text node, so obviously name and address are web-elements. You should use

for i in range(10):
    name = driver.find_element_by_id("ctl00_ResultsList_ctrl" + str(i) + "_lnkName").text
    address = driver.find_element_by_id("ctl00_ResultsList_ctrl" + str(i) + "_lblAddress").text
    print(name, address)

Upvotes: 1

Related Questions