CODEWITHSUNDEEP
pythonselenium-webdriverxpathscreen-scraping
mshahwan
mshahwan

Reputation: 3

web element exists in page, text content but not scraped by selenium

I'm tryng to scrape the (number of last page) by XPATH, the problem is when i search by xpath in developer i find the element and its text. However, when scraping by selenium, it returns empty string ''.

url = 'https://www.audible.com/search
driver = webdriver.Chrome(service=service, options = options)
driver.get(url)
all_pages =   WebDriverWait(driver,50).until(EC.presence_of_all_elements_located((By.XPATH,'//ul[contains(@class,"pagingElements ")]/li')))

last_page = all_pages[-2]
int (last_page.get_attribute('textContent'))`

Upvotes: 0

Views: 26

Answers (1)

Yaroslavm
Yaroslavm

Reputation: 4804

You should scroll down, using ActionChains and only then, when pagination is rendered, get needed element attribute. Also it's better to get innerText of a than textContent of li as far as textContent contain line separators.

from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver import ActionChains

actionChains = ActionChains(driver)
wait = WebDriverWait(driver, 10)

url = 'https://www.audible.com/search'
driver.get(url)
footer = wait.until(EC.presence_of_element_located((By.CLASS_NAME,'ui-it-footer-wrapper')))
actionChains.move_to_element(footer).perform()
all_pages = wait.until(EC.presence_of_all_elements_located((By.XPATH,'//ul[contains(@class,"pagingElements ")]//a')))

last_page_attrubute = all_pages[-2].get_attribute('innerText')

Upvotes: 0

Related Questions