CODEWITHSUNDEEP
pythonseleniumxpathcss-selectorswebdriverwait
Haunter
Haunter

Reputation: 105

How can i scrape the text from a located element using Selenium and Python

I am trying to run the following code

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
import time
options = Options()
options.add_argument("start-maximized")
options.add_argument("disable-infobars")
options.add_argument("--disable-extensions")
driver = webdriver.Chrome(options=options)
driver.get('https://theunderminejournal.com/#eu/draenor/battlepet/1155')
time.sleep(20) #bypass cloudflare
price = driver.find_element_by_xpath('//*[@id="battlepet-page"]/div[1]/table/tr[3]/td/span')
print (price)

so i can scrape the "Current Price" from the page . But this xpath location won't return the text value( i also tried the "text" varriant in the end with no success.

thanks in advance for any reply

Upvotes: 0

Views: 273

Answers (3)

undetected Selenium
undetected Selenium

Reputation: 193298

To scrape the value of Current Price from the webpage you need to induce WebDriverWait for the visibility_of_element_located() and you can use either of the following Locator Strategies:

  • Using CSS_SELECTOR:

    print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "tr.current-price td>span"))).text)

  • Using XPATH:

    print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//th[text()='Current Price']//following::td[1]/span"))).text)

  • Note : You have to add the following imports :

    from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Upvotes: 1

S A
S A

Reputation: 1

First, use WebdriverWait to wait for the element instead of sleep.

Second, Your locator is not finding the element.

Try this, 

driver.get('https://theunderminejournal.com/#eu/draenor/battlepet/1155')
price = WebDriverWait(driver,30).until(EC.visibility_of_element_located((By.XPATH,"//div[@id='battlepet-page']/div/table/tr[@class='current-price']/td/span")))

print(price.text)

To use wait import the followings,

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

Upvotes: 2

Sers
Sers

Reputation: 12255

You should wait for visibility of element before getting text. Check WebDriverWait in example below:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as ec
from selenium.webdriver.support.ui import WebDriverWait
rom selenium.webdriver.chrome.options import Options

options = Options()
options.add_argument("start-maximized")
options.add_argument("disable-infobars")
options.add_argument("--disable-extensions")
driver = webdriver.Chrome(options=options)

wait = WebDriverWait(driver, 20)

driver.get('https://theunderminejournal.com/#eu/draenor/battlepet/1155')
current_price = wait.until(ec.visibility_of_element_located((By.CSS_SELECTOR, ".current-price .price"))).text

print(current_price)

Upvotes: 1

Related Questions