CODEWITHSUNDEEP
pythonseleniumxpathphantomjslxml
user7910719
user7910719

Reputation: 105

explicit waits in selenium and phantomjs with python

I'm trying to scrape some data off of this site, and many other "wines" on this site, and am using selenium to do so as its a JS site. however, I'm finding that my code only sometimes works and other times it does not return any values even though nothing is changing.

I think I should use explicit waits with selenium to overcome this challenge, however I'm not sure how to integrate them, so any guidance on doing so would be helpful!

my code is

def ct_content(url):
    browser = webdriver.PhantomJS()
    browser.get(url)
    wait = WebDriverWait(driver, 10)
    html = browser.page_source
    html = lxml.html.fromstring(html)
    try:
        content = html.xpath('//a[starts-with(@href, "list.asp?Table=List")]/text()')
        browser.quit()
        return content
    except:
        browser.quit()
        return False

Thanks!

Upvotes: 1

Views: 957

Answers (2)

Andersson
Andersson

Reputation: 52665

Try to use more specific XPath:

//ul[@class="twin_set_list"]//a/text()

Also there is no need to use lxml. Simply try:

from selenium.webdriver.support.ui import WebDriverWait as wait
from selenium.webdriver.support import expected_conditions as EC

data = [link.get_attribute('textContent') for link in wait(browser, 10).until(EC.presence_of_all_elements_located((By.XPATH, '//ul[@class="twin_set_list"]//a')))]

Upvotes: 2

TitusLucretius
TitusLucretius

Reputation: 181

It looks like you never actually use the implicit wait. This is how I would write script with an explicit wait.

def ct_content(url):
    browser = webdriver.PhantomJS()
    browser.get(url)
    wait = WebDriverWait(browser, 10)
    try:
        content = wait.until(EC.element_to_be_clicable((By.XPATH, '//a[starts-with(@href, "list.asp?Table=List")]')))
        browser.quit()
        return content.text
    except:
        browser.quit()
        return False

Also, the way to set implicit waits is: 

browser.implicitly_wait(10) # seconds

Upvotes: 0

Related Questions