CODEWITHSUNDEEP
pythonseleniumselenium-webdriver
JamesHudson81
JamesHudson81

Reputation: 2273

Executing event and scrape using selenium

I am pretty new in selenium and I was looking forward to execute a click() button that expands a table, and after that I was looking forward to scrape the content of the expanded table.

This is what I tried:

url = 'https://www.telegraph.co.uk/markets-hub/assets/shares/'
phantomjs_path = '/usr/local/bin/phantomjs'
driver = webdriver.PhantomJS(executable_path=phantomjs_path,service_args=['--load-images=no'])
driver.get(url)

element = driver.find_element_by_xpath("//div[@class='tmg-show-more']")

html_source = driver.page_source
element.click()
driver.quit()
soup = BeautifulSoup(html_source,"html5lib")
tables=soup.find('table',class_="table-static kurser-table")
link=pd.read_html(str(tables),flavor='html5lib',thousands='.',decimal=',',header=0)



print(link)

The current output is as it follows:

raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.ElementNotVisibleException: Message: {"errorMessage":"Element is not currently visible and may not be manipulated",
"request":{"headers":{"Accept":"application/json","Accept-Encoding":"identity","Connection":"close","Content-Length":"81","Content-
Type":"application/json;charset=UTF-8","Host":"127.0.0.1:58447","User-Agent":"Python http auth"},
"httpVersion":"1.1","method":"POST","post":"{\"id\": \":wdc:1556893378248\", \"sessionId\": \"f223cb80-6dae-11e9-b00b-4bb158952171\"}",
"url":"/click","urlParsed":{"anchor":"","query":"","file":"click","directory":"/","path":"/click","relative":"/click","port":"","host":"","password":"","user":"",
"userInfo":"","authority":"","protocol":"","source":"/click","queryKey":{},"chunks":["click"]},"urlOriginal":"/session/f223cb80-6dae-11e9-b00b-4bb158952171/element/:wdc:1556893378248/click"}}
Screenshot: available via screen

Upvotes: 1

Views: 115

Answers (2)

cruisepandey
cruisepandey

Reputation: 29362

You would need to scroll down to let the selenium web driver knows where the element is actually present.

Code : 

driver.get("https://www.telegraph.co.uk/markets-hub/assets/shares/")

wait = WebDriverWait(driver, 10)

wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, 'div.section-box.kurser-table-container')))

element = driver.find_element_by_xpath("//a[@ng-click='stocksShowMore()']")

actions = ActionChains(driver)
actions.move_to_element(element).perform()

element.click()

Imports : 

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC 
from selenium.webdriver.common.action_chains import ActionChains

Upvotes: 1

S A
S A

Reputation: 1823

You are getting ElementNotVisibleException means you are trying to click the element before it is visible.

Also, you are getting the page source before clicking the button. You need to get the source after clicking and loading the new data.

Try the following code before getting the page source. 

wait = WebDriverWait(driver, 20)
element = wait.until(EC.element_to_be_clickable(driver.find_element_by_xpath("//div[@class='tmg-show-more']")))
element.click()
wait.until(EC.invisibility_of_element(element))

To use the WebDriverWait you need to import these

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

Upvotes: 1

Related Questions