CODEWITHSUNDEEP
python-3.xseleniumselenium-chromedriverwebdriverwait
user13301749
user13301749

Reputation:

how to do scraping for the no.of elements having the same div using python selenium

I am trying to scrape all the matches in this website

https://web.bet9ja.com/Sport/OddsToday.aspx?IDSport=590

The requirement which I need is

1.click the match name for example kuttosh kujand in the link which I shared and scrape the data and again navigate back and again click the other match name and the process should be done for the matches that are present in the link

So far I wrote a code like this with my code I am able to do the process which I mentioned above one for match but how can I get the process be done for all the matches

The code I wrote:

# Here using selenium for scraping
# importing necessary modules
import selenium.webdriver
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
import mysql.connector
import pymysql
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

# the relevant url
url = 'https://web.bet9ja.com/Sport/OddsToday.aspx?IDSport=590'

# the driver path
driver = webdriver.Chrome(r"c:/Users/SATYA/mysite/chromedriver")
driver.get(url)
driver.implicitly_wait(10) # seconds
buttons = WebDriverWait(driver,15).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, "div.Event.ng-binding")))
for btn in buttons:
    btn.click()
    headings= [item.text for item in driver.find_elements_by_css_selector("div.SECQ.ng-binding")]
    keys = [item.text for item in driver.find_elements_by_css_selector("div.SEOdd.g1")]
    values = [item.text for item in driver.find_elements_by_css_selector("div.SEOddLnk.ng-binding")]
    driver.execute_script("window.history.go(-1)")
    print(headings,keys,values)

Can anyone please help me with this

after scraping first match data I am getting this error with the code which I wrote

Traceback (most recent call last):
  File "dynamicscrape.py", line 21, in <module>
    btn.click()
  File "C:\Users\SATYA\AppData\Local\Programs\Python\Python37\lib\site-packages\selenium\webdriver\remote\webelement.py", line 80, in click
    self._execute(Command.CLICK_ELEMENT)
  File "C:\Users\SATYA\AppData\Local\Programs\Python\Python37\lib\site-packages\selenium\webdriver\remote\webelement.py", line 633, in _execute
    return self._parent.execute(command, params)
  File "C:\Users\SATYA\AppData\Local\Programs\Python\Python37\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute
    self.error_handler.check_response(response)
  File "C:\Users\SATYA\AppData\Local\Programs\Python\Python37\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: element is not attached to the page document
  (Session info: chrome=81.0.4044.113)

Upvotes: 2

Views: 48

Answers (1)

KunduK
KunduK

Reputation: 33384

stale element reference: element is not attached to the page document error comes when element is not attached to the page you have already captured since you have refreshed the page.

To overcome this problem you need to re-assigned the elements again to avoid stale.

buttons = WebDriverWait(driver,15).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, "div.Event.ng-binding")))
for btn in range(len(buttons)):
    #elements re-assigned again to avoid stale.
    buttons = WebDriverWait(driver, 15).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, "div.Event.ng-binding")))
    buttons[btn].click()
    headings= [item.text for item in driver.find_elements_by_css_selector("div.SECQ.ng-binding")]
    keys = [item.text for item in driver.find_elements_by_css_selector("div.SEOdd.g1")]
    values = [item.text for item in driver.find_elements_by_css_selector("div.SEOddLnk.ng-binding")]
    driver.execute_script("window.history.go(-1)")
    print(headings,keys,values)

If clause.

buttons = WebDriverWait(driver,15).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, "div.Event.ng-binding")))
for btn in range(len(buttons)):
    buttons = WebDriverWait(driver, 15).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, "div.Event.ng-binding")))
    if (btn==1) or (btn==3) or (btn==4):
        buttons[btn].click()
        headings= [item.text for item in driver.find_elements_by_css_selector("div.SECQ.ng-binding")]
        keys = [item.text for item in driver.find_elements_by_css_selector("div.SEOdd.g1")]
        values = [item.text for item in driver.find_elements_by_css_selector("div.SEOddLnk.ng-binding")]
        driver.execute_script("window.history.go(-1)")
        print(headings,keys,values)

Upvotes: 1

Related Questions