Raisul Islam
Raisul Islam

Reputation: 329

Scrapy: stale element reference: element is not attached to the page document

I'm trying to run this code using Scrapy-Selenium. Whenever I'm trying to execute this, selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: element is not attached to the page document error is occurring. I don't know why this is happening! Besides this, I want to scrape all the pages (almost 447 products) from the "product_tab" list but it is only scraping a single page.

import scrapy
from scrapy_selenium import SeleniumRequest
import time
from daraz.settings import *
from scrapy.selector import Selector 


class ProductsSpider(scrapy.Spider):
    name = 'products'
    allowed_domains = ['www.daraz.com.bd']
    start_urls = ['https://www.daraz.com.bd/wow/i/bd/landingpage/flash-sale?spm=a2a0e.home.flashSale.1.735212f797l5Fo&wh_weex=true&wx_navbar_transparent=true&scm=1003.4.icms-zebra-100031732-2896540.OTHER_5530854870_2643759&skuIds=164180152,184533214,129330070,132110291,169152164,131854718,189322173']

    def parse(self, response):
        url = "https://www.daraz.com.bd/wow/i/bd/landingpage/flash-sale?spm=a2a0e.home.flashSale.1.735212f797l5Fo&wh_weex=true&wx_navbar_transparent=true&scm=1003.4.icms-zebra-100031732-2896540.OTHER_5530854870_2643759&skuIds=164180152,184533214,129330070,132110291,169152164,131854718,189322173"
        yield SeleniumRequest(url=url, callback=self.parse_result)

    def parse_result(self, response):

        driver = response.request.meta['driver']


        product_tab = driver.find_elements_by_xpath("//div[@class='item-list-content']/a")

        for product in product_tab:
            
            link = product.find_element_by_xpath("./div/div[2]")
            link.click()
            time.sleep(1)
            html = driver.page_source
            resp = Selector(text=html)

            yield {
                'Title': resp.xpath(".//span[@class='pdp-mod-product-badge-title']/text()").get()
            }

Output

selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: element is not attached to the page document

Upvotes: 0

Views: 57

Answers (0)

Related Questions