CODEWITHSUNDEEP
pythonseleniumweb-scrapingweb-crawlerscrapy
dkx22
dkx22

Reputation: 1133

Scrapy not entering parse method

I don't understand why this code is not entering the parse method. It is pretty similar to the basic spider examples from the doc: http://doc.scrapy.org/en/latest/topics/spiders.html And I'm pretty sure this worked earlier in the day... Not sure if I modified something or not..

from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium import webdriver
from scrapy.spider import Spider
from scrapy.selector import HtmlXPathSelector
from scrapy import log
from scrapy.selector import Selector


class jobSpider(Spider):
    name='jobSpider'
    wd = webdriver.Chrome()
    wd.get("some url")
    wd.switch_to_frame("cible")


def parse(self, response):
    log.start()

    wait = WebDriverWait(wd, 10).until(
    (EC.visibility_of_element_located((By.ID, 'blocResultat'))))
    print(wd.page_source)

    stuff=Selector(text=wd.page_source).xpath('//a[contains(@onclick,"win=window.o    pen(\'JobOffs")]').extract() 
    print(stuff)

Upvotes: 3

Views: 1717

Answers (2)

Morgan Thrapp
Morgan Thrapp

Reputation: 9986

Your parse(self, response): method is not part of the jobSpider class. If you look at the Scrapy documentation you'll see that the parse method needs to be a method of your spider class.

from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium import webdriver
from scrapy.spider import Spider
from scrapy.selector import HtmlXPathSelector
from scrapy import log
from scrapy.selector import Selector


class jobSpider(Spider):
    name='jobSpider'
    wd = webdriver.Chrome()
    wd.get("https://www.leforem.be/particuliers/offres-emploi-recherche-par-criteres.html?exParfullText=&exPar_search_=true&exParGeographyEdi=true")
    wd.switch_to_frame("cible")


    def parse(self, response):
        log.start()

        wait = WebDriverWait(wd, 10).until(
        (EC.visibility_of_element_located((By.ID, 'blocResultat'))))
        print(wd.page_source)

        stuff=Selector(text=wd.page_source).xpath('//a[contains(@onclick,"win=window.o    pen(\'JobOffs")]').extract() 
        print(stuff)

Also, you need to reference the class data in your parse method by using the self. prefix on any data in the class.

Additionally, you're missing the start_urls list on your spider. Without it, the spider won't know where to start, and will do nothing.

Upvotes: 2

Nizam Mohamed
Nizam Mohamed

Reputation: 9220

You have to access attributes with self and parse is part of the class. 

def parse(self, response):
    log.start()
    wait = WebDriverWait(self.wd, 10).until(
    (EC.visibility_of_element_located((By.ID, 'blocResultat'))))
    print(self.wd.page_source)
    stuff=Selector(text=self.wd.page_source).xpath('//a[contains(@onclick,"win=window.o    pen(\'JobOffs")]').extract() 
    print(stuff)

Upvotes: 1

Related Questions