CODEWITHSUNDEEP
scrapysignals
Chris
Chris

Reputation: 1317

Scrapy - idle signal spider running into an error

I'm trying to create a spider, which is running all the time and as soon as it arrives in it's idle state, it should fetch the next url to parse from the database. Unfortunately, I got stack at the very beginning already:

# -*- coding: utf-8 -*-
import scrapy

from scrapy import signals
from scrapy import Spider

import logging

class SignalspiderSpider(Spider):
    name = 'signalspider'
    allowed_domains = ['domain.de']

    yet = False

    def start_requests(self):
        logging.log(logging.INFO, "______ Loading requests")
        yield scrapy.Request('https://www.domain.de/product1.html')

    @classmethod
    def from_crawler(cls, crawler, *args, **kwargs):
        logging.log(logging.INFO, "______ From Crawler")
        spider = spider = super(SignalspiderSpider, cls).from_crawler(crawler, *args, **kwargs)
        crawler.signals.connect(spider.idle, signal=scrapy.signals.spider_idle)
        return spider


    def parse(self, response):
        self.logger.info("______ Finished extracting structured data from HTML")
        pass

    def idle(self):
        logging.log(logging.INFO, "_______ Idle state")
        if not self.yet:
            self.crawler.engine.crawl(self.create_request(), self)
            self.yet = True


    def create_request(self):
        logging.log(logging.INFO, "_____________ Create requests")
        yield scrapy.Request('https://www.domain.de/product2.html?dvar_82_color=blau&cgid=')

and the error that I get:

2019-03-27 21:41:38 [root] INFO: _______ Idle state
2019-03-27 21:41:38 [root] INFO: _____________ Create requests
2019-03-27 21:41:38 [scrapy.utils.signal] ERROR: Error caught on signal handler: <bound method RefererMiddleware.request_scheduled of <scrapy.spidermiddlewares.referer.RefererMiddleware object at 0x7f93bcc13978>>
Traceback (most recent call last):
  File "/home/spidy/Documents/spo/lib/python3.5/site-packages/scrapy/utils/signal.py", line 30, in send_catch_log
    *arguments, **named)
  File "/home/spidy/Documents/spo/lib/python3.5/site-packages/pydispatch/robustapply.py", line 55, in robustApply
    return receiver(*arguments, **named)
  File "/home/spidy/Documents/spo/lib/python3.5/site-packages/scrapy/spidermiddlewares/referer.py", line 343, in request_scheduled
    redirected_urls = request.meta.get('redirect_urls', [])
AttributeError: 'NoneType' object has no attribute 'meta'

What am I doing wrong?

Upvotes: 0

Views: 374

Answers (1)

Stefano Fiorucci - anakin87
Stefano Fiorucci - anakin87

Reputation: 3536

Try with:

def idle(self, spider):
    logging.log(logging.INFO, "_______ Idle state")
    if not self.yet:
        self.yet = True
        self.crawler.engine.crawl(Request(url='https://www.domain.de/product2.html?dvar_82_color=blau&cgid=', callback=spider.parse), spider)

I'm not sure if it is correct to create a request in method spider_idle, passing another method wich makes the request, like you do.

See more at Scrapy spider_idle signal - need to add requests with parse item callback

Upvotes: 1

Related Questions