CODEWITHSUNDEEP
python-3.xscrapy
Takamura
Takamura

Reputation: 475

Is it correct in Scrapy to have multiple parse methods in one spider?

Is it correct in Scrapy to have multiple parse methods in one spider ?

Something looking like:

import scrapy


class FooSpider(scrapy.Spider):
    name = 'foo'
    start_urls = ['https://example.com']

    def parse(self, response):
        ...
        yield {'foo': foo}

    def parse(self, response):
        ...
        yield {'bar': bar}

Upvotes: 1

Views: 759

Answers (1)

SuperUser
SuperUser

Reputation: 4822

No, but you can create different methods and call them from start_requests for example.

import scrapy


class FooSpider(scrapy.Spider):
    name = 'POC'
    start_urls = ['https://scrapingclub.com/exercise/detail_basic/']

    def start_requests(self):
        for url in self.start_urls:
            yield scrapy.Request(url=url, callback=self.get_title, dont_filter=True)
            yield scrapy.Request(url=url, callback=self.get_price, dont_filter=True)

    def get_title(self, response):
        yield {'title': response.xpath('//h3/text()').get()}

    def get_price(self, response):
        yield {'price': response.xpath('//div[@class="card-body"]/h4/text()').get()}

Upvotes: 1

Related Questions