CODEWITHSUNDEEP
python-3.xscrapypython-requests
M. T.
M. T.

Reputation: 396

Using python requests library on Scrapy

How can I use requests on a spider in Scrapy?

import scrapy, requests

def parse(self, response):
    # do things...
    # then

    yield requests.get(response.url, callback=self.parse, dont_filter=True)

Upvotes: 1

Views: 1167

Answers (2)

Arun Augustine
Arun Augustine

Reputation: 1766

Use yield scrapy.Request(response.url, callback=self.parse, dont_filter=True)

Upvotes: 1

Wim Hermans
Wim Hermans

Reputation: 2116

You can use scrapy.Request for this. Use it as follows:

from scrapy.spiders import Spider
from scrapy import Request

def parse(self, response):
    # do things...
    # then

    yield Request(response.url, 
                  callback=self.parse, 
                  dont_filter=True)

Upvotes: 0

Related Questions