CODEWITHSUNDEEP
pythonweb-scrapingscrapycrawlera
Aminah Nuraini
Aminah Nuraini

Reputation: 19186

Is it possible to set different settings for different request in the same Scrapy spider?

I want to use Crawlera only for some requests in a Scrapy spider. So I want to set CRAWLERA_ENABLED differently for different requests. Is it possible?

Upvotes: 1

Views: 84

Answers (1)

Thiago Curvelo
Thiago Curvelo

Reputation: 3740

You can use the dont_proxy key in meta for those requests you don't want to use Crawlera. E.g.

# Supposing you have crawlera enabled in `settings.py`
yield scrapy.Request(
    url, 
    meta={"dont_proxy": True}, 
    callback=self.parse
)

Upvotes: 1

Related Questions