CODEWITHSUNDEEP
pythonscrapy
Mark
Mark

Reputation: 744

Scrapy finish redirect before crawling next URL

Whenever Scrapy gets a 302, that action is added as the last item in the queue. Is there a way to force Scrapy to finish the redirection and process next urls after that?

Upvotes: 3

Views: 288

Answers (1)

Granitosaurus
Granitosaurus

Reputation: 21436

As stated by Tomáš in the comment REDIRECT_PRIORITY_ADJUST controls redirect priority.

However what you describe with default scrapy settings shouldn't happen since this setting defaults to +2. By default all scrapy requests are scheduled at 0, so all redirected requests should have priority over other requests.

You can schedule indidividual requests priority with priority argument.
for example if you want to set priority at 100, you'd write this: 

yield Request("http://someurl.com", priority=100)

Upvotes: 2

Related Questions