CODEWITHSUNDEEP
pythonweb-scrapingscrapy
lollercoaster
lollercoaster

Reputation: 16493

scrapy random waits per spider

I'd like to have scrapy use random waits between requests, and have that wait be set per scraper.

Ideally I'd be able to set an average and stddev, something like injecting:

import time
import numpy as np 

avg, stddev = 10, 5
time.sleep(np.random.normal(loc=avg, scale=stddev))

into each web call.

Is this possible in scrapy? Perhaps a custom middleware?

Upvotes: 0

Views: 917

Answers (1)

MrPandav
MrPandav

Reputation: 1861

you should use DOWNLOAD_DELAY AND RANDOMIZE_DOWNLOAD_DELAY

You can set the DOWNLOAD_DELAY and scrapy will randomize it by default i.e. if you set delay to 10 second

Scrapy will use the random wait between each request in a range ( 5 - 15 Seconds)or (Delay-50% to Delay+50%)

To set the delay at Spider level You can use custom_setting-1 , custom_setting-2

custom_settings = {
        'DOWNLOAD_DELAY': 10
    }

Upvotes: 2

Related Questions