CODEWITHSUNDEEP
pythonweb-scrapingscrapy
user4389295
user4389295

Reputation: 23

update scrapy spider while running

I am using python scrapy package. Is there a way to update scrapy spider settings while the spider is running? I tried telnet console and update a setting by the following:

settings.set('AUTOTHROTTLE_ENABLED', False)

'AUTOTHROTTLE_ENABLED' was True when I started the spider. However, after the above command, the spider behavior didn't change. Did I miss something? Thanks!

Upvotes: 2

Views: 954

Answers (1)

alecxe
alecxe

Reputation: 473763

Scrapy settings are loaded when the engine is fired up and cannot be changed on the fly.

I'm not sure what you are trying to achieve, but you should consider tweaking download_delay spider attribute on the fly.

Besides, looking into autothrottle extension source code can help you come up with more options.

Upvotes: 4

Related Questions