CODEWITHSUNDEEP
pythonconcurrencyscrapyrequestsimultaneous
jeremyl
jeremyl

Reputation: 135

send multiple requests simultaneously using CONCURRENCY_REQUEST using Scrapy

Please, How to configure exactly the CONCURRENCY_REQUEST, CONCURRENCY_REQUEST_PER_DOMAIN for to send multiple requests simultaneously ? My DOWNLOAD_DELAY = 5 and CONCURRENCY_REQUEST_PER_IP = 1

When I set CONCURRENCY_REQUEST = 4 there is no difference in time and I don't understand

Upvotes: 1

Views: 282

Answers (1)

Georgiy
Georgiy

Reputation: 3561

You mistyped name of settings
there is no CONCURRENCY_REQUEST , CONCURRENCY_REQUEST_PER_DOMAIN and CONCURRENCY_REQUEST_PER_IP settings in scrapy.

You need to use CONCURRENT_REQUESTS , CONCURRENT_REQUESTS_PER_DOMAIN
CONCURRENT_ not CONCURRENCY_

Upvotes: 1

Related Questions