CODEWITHSUNDEEP
multithreading
Den
Den

Reputation: 29

How many threads do I need for web scraper?

I'm going to write a web scraper that uses proxies. Having read about std::thread and std::async I'm not sure that will cores be doing while content is being downloaded through a slow proxies. Should there be threads amount equivalent to number of cores or I can make use of additional large amount of threads and they will not be hanging the RAM, waiting for their turn to get some cpu time.

If I should use a larger amount of threads then there are cpu cores, how do I define optimal amount of threads?

Upvotes: 0

Views: 523

Answers (1)

ivv
ivv

Reputation: 26

how do I define optimal amount of threads?

You can try to keep incrementing the number of threads until your scraper's performance (e.g. how many websites are processed per second) stops growing.

Upvotes: 1

Related Questions