CODEWITHSUNDEEP
pythonmultithreading
Incorrect syntax near
Incorrect syntax near

Reputation: 171

Multithreaded crawler in Python

Is it possible to create as many threads to use 100% of CPU and is it really efficient? I'm planning to create a crawler in Python and, in order to make the program efficient, I want to create as many threads as possible, where each thread will be downloading one website. I tried looking up for some information online; unfortunately I didn't find much.

Upvotes: 0

Views: 153

Answers (1)

Snek
Snek

Reputation: 59

You are confusing your terminology, but that is ok. A very high level overview would help.

Concurrency can consist of IO bound (reading and writing from disk, http requests, etc) and CPU bound work (running a machine learning optimization function on a big set of data).

With IO bound work, which is what you are referring to I am assuming, in fact your CPU is not working very hard but rather waiting around for data to come back. Contrast that with multi-processing where you can use multiple core of your machine to do more intense CPU bound work.

That said multi-threading could help you. I would advise to use the asyncio and aiohttp modules for Python. These will help you make sure whilst you are waiting for some response to be returned, the software can continue with other requests.

I use asyncio, aiohttp and bs4 when I need to do some web-scraping.

Upvotes: 1

Related Questions