CODEWITHSUNDEEP
pythonpython-requestsmultiprocessing
xnarf
xnarf

Reputation: 100

Multiprocessing and lists python

I am here with another simple question in which I really need some help. I have a list of websites and I want to go through them all with requests. However to make this faster, I want to use multi processing. How can I execute this?

Example:

import requests
import threading
from threading import Thread

list_ex = ["www.google.com","www.bing.com","www.yahoo.com"]
def list_request():
  for item in list_ex:
    ex = request.get(list_ex)
    print(ex.text)

How can I do this but with multi processing due to me having 100+ so sites :)

Upvotes: 0

Views: 116

Answers (2)

Niklas Mertsch
Niklas Mertsch

Reputation: 1489

Multithreading is an option here, because fetching a url is not CPU-bound, but I/O-bound. Thus, a single process can have multiple threads running requests.get in parallel.

import requests
from multiprocessing.pool import ThreadPool

def read_url(url):
    return requests.get(url).text


urls = ["www.google.com","www.bing.com","www.yahoo.com"]
with ThreadPool() as pool:
    texts = pool.map(read_url, urls)

You can use multiprocessing by replacing ThreadPool with Pool. For the three provided URLs, I get similar runtimes with Pool and ThreadPool, and both are faster than running read_url sequentially in a loop.

Upvotes: 0

M_x
M_x

Reputation: 867

  • you can use DASK for multiprocessing ..

It will uses all your system cores .. while traditional python uses only one core

Dask Official ..... Dask Wikipedia

Upvotes: 1

Related Questions