CODEWITHSUNDEEP
pythonweb-scrapingpython-requestspython-multiprocessing
Pro Girl
Pro Girl

Reputation: 952

How to run multiprocessing python request with function with multiple agruments

I'm trying to gather data from a single url using python requests library.

I want to run multiprocessing to speed up the data collection however I get an error when I pass in the argument of my function inside of Pool.

Kindly note I have already read the following previous questions:

a link and a link however none of those answer my question.

How can I run those get request simultaneously passing the 3 mandatory arguments?

Here is my code:

from multiprocessing import Pool
import requests
url = 'http://icanhazip.com'
url_two = 'https://httpbin.org/ip'
url_three = 'https://httpbin.org/get'
start_point = 'a'
start_point_two = 'b'
start_point_three ='c'
ending_point = 'c'
ending_point_two = 'z'
ending_point_three = 'x'


def get_info(url,start_point,ending_point):
    r = requests.get(url)
    html = r.text
    if start_point in html:
        print('Do Something')
    elif ending_point in html:
        print('Do Something else')
   else:
        pass

if __name__ == '__main__':
    with Pool(5) as p:
        print(p.map(get_info, [[url,start_point,ending_point]]))

This is the error I get:

TypeError: get_info() missing 2 required positional arguments: 'start_point' and 'ending_point'

Upvotes: 1

Views: 335

Answers (1)

RomanPerekhrest
RomanPerekhrest

Reputation: 92854

To pass multiple arguments to a target function - use Pool.starmap feature:

In your case it'd look as below:

if __name__ == '__main__':
    with Pool(5) as p:
        print(p.starmap(get_info, [(url, start_point, ending_point),
                               (url_two, start_point_two, ending_point_two),
                               (url_three, start_point_three, ending_point_three),]

Upvotes: 1

Related Questions