CODEWITHSUNDEEP
pythonpython-3.xconcurrent.futures
robots.txt
robots.txt

Reputation: 137

Unable to use pair of items within a function when I go for concurrent.futures

I'm trying to make use of concurrent.futures within my script below. There are two functions make_requests() and get_info() within it. When make_requests() function yields links and proxy_list, I intend to pass them within get_info() function.

Usually this is how it works:

if __name__ == '__main__':
    # "proxy_lists" is the holder of different proxies
    proxies = [proxy_lists]

    # make_requests() function yields "links" and "proxy_list"
    for item,elem in make_requests(url,proxies):

        # pass "links" and "proxy_list" to get_info() function
        get_info(item,elem)

I don't find any idea to do the same using concurrent.futures as it requires a paired loop to produce two type of elements by this make_requests() function and then pass the two type of elements within get_info() function like I did above:

import concurrent.futures

if __name__ == '__main__':
    proxies = [proxy_lists]

    with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor:
        future_to_url = {executor.submit(get_info,link): link for link in make_requests(url,proxies)}
        concurrent.futures.as_completed(future_to_url)

How can I pass pair of items within get_info() function when I use concurrent.futures?

Upvotes: 0

Views: 29

Answers (1)

Sam Mason
Sam Mason

Reputation: 16184

I think you just want to do: executor.submit(get_info, *link)

this is because Executor.submit() is documented as:

submit(fn, *args, **kwargs)

Schedules the callable, fn, to be executed as fn(*args **kwargs)

this uses standard arbitrary argument handling to make sure that everything goes through to fn

the *link treats link as a sequence, unpacking it as arguments to submit, which in turn does then same when it eventually invokes fn

Upvotes: 1

Related Questions