CODEWITHSUNDEEP
pythonmultithreadingmultiprocessing
Exceen
Exceen

Reputation: 765

Running two methods (with arguments) simultaneously?

I have a Python-script where I want to run two methods simultaneously. I tried the following, but unfortunately, threads/processes only run simultaneously if there is no argument. So how can I solve this problem?

from multiprocessing import Process
def download_from_file(filename):
    if os.path.exists(filename):
        links = [line.strip() for line in open(filename)]

        for link in links:
            Process(target = standard_download(link)).start()

Upvotes: 0

Views: 377

Answers (2)

abarnert
abarnert

Reputation: 365707

but unfortunately, threads/processes only run simultaneously if there is no argument.

Nope, that's not the problem.

The problem is that you're calling standard_download(link), then passing the result of that call as the target argument. That's not what you want; you want to pass the function itself, and the arguments, to the Process object so that it can call them. That's exactly what the args parameter is for, as shown in most of the examples for multiprocessing. In your case:

Process(target=standard_download, args=(link,)).start()

Or, you can always build a zero-argument function with lambda or partial, of course:

Process(target=functools.partial(standard_download, link)).start()

As a side note, just creating, starting, and leaking a bunch of child processes is a bad idea. You probably want to store them all in a list, and then join them all (whether immediately, or later). For example:

def download_from_file(filename):
    if os.path.exists(filename):
        links = [line.strip() for line in open(filename)]

        for link in links:
            p = Process(target=standard_download, args=(link,))
            p.start()
            yield p

processes = list(download_from_file(filename))
# later, when we need to wait until we're done with all the downloads
for process in processes:
    process.join()

Upvotes: 1

mdurant
mdurant

Reputation: 28683

You want the args keyword argument:

Process(target = standard_download, args=(link,)).start()

Upvotes: 1

Related Questions