CODEWITHSUNDEEP
pythonpython-3.xmultithreadingparallel-processing
randomUser
randomUser

Reputation: 11

How can I run a function on two lists at the same time?

I've got a simple question. Essentially I want to run a function on a list of lists at the same time or in parallel.

Here is essentially what I'm working with

def func(x):
    print(x)
for objectList in space:
    for subObject in objectList:
        func(subObject)

I'd ideally like to be able to have func() run on every objectList at once. I know (actually I don't, which is why I'm asking how to do this) this is threading or parallelism but this is my first attempt at such.

My question is, is there a module or built in function in python or standard way to accomplish this task?

And sorry if I somehow broke a rule on this site or something or if you think this question is stupid or a duplicate or whatever.

Upvotes: 0

Views: 318

Answers (2)

mlotz
mlotz

Reputation: 140

As pointed out by others you should have a valid point to use threading and parellel computting as it also requiers other actions from you like input/load distribution and managing workers (working threads). There are certainly libraries that have built in those mechanics. For simple applications you can just use (threading.Thread) class. Here's a simple example of using threads. This code will create worker for every element of space.

import threading


def func(x):
    print(x)


class FuncThread(threading.Thread):
    def __init__(self, x):
        threading.Thread.__init__(self)
        self.x = x

    def run(self):
        #worker code = func(x)
        print('doing stuff with:', self.x)


space = [[1, 1, 1], [2, 2, 2], [3, 3, 3]]
worker_handlers = list()

for objectList in space:
    for subObject in objectList:
        worker_handlers.append(FuncThread(subObject))

for worker in worker_handlers:
    worker.start()

print('The main program continues to run in foreground.')

for worker in worker_handlers:
    worker.join()  # Wait for the background tasks to finish
print('Main program waited until background was done.')

Upvotes: 0

Kapocsi
Kapocsi

Reputation: 1032

From the python docs:

the Pool object [...] offers a convenient means of parallelizing the execution of a function across multiple input values, distributing the input data across processes (data parallelism)

You could use this pattern, which is also from the docs:

from multiprocessing import Pool

def f(x):
    return x*x

if __name__ == '__main__':
    with Pool(5) as p:
        print(p.map(f, [1, 2, 3]))

Upvotes: 1

Related Questions