CODEWITHSUNDEEP
pythonparallel-processing
PeCaDe
PeCaDe

Reputation: 374

Parallelize for loops in python

Is there any possibility to parallelize the following code in python? I was wondering how to convert this code with map and lambda functions..

values = (1,2,3,4,5 )

def op(x,y):
    return x+y

[(i, j, op(i, j))
        for i in values
        for j in values
        if i is not j]

Upvotes: 1

Views: 8119

Answers (3)

Hamza
Hamza

Reputation: 6025

You can use asyncio. (Documentation can be found [here][1]). It is used as a foundation for multiple Python asynchronous frameworks that provide high-performance network and web-servers, database connection libraries, distributed task queues, etc. Plus it has both high-level and low-level APIs to accomodate any kind of problem.

import asyncio

def background(f):
    def wrapped(*args, **kwargs):
        return asyncio.get_event_loop().run_in_executor(None, f, *args, **kwargs)

    return wrapped

@background
def your_function(argument):
    #code

Now this function will be run in parallel whenever called without putting main program into wait state. You can use it to parallelize for loop as well. When called for a for loop, though loop is sequential but every iteration runs in parallel to the main program as soon as interpreter gets there.

For example for your specific case, you can do:

@background
def op(x,y):
    time.sleep(1) # Added sleep to demonstrate parallelization
    print(f"function called for {x=}, {y=}\n", end='')
    return x, y, x+y

values = [1,2,3,4,5]                                                     

loop = asyncio.get_event_loop()           # Have a new event loop

looper = asyncio.gather(*[op(i, j)
        for i in values
        for j in values
        if i is not j])                   # Run the loop
                             
results = loop.run_until_complete(looper) # Wait until finish

print('Loop has finished and results are gathered!')      
results

This produces following output:

function called for x=1, y=5
function called for x=1, y=2
function called for x=2, y=3
function called for x=2, y=1
function called for x=1, y=3
function called for x=1, y=4
function called for x=2, y=4
function called for x=2, y=5
function called for x=3, y=1
function called for x=3, y=4
function called for x=4, y=1
function called for x=3, y=5
function called for x=3, y=2
function called for x=4, y=5
function called for x=4, y=3
function called for x=4, y=2
function called for x=5, y=3
function called for x=5, y=4
function called for x=5, y=2
function called for x=5, y=1
Loop has finished and results are gathered!
[(1, 2, 3), (1, 3, 4), (1, 4, 5), (1, 5, 6), (2, 1, 3), (2, 3, 5), (2, 4, 6), (2, 5, 7), (3, 1, 4), (3, 2, 5), (3, 4, 7), (3, 5, 8), (4, 1, 5), (4, 2, 6), (4, 3, 7), (4, 5, 9), (5, 1, 6), (5, 2, 7), (5, 3, 8), (5, 4, 9)]

Upvotes: 1

gushitong
gushitong

Reputation: 2006

Check this out:

from itertools import permutations

values = (1,2,3,4,5 )
[(i, j, i+j) for i, j in permutations(values, 2)]

It's in python's stdlib.

If you want run in parallel, check out this using python3:

import multiprocessing
from itertools import permutations

values = [1, 2, 3, 4, 5]
l = permutations(values, 2)


def f(x):
    return x[0], x[1], x[0] + x[1]

with multiprocessing.Pool(5) as p:
    data = p.map(f, l)

Upvotes: 2

Didac Cortiada rovira
Didac Cortiada rovira

Reputation: 38

You can parallelize the function op with multiprocessing and map:

from multiprocessing.dummy import Pool as ThreadPool
from itertools import permutations

pool = ThreadPool(4)  # Number of threads

values = (1,2,3,4,5)
aux_val = [(i, j) for i,j in permutations(values,2)]

def op(tupx):
    result = (tupx[0], tupx[1], tupx[0] + tupx[1])
    return result

results = pool.map(op, aux_val)

Upvotes: 2

Related Questions