CODEWITHSUNDEEP
pythonmultiprocessingjoblib
Rohit Kewalramani
Rohit Kewalramani

Reputation: 448

How to use nested loops in joblib library in python

Actual code looks like:

def compute_score(row_list,column_list): 

    for i in range(len(row_list)):
            for j in range(len(column_list)):
                tf_score = self.compute_tf(column_list[j],row_list[i])

I am tying to achieve multi-processing i.e. at every iteration of j I want to pool column_list. Since compute_tf function is slow I want to multi-process it.

I've found have to do it using joblib in Python, But I am unable to workaround with nested loops.

Parallel(n_jobs=2)(delayed(self.compute_tf)<some_way_to_use_nested_loops>)

This is what is to be achieved. It would be a great help if any solution on this is provided or any-other solution.

Upvotes: 7

Views: 10586

Answers (2)

Thomas Moreau
Thomas Moreau

Reputation: 4467

Another solution without having to implement a generator function, is to use the nested list comprehension for the generator:

Parallel(n_jobs=2)(delayed(self.compute_tf)(i, j) for j in column_list for i in row_list)

The order will be given as:

[(i, j) for j in range(10) for i in range(10)]

Upvotes: 15

joel goldstick
joel goldstick

Reputation: 4493

I'm not understanding your part about Parallel. However, your code can be simplified to this:

def compute_score(row_list,column_list): 

    for i in row_list:
            for j in column_list:
                tf_score = self.compute_tf(j,i)

It may be useful to create a generator that yields each j,i so that you can process them outside of the generator

def compute_pairs(row_list,column_list): 

    for i in row_list:
            for j in column_list:
                yield j, i

then use:

 scored = [compute_tf(j,i) for j, i in compute_pairs(row_list, column_list)]

Upvotes: 3

Related Questions