CODEWITHSUNDEEP
pythonnumpyfor-loopoptimizationnested
tooty44
tooty44

Reputation: 7089

Optimizing nested Python for loops?

I am having some trouble with optimizing the following for loops. I have looked into the map function, comprehension expressions, and a bit of itertools and generators, but am unsure at how to approach optimization for nested loops. Any help or suggestions is greatly appreciated. Thanks in advance!

Note that object architecture is: 

self.variables.parameters.index1/index2/value
self.variables.rate
self.state.num

Loop 1:

mat1 = np.zeros(m, n)
for idx, variable in enumerate(self.variables):
    for parameter in variable.parameters:
        tracking_idx = parameter.index1 + parameter.index2
        mat1[tracking_idx, idx] = parameter.value

Loop 2:

mat2 = []
for variable in self.variables:
    rate = variable.rate
    for parameter in variable.parameters:
        if parameter.value < 0 and self.state.num[parameter.index1, parameter.index2] <= 0:
            rate = 0
    mat2.append(rate)

Upvotes: 0

Views: 1149

Answers (1)

hpaulj
hpaulj

Reputation: 231385

With a numpy tag I assume 'optimize' means cast these loops as compiled numpy array expressions. I don't think there's a way to do without first collecting all the data as numpy arrays, which will require the same loops.

You have a list of n variables. Each variable has a list of ? parameters. Each parameter has 3 or 4 attributes.

So

rates = np.array([v.rate for v in variables])
values = np.array([[p.value for p in v.parameters] for v in variables]
index1s = <dito>   (?,n) array
index2s = <dita>   (?,n) array

self.state.num is already a 2d array, size compatible with the range of values in index1s and index2s.

Given those 1 and 2d arrays we should be able to derive mat1 and mat2 with whole array operations. If ? is small relative to m and the range of values in index1 and index2 it might be worth do this. I don't have a realistic feel for your data.

===========

You mention the map function, comprehension expressions, and a bit of itertools and generators. These can make the code look a little cleaner but don't make much difference in speed.

I demonstrated the use of list comprehensions. Those expression can be more elaborate, but often at the cost of readability. I like writing helper functions to hide details. A generator comprehension can replace a list comprehension that feeds another. Maps cover the same functionality.

Since variables and parameters have attributes I assume you have defined them in classes. You could write methods that extract those attributes as simple lists or arrays.

class Variable(....):
   ....
   def get_values(self):
       return [p.value for p in self.parameters]
   def get_rate(self, state):
       rate = self.rate
       for parameter in self.parameters:
            if parameter.value < 0 and                
                state.num[parameter.index1, parameter.index2] <= 0:
            rate = 0
       return rate

values = [v.get_values() for v in variables]
rates = [v.get_rate(self.state) for v in variables]

You could even write these helper function without the class structure.

That doesn't speed up anything; it just hides some details in the object.

Upvotes: 1

Related Questions