CODEWITHSUNDEEP
pythonarraysnumpymemory
GeoMonkey
GeoMonkey

Reputation: 1665

Memory issue when iterating over a list or arrays

I have a list of arrays such that:

arr = [array([1,2,3,4,5]), array([1,4,6,7]) .......]

which contains 40000 arrays. I would ideally like to have it as a 2d numpy array but I can't guarantee that all the arrays will be the same length.

I want to do something basic to all of the values in the list, such as:

out = (3*arr)+2

but I quickly run out of memory (on a 32GB machine) so its obviously very inefficient. I have tried iterating over the list and appending the results to a new list but this is equally inefficient.

Is there an efficient way to achieve this?

--------------- EDIT --------------

arr looks like:

[array([ 451.481649,  456.490319,  461.498989,  466.507659,  471.516329,
    476.524999,  481.533669,  486.542339,  491.551009,  496.559679,
    501.568349,  506.577019,  511.585689,  516.594359,  521.603029,
    526.611699,  531.62037 ,  536.62904 ,  541.63771 ,  546.64638 ,
    551.65505 ,  556.66372 ,  561.67239 ,  566.68106 ,  571.68973 ,
    576.6984  ,  581.70707 ,  586.71574 ,  591.72441 ,  596.73308 ,
    601.74175 ,  606.75042 ,  611.75909 ,  616.76776 ,  621.77643 ,
    626.7851  ,  631.79377 ,  636.80244 ,  641.811111,  646.819781,
    651.828451,  656.837121,  661.845791,  666.854461,  671.863131]),
array([ 451.481649,  456.490319,  461.498989,  466.507659,  471.516329,
    476.524999,  481.533669,  486.542339,  491.551009,  496.559679,
    501.568349,  506.577019,  511.585689,  516.594359,  521.603029,
    526.611699,  531.62037 ,  536.62904 ,  541.63771 ,  546.64638 ,
    551.65505 ,  556.66372 ,  561.67239 ,  566.68106 ,  571.68973 ,
    576.6984  ,  581.70707 ,  586.71574 ,  591.72441 ,  596.73308 ,
    601.74175 ,  606.75042 ,  611.75909 ,  616.76776 ,  621.77643 ,
    626.7851  ,  631.79377 ,  636.80244 ,  641.811111,  646.819781,
    651.828451,  656.837121,  661.845791,  666.854461,  671.863131]),
array([ 451.481649,  456.490319,  461.498989,  466.507659,  471.516329,
    476.524999,  481.533669,  486.542339,  491.551009,  496.559679,
    501.568349,  506.577019,  511.585689,  516.594359,  521.603029,
    526.611699,  531.62037 ,  536.62904 ,  541.63771 ,  546.64638 ,
    551.65505 ,  556.66372 ,  561.67239 ,  566.68106 ,  571.68973 ,
    576.6984  ,  581.70707 ,  586.71574 ,  591.72441 ,  596.73308 ,
    601.74175 ,  606.75042 ,  611.75909 ,  616.76776 ,  621.77643 ,
    626.7851  ,  631.79377 ,  636.80244 ,  641.811111,  646.819781,
    651.828451,  656.837121,  661.845791,  666.854461,  671.863131])]

Upvotes: 0

Views: 87

Answers (1)

Erol
Erol

Reputation: 6526

If you are really doing it like you wrote, it is likely that you will run into memory issues, since

out = (3*arr)+2

replicates your array 3 times and then tries appending to it. Larger that constant, larger your array gets and hence the memory explosion.

To achieve what you want without memory issues, use

out = [[3*x + 2 for x in arr_list] for arr_list in arr]

Upvotes: 2

Related Questions