CODEWITHSUNDEEP
pythonlistnumpyperformancematlab
Archimedes_91
Archimedes_91

Reputation: 61

Fastest way to store 3D numpy array in a loop

I need to store a numpy array of shape (2000,720,1280) which is created in every loop. My code looks like:

    U_list = []   
      
    for N_f in range(N): 
             U = somefunction(N_f)
             U_list.append(U)
             del U

So I delete the matrix U in every loop because my RAM get full.

Is this a good method to store the matrix U or would you recommend another solution? I compare the code to matlab and matlab need the half time to compute. I think the storage of U in a list could be the reason.

Upvotes: 0

Views: 652

Answers (1)

Will.Evo
Will.Evo

Reputation: 1177

Using this method will tell you if you are able to store the total U arrays right out the gate. If N is so large that you can't make the results numpy array, you'll have to get creative. Maybe save every 20 into a pickle file or something.

import numpy as np

N = 20
shape = (2000, 720, 1280)
#Make sure to match the dtype returned by somefunction
results = np.zeros((N, *shape)) 
for N_f in range(N):
    results[N_f] = somefunction(N_f)

Upvotes: 1

Related Questions