CODEWITHSUNDEEP
pythonarraysnumpymultidimensional-array
Paradox
Paradox

Reputation: 798

MemoryError trying to convert Numpy 2D arrays into a 3D array

I have some trouble converting some amount (in this case, 153) of Numpy 2D arrays into a 3D array (these 2D arrays represent gray images - i.e. 2048x2048x1 - in order to deal with an image sequence instead of a set of 2D images). I need this to obtain the signal formed by each pixel value over time (which should be convenient with Numpy, once this problem is solved).

My code is (pretty much) the following :

zdim = len(imglist) # 'imglist' a Python list of the path for each image I need to process
windowspan = 512
xmin = ymin = 2
xmax = ymax = xmin + windowspan

sequence = []
for i in range(zdim):
    hdulist = fits.open(imglist[i],'readonly') # allow to open FITS image files
    hdr = hdulist[0].header['DATE-OBS'] # fetch the image date/time
    img = fc.readfitsimg(imglist[i]) # return a np ndarray (2D)
    patch = img[ymin:ymax, xmin:xmax] # take a small of the original image
    print("patchSize : " + str(patch.size*4))
    sequence.append(patch) # adding to the list
    print("it : " + str(i))
sequence = np.array(sequence) # transform to numpy array

The interpreter returns a MemoryError after about 85 iterations...

Anyone would have any hints of what's happening ? (See some details below)

Some others details : - I'm using WinPython 32 bits (portable), because I cannot install a 'proper' Python distribution (I switched between Python 2.7.9.4 and 3.4.3.3 for testing purpose) - I'm forced to use a 32-bits Windows 7, on a PC which has 4GB, so 3.5GB usable / I've tried executing my script on another computer (Win7 64bits, 16GB of RAM)

Thanks for any help you could provide me with.

Upvotes: 0

Views: 594

Answers (1)

Jblasco
Jblasco

Reputation: 3967

The MemoryError happens when your computer runs out of RAM memory. In this case, you seem to run out while adding all the images into a cube, when hitting the limit 85x512x512. If this was the only problem of the code, I would advice to use memmap to save the results directly to the hard drive instead of in the RAM. The memmap option is also available when you open a fits file fits.open(..., memmap=True). In that case, you only open the image in the disc, and read the parts you need, instead of loading the whole image in RAM.

But the real problem here, I suspect, is that you have been opening fits files without any closing at the end of the loop (hdu.close() in your case).

Upvotes: 2

Related Questions