CODEWITHSUNDEEP
pythonfitsastropy
Arcturus B
Arcturus B

Reputation: 5621

OSError 24 (Too many open files) when reading bunch of FITS with astropy.io

I’m trying to load into memory a few 2 000 FITS using astropy.io.fits:

def readfits(filename):
    with fits.open(filename) as ft:
        # the fits contain a single HDU
        data = ft[0].data
    return data

data_sci = []
for i in range(2000):
    data_sci.append(readfits("filename_{}.fits".format(i)))

However, when reaching the 1015th file, OSError: [Errno 24] Too many open files is raised.

I have the same issue with:

def readfits(filename):
    ft = fits.open(filename) as ft:
    data = ft[0].data
    ft.close()
    return data

I suspect that astropy.io.fits does not properly close the file. Is there a way I can force the files to be closed?

Upvotes: 7

Views: 1851

Answers (3)

Miladiouss
Miladiouss

Reputation: 4710

You can call this function and store its output as long as you have memory. I thought it worths mentioning the answer explicitly but the credit goes to Iguananaut, bkaf, and this page.

def get_single_fits_data(fits_dir):

    hdul  = fits.open(fits_dir)
    for hdu in hdul:
        image_data = hdu.data.copy()
    hdul.close()
    gc.collect()

    return image_data

Upvotes: 0

Iguananaut
Iguananaut

Reputation: 23306

Your readfits function actually leaves the file handle open in order to keep access to the data, because by default it creates a mmap to the data and does not read it entirely into physical memory, as explained: http://astropy.readthedocs.org/en/latest/io/fits/appendix/faq.html#i-m-opening-many-fits-files-in-a-loop-and-getting-oserror-too-many-open-files

Incidentally, if you just want a function that reads the data out of the first HDU this is already built in: http://docs.astropy.org/en/v1.0.5/io/fits/api/files.html#astropy.io.fits.getdata

It's not necessary to reinvent the wheel.

Upvotes: 3

bkaf
bkaf

Reputation: 486

After taking a look at the astropy documentation i found this: http://astropy.readthedocs.org/en/latest/io/fits/appendix/faq.html#i-m-opening-many-fits-files-in-a-loop-and-getting-oserror-too-many-open-files

Upvotes: 4

Related Questions