Error "buffer is too small for requested array" when modify a fits file with python

Question

I'm trying to create a new fits file from an initial template.fits

This template.fits has the table of the voice 1 with 3915 rows, instead, my new file, must have more then 50000 rows.

The part of the code is the following:

hdulist = fits.open('/Users/Martina/Desktop/Ubuntu_Condivisa/Post_Doc_IAPS/ASTRI/ASTRI_scienceTools/Astrisim_MC/template.fits') 

hdu0=hdulist[0]

hdu0.writeto(out_pile+'.fits', clobber=True)  

hdu1=hdulist[1]

hdu1.header['NAXIS2'] = na           

hdu1.header['ONTIME'] = tsec              

hdu1.header['LIVETIME'] = tsec       

hdu1.writeto(out_pile+'.fits', clobber=True) 

hdu1_data=hdu1.data              

for j in range(na-1):

  hdu1_data[j+1][1]=j+1

  hdu1_data[j+1][3]=t[j]+0.

  hdu1_data[j+1][7]=ra[j]

  hdu1_data[j+1][8]=dec[j]

  hdu1_data[j+1][21]=enetot[j]

hdu1.writeto(out_pile+'.fits', clobber=True)

When I try to fill the new table (the last part of the code), the error is the following:

Traceback (most recent call last):

  File "C:\Users\Martina\AppData\Local\Programs\Python\Python36\lib\site-packages\astropy\utils\decorators.py", line 734, in __get__
    return obj.__dict__[self._key]

KeyError: 'data'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "Astrisim_MC_4.py", line 340, in 

    hdu1_data=hdu1.data              

  File "C:\Users\Martina\AppData\Local\Programs\Python\Python36\lib\site-packages\astropy\utils\decorators.py", line 736, in __get__
    val = self.fget(obj)

  File "C:\Users\Martina\AppData\Local\Programs\Python\Python36\lib\site-packages\astropy\io\fits\hdu\table.py", line 404, in data
    data = self._get_tbdata()

  File "C:\Users\Martina\AppData\Local\Programs\Python\Python36\lib\site-packages\astropy\io\fits\hdu\table.py", line 171, in _get_tbdata
    self._data_offset)

  File "C:\Users\Martina\AppData\Local\Programs\Python\Python36\lib\site-packages\astropy\io\fits\hdu\base.py", line 478, in _get_raw_data
    return self._file.readarray(offset=offset, dtype=code, shape=shape)

  File "C:\Users\Martina\AppData\Local\Programs\Python\Python36\lib\site-packages\astropy\io\fits\file.py", line 279, in readarray
    buffer=self._mmap)

TypeError: buffer is too small for requested array

I tried to vary the number of rows and the code works correctly up to 3969 rows.

How can I solve the problem?

Thank you very much in advance,

cheers!

Martina

Answer 1

Your initial problem where where you did this:

hdu1.header['NAXIS2'] = na 

A natural thing to think you might be able to do, but you actually should not. In general when working with astropy.io.fits, one should almost never manually mess with keywords in the FITS header that describe the structure of the data itself. This stems in part from the design of FITS itself--that it mixes these structural keywords in with metadata keywords--and partly a design issue with astropy.io.fits that it lets you manipulate these keywords at all, or that it doesn't more tightly tie the data to them. I wrote about this issue at more length here: https://github.com/astropy/astropy/issues/3836 but never got around to adding more explanation of this to the documentation.

Basically the way you can think about it is that when a FITS file is opened, its header is first read and parsed into a Header object containing all the header keywords. Some book-keeping is also done to keep track of how much data is in the file after the header. Then when you access the data of the HDU the header keywords are used to determine what the type and shape of the data is. So by doing something like

hdu1.header['NAXIS2'] = na
hdu1_data = hdu1.data

this isn't somehow growing the data in the file. Instead it's just confusing it into thinking there are more rows of data in the file then there actually are, hence the error "buffer is too small for requested array". The "buffer" it's referring to in this case is the rest of the data in the file, and you're requesting that it read an array that's longer than there is data in the file.

The fact that it allows you do break this at all is a bug in Astropy IMO. When the file is first opened it should save away all the correct structural keywords in the background, so that the data can still be loaded properly even if the user accidentally modifies these keywords (or perhaps the user should be completely prevented from modifying these keywords directly.

That's a long way to explain where you went wrong, but maybe it will help better understand how the library works.

As to your actual question, I think @Evert's advice is good, to use the higher-evel and easier to work with astropy.table to create a new table that's the size you need, and then copy the existing table into the new one. You can open the FITS table directly as a Table object as well with Table.read. I think you can also copy the FITS metadata over but I'm not sure exactly the best way to do that.

One other minor comment unrelated to your main question--when working with arrays you don't have to (and in fact shouldn't) use for loops to perform vectorizable operations.

For example since this is just looping over array indices:

for j in range(na-1): hdu1_data[j+1][1]=j+1 hdu1_data[j+1][3]=t[j]+0. hdu1_data[j+1][7]=ra[j] hdu1_data[j+1][8]=dec[j] hdu1_data[j+1][21]=enetot[j]

you can write operations like this like:

hdu1_data[:][1] = np.arange(na)
hdu1_data[:][3] = t + 0.
hdu1_data[:][7] = ra

and so on (I'm not sure why you were doing j+1 because this is skipping the first row, but the point still stands). This assumes of course that the array being updated (hdu1_data, in this case) already has na rows. But that's why you need to grow or concatenate to your array first if it's not already that size.