CODEWITHSUNDEEP
pythonastropypyfits
vinu
vinu

Reputation: 453

How to merge two table with pyfits?

I am using Python 2.7.10 And pyfits 3.3. Earlier, I have used the following code to merge two tables. However, now I am getting some errors

t1 = pyfits.open(table1)[1].columns
t2 = pyfits.open(table2)[1].columns
new_columns = t1 + t2
hdu = pyfits.BinTableHDU.from_columns(new_columns)
hdu.writeto(outtable)

The error is:

Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
 File "/home/vvikraman/anaconda2/lib/python2.7/site-packages/pyfits/hdu/table.py", line 116, in from_columns
    data = FITS_rec.from_columns(coldefs, nrows=nrows, fill=fill)
 File "/home/vvikraman/anaconda2/lib/python2.7/site-packages/pyfits/fitsrec.py", line 315, in from_columns
    if arr.hdu.data is None:
ReferenceError: weakly-referenced object no longer exists

Upvotes: 3

Views: 1346

Answers (2)

Tom Aldcroft
Tom Aldcroft

Reputation: 2542

Is there a reason you cannot use astropy (i.e. astropy.io.fits)?

In that case the idiom would be:

from astropy.table import Table, hstack
t1 = Table.read(table1)
t2 = Table.read(table2)
new = hstack([t1, t2])
new.write(outtable)

In both the read and write calls, you need to provide format='fits' if the table name extension(s) do not imply that it is FITS.

Upvotes: 4

user707650
user707650

Reputation:

It's a bit confusing, and I'm not 100% sure about the cause, but here's my guess:

The result of opening the file itself, pyfits.open(...), isn't assigned to anything, because you immediately access the columns of the second HDU. That gives you a weak reference: essentially, the actual data isn't available anymore, because the file is, in a sense (and possibly real, I haven't checked) closed.
Pyfits does this to save memory.

The confusing part is that, after you've assigned t1 and t2, you can still print their definition, so it seems the actual columns are around. In fact, it's the column definitions that are still around: the actual data can't be retrieved. Hence, when you tried to create a new binary HDU from the combination t1 and t2, where you need the actual data, things fail.

The solution is to make sure you keep a proper reference to the data. There might be multiple ways to do this; this seems to work for me:

hdulist1 = pyfits.open(table1)
hdulist2 = pyfits.open(table2)
t1 = hdulist1[1].columns
t2 = hdulist2[1].columns
new_columns = t1 + t2
hdu = pyfits.BinTableHDU.from_columns(new_columns)
hdu.writeto(outtable)
# explicitly close the HDUs; might free up memory
hdulist1.close()
hdulist2.close()

Upvotes: 3

Related Questions