CODEWITHSUNDEEP
pythonastropy
Gabriel
Gabriel

Reputation: 42349

Drop rows with masked elements in astropy Table

Consider an astropy Table() object.

from astropy.io import ascii

weather_data = """
  day,precip,type
  ,1.5,rain
  Tues, 9.,
  Wed,1.1,snow
  ed,,aaa
  Wd,1.1,snow
  """

dat = ascii.read(weather_data)
print(dat)

day  precip type
---- ------ ----
  --    1.5 rain
Tues    9.0   --
 Wed    1.1 snow
  ed     --  aaa
  Wd    1.1 snow

I need to drop all rows that contain at least one masked element. The final table should then look like:

day  precip type
---- ------ ----
 Wed    1.1 snow
  Wd    1.1 snow

This seems like a simple task, but I could not find a way t do it in the docs.

Add

I know that I could use .to_pandas() to convert the table to a pandas object and then use something like .dropna(), but this forces me to have pandas installed which I don't want.

Upvotes: 3

Views: 1136

Answers (2)

rafaelc
rafaelc

Reputation: 59274

For a generalized way 

import operator
# if python3: from functools import reduce
dat[reduce(operator.and_, [~dat[col].mask for col in dat.columns])]

day   precip type
str4 float64 str4
---- ------- ----
 Wed     1.1 snow
  Wd     1.1 snow

What this process does is simply a generalized way of doing 

dat[(~dat['day'].mask) & (~dat['precip'].mask) & (~dat['type'].mask)]

Upvotes: 3

AGN Gazer
AGN Gazer

Reputation: 8378

Here is another way:

dat.remove_rows(np.where([c.data for c in dat.mask.itercols()])[-1])

But I agree with you that this filtering of invalid values should be supported by a masked table.

Upvotes: 1

Related Questions