CODEWITHSUNDEEP
pythonpandaspandas-groupby
ihightower
ihightower

Reputation: 3253

pandas dataframe filter to return True for ALL rows. how?

Hi I have a filter 'm' set that is flexible enough to change by me. Sometimes, I want to filter by Car or x_acft_body , or any of the various other fields, etc. Sometime I want to have all of the rows returned by commenting and uncommenting the required lines. But without changing the subsequent code, after the filter 'm' line.

How can I have a filter that will return true for ALL rows, when I don't want the filters applied? For e.g. something like 1==1 but i know this doesn't work.

I don't want to set dfdata.somefield.notnull() etc. as I will not be too sure if this field will be always not null or not. also I DO NOT want to change subsequent code to be like dfdata.groupby. i.e. without the [m]

# set filter if needed
m = (   1==1 #& return true at all times
#         (dfdata.Car == 'PG') #&
#         (dfdata.x_acft_body == 'N')# &
#         (dfdata.Car.isin(['PG', 'VJ', 'VZ']))
)


dft1 = dfdata[m].groupby(['FLD1']).agg({'FLD2': 'count'})

Upvotes: 14

Views: 4811

Answers (2)

SultanOrazbayev
SultanOrazbayev

Reputation: 16561

This is not directly relevant to the original question, but as of pandas==2.2.3, attempting to create a mask from a combination of list of bool and a pandas.Series will raise this warning:

FutureWarning: Logical ops (and, or, xor) between Pandas objects and dtype-less sequences (e.g. list, tuple) are deprecated and will raise in a future version. Wrap the object in a Series, Index, or np.array before operating instead.

One simple way to fix it, and to solve the original question in this thread, is to create a mask of all True values by comparing the dataframe index to itself:

m = dfdata.index == dfdata.index

Note that if your dataframe index can contain None values, the above will fail, since None == None is False. To fix that for any general case, fill-in the nan values:

m = df.index.fillna(-1) == df.index.fillna(-1)

Upvotes: 0

jezrael
jezrael

Reputation: 862771

You can create bool constant and change final mask by it:

#True for return all rows
m = (dfdata.Car == 'PG') | True

And:

#False for apply filter
m = (dfdata.Car == 'PG') | False

First solutions:

m = [True] * len(df.index)

m = np.repeat(True, len(df.index))

Upvotes: 14

Related Questions