CODEWITHSUNDEEP
pythonpandasfilter
Vince
Vince

Reputation: 557

Dynamic pandas dataframe filter not working

I can't get this dynamic filter work

df_dates

print(df_dates)

    Type  Entry      Exit 
    0     2008-03-03 2008-03-17  
    1     2010-05-19 2010-06-10

This hardcoded filter is working

df_to_filter = df_to_filter[
    (df_to_filter['date']>='2008-03-03 00:00:00') & (df_to_filter['date']<='2008-03-17 00:00:00') | 
    (df_to_filter['date']>='2010-05-19 00:00:00') & (df_to_filter['date']<='2010-06-10 00:00:00')
]

Dynamic filter not working, the string seems to be exactly the same

df_str = "df_to_filter['date']"
    
filter_mask = ' | '.join(f'({df_str}>=\'{start}\') & ({df_str}<=\'{stop}\')' for start,stop in zip(df_dates['Entry'],df_dates['Exit']))
filter_mask = filter_mask + ']'

print(filter_mask)

(df_to_filter['date']>='2008-03-03 00:00:00') & (df_to_filter['date']<='2008-03-17 00:00:00') | (df_to_filter['date']>='2010-05-19 00:00:00') & (df_to_filter['date']<='2010-06-10 00:00:00')]
    
df_to_filter = df_to_filter[filter_mask]

Error

KeyError: "(df_to_filter['date']>='2008-03-03 00:00:00') & (df_to_filter['date']<='2008-03-17 00:00:00') | (df_to_filter['date']>='2010-05-19 00:00:00') & (df_to_filter['date']<='2010-06-10 00:00:00')]"

Upvotes: 0

Views: 62

Answers (1)

ipj
ipj

Reputation: 3598

If You have for example dataframes:

df_dates:

Type  Entry      Exit 
0     2008-03-03 2008-03-17  
1     2010-05-19 2010-06-10

df_to_filter:

date
2008-03-03 
2010-06-11

Then You can filter it using filter_mask expression:

filter_mask = ' | '.join(f'({df_str}>=\'{start}\') & ({df_str}<=\'{stop}\')' for start,stop in zip(df_dates['Entry'],df_dates['Exit']))
"(df_to_filter['date']>='2008-03-03') & (df_to_filter['date']<='2008-03-17') | (df_to_filter['date']>='2010-05-19') & (df_to_filter['date']<='2010-06-10')"

print(df_to_filter[eval(filter_mask)])

result:

         date
0  2008-03-03

To call literal expression eval() function is needed.

Upvotes: 1

Related Questions