CODEWITHSUNDEEP
pythonpandasdataframe
Deluxe88888
Deluxe88888

Reputation: 113

check if one dataframe exists in another

I have 2 dataframes Overall and df2. Overall

Time                ID_1    ID_2               
2020-02-25 09:24:14 140209  81625000
2020-02-25 09:24:14 140216  91625000
2020-02-25 09:24:18 140219  80250000
2020-02-25 09:24:18 140221  90250000
25/02/2020 09:42:02     143982  39075000

df2

ID_1    ID_2            Time                  Match?
140209  81625000    25/02/2020 09:24:14    no_match
143983  44075000    25/02/2020 09:42:02    no_match
143982  39075000    25/02/2020 09:42:02    match
143984  39075000    25/02/2020 09:42:02    no_match

I want to check if df2 exists in Overall and if so does df2.Match?of that same row say match. If so return a new column saying yes, if it doesn't say match return no.

I have tried

Overall_1 = pds.merge(Overall, df2, on=….., how='left', indicator= 'Exist')
Overall_1.drop([...], inplace = True, axis =1 )
Overall_1['Exist']= np.where((Overall_1.Exist =='both') & (Overall_1.Match? == match), 'yes', 'no')

But an error prevails

TypeError: Cannot perform 'rand_' with a dtyped [bool] array and scalar of type [float]

So resulting Overall_1 dataframe should look like:

Time                ID_1    ID_2             Exist   
2020-02-25 09:24:14 140209  81625000     No
2020-02-25 09:24:14 140216  91625000     NaN
2020-02-25 09:24:18 140219  80250000     NaN
2020-02-25 09:24:18 140221  90250000     Nan
25/02/2020 09:42:02     143982  39075000     Yes

Upvotes: 4

Views: 145

Answers (2)

Umar.H
Umar.H

Reputation: 23099

Using merge and np.select.

import numpy as np
#df1 = Overall
df3 = pd.merge(df1,df2,on=['ID_1','ID_2','Time'],how='left',indicator='Exists')


col1 = df3['Match?']
col2 = df3['Exists']

conditions = [(col1.eq('match') & (col2.eq('both'))),
              (col1.eq('no_match') & (col2.eq('both')))
             ]

choices = ['yes','no']

df3['Exists'] = np.select(conditions,choices,default=np.nan)

print(df3.drop('Match?',axis=1))


                 Time    ID_1      ID_2 Exists
0 2020-02-25 09:24:14  140209  81625000     no
1 2020-02-25 09:24:14  140216  91625000    nan
2 2020-02-25 09:24:18  140219  80250000    nan
3 2020-02-25 09:24:18  140221  90250000    nan
4 2020-02-25 09:42:02  143982  39075000    yes

or more simply just using replace dict and .merge

df3 = pd.merge(df1,df2,on=['ID_1','ID_2','Time'],how='left')\
                                      .replace({'no_match' : 'no', 
                                                'match' : 'yes'})\
                                      .rename(columns={'Match?' : 'Exists'})

print(df3)

                 Time    ID_1      ID_2 Exists
0 2020-02-25 09:24:14  140209  81625000     no
1 2020-02-25 09:24:14  140216  91625000    NaN
2 2020-02-25 09:24:18  140219  80250000    NaN
3 2020-02-25 09:24:18  140221  90250000    NaN
4 2020-02-25 09:42:02  143982  39075000    yes

Upvotes: 1

user13531614
user13531614

Reputation:

you can try : df_diff = pd.concat([Overall,df2]).drop_duplicates(keep=False)

Upvotes: 0

Related Questions