CODEWITHSUNDEEP
pythonpython-3.xpandasunordered
Piyush S. Wanare
Piyush S. Wanare

Reputation: 4933

Compare values from two pandas data frames, order-independent

I am new to data science. I want to check which elements from one data frame exist in another data frame, e.g.

df1 = [1,2,8,6]
df2 = [5,2,6,9]

# for 1 output should be False

# for 2 output should be True

# for 6 output should be True

etc.

Note: I have matrix not vector.

I have tried using the following code:

import pandas as pd
import numpy as np

    priority_dataframe = pd.read_excel(prioritylist_file_path, sheet_name='Sheet1', index=None)

    priority_dict = {column: np.array(priority_dataframe[column].dropna(axis=0, how='all').str.lower()) for column in
                         priority_dataframe.columns}
    keys_found_per_sheet = []
    if file_path.lower().endswith(('.csv')):
        file_dataframe = pd.read_csv(file_path)
    else:
        file_dataframe = pd.read_excel(file_path, sheet_name=sheet, index=None)

    file_cell_array = list()
    for column in file_dataframe.columns:
        for file_cell in np.array(file_dataframe[column].dropna(axis=0, how='all')):
            if isinstance(file_cell, str) == 'str':
                file_cell_array.append(file_cell)
            else:
                file_cell_array.append(str(file_cell))

    converted_file_cell_array = np.array(file_cell_array)

    for key, values in priority_dict.items():
        for priority_cell in values:
            if priority_cell in converted_file_cell_array[:]:
                keys_found_per_sheet.append(key)
                break

I am doing something wrong in if priority_cell in converted_file_cell_array[:] ?

Is there any other efficient way to do that?

Upvotes: 1

Views: 706

Answers (2)

DYZ
DYZ

Reputation: 57033

You can take the .values from each dataframe, convert them to a set(), and take the set intersection.

set1 = set(df1.values.reshape(-1).tolist())
set2 = set(dr2.values.reshape(-1).tolist())
different = set1 & set2

Upvotes: 2

jezrael
jezrael

Reputation: 862771

You can flatten all values of DataFrames by numpy.ravel and then use set.intersection():

df1 = pd.DataFrame({'A':list('abcdef'),
                   'B':[4,5,4,5,5,4],
                   'C':[7,8,9,4,2,3],
                   'D':[1,3,5,7,1,0],
                   'E':[5,3,6,9,2,4],
                   'F':list('aaabbb')})

print (df1)
   A  B  C  D  E  F
0  a  4  7  1  5  a
1  b  5  8  3  3  a
2  c  4  9  5  6  a
3  d  5  4  7  9  b
4  e  5  2  1  2  b
5  f  4  3  0  4  b

df2 = pd.DataFrame({'A':[2,3,13,4], 'Z':list('abfr')})
print (df2)
    A  Z
0   2  a
1   3  b
2  13  f
3   4  r

L = list(set(df1.values.ravel()).intersection(df2.values.ravel()))
print (L)
['f', 2, 3, 4, 'a', 'b']

Upvotes: 2

Related Questions