CODEWITHSUNDEEP
pythonpandasindexingfuzzy-comparisonfuzzywuzzy
DavNej
DavNej

Reputation: 186

Python Pandas - Fuzzy duplicates matching

I have a Dataframe like this:

    make                model
0   allard              K1
1   alllard             J2
2   alpine renault      A110
3   alpine renualt      A310
4   amc (rambler        American
5   amc (rambler)       Marlin
6   aries               1907
7   ariès               1932
8   austin healey       3000
9   austin-healey       Sprite
10  benjamin et benova  Type B3
11  benjamin/benova     Type P2
12  benjmin/benova      Type P3

The goal would be to have a third column with the index of the row which has the highest fuzzy ratio (closest fuzzy match).

How could I compare rows together in an efficient way?

Upvotes: 1

Views: 2547

Answers (1)

andersource
andersource

Reputation: 829

Using fuzzywuzzy, and assuming the fuzziness should be matched for the make column, you can try:

import pandas as pd
from itertools import product
from fuzzywuzzy.fuzz import ratio

df = pd.read_csv('data.csv')
keys = list(set(df['make']))
ratios = pd.DataFrame([{'k1': k1, 'k2': k2, 'ratio': ratio(k1, k2)} for k1, k2 in product(keys, keys) if k1 != k2])

def find_closest(make):
    return df[df['make'] == ratios.loc[ratios[ratios['k1'] == make]['ratio'].argmax(), 'k2']].index.values[0]

df['closest_index'] = df['make'].apply(find_closest)

print(df)

Output for your data:

                  make     model  closest_index
0               allard        K1              1
1              alllard        J2              0
2       alpine renault      A110              3
3       alpine renualt      A310              2
4         amc (rambler  American              5
5        amc (rambler)    Marlin              4
6                aries      1907              7
7                ariès      1932              6
8        austin healey      3000              9
9        austin-healey    Sprite              8
10  benjamin et benova   Type B3             11
11     benjamin/benova   Type P2             12
12      benjmin/benova   Type P3             11

Upvotes: 2

Related Questions