FuzzyWuzzy for very similar records in Python

Question

I have a dataset with which I want to find the closest string match. For that purpose I'm using FuzzyWuzzy in this way

sol=process.extract(t,dev2,scorer=fuzz.token_sort_ratio)

Where t is the string and dev2 is the list to compare to. My problem is that sometimes it has very similar records and options provided by FuzzyWuzzy seems to be lacking. And I've tested with token_sort, token_set, partial_token sort and set, ratio, partial_ratio, and WRatio.

For example, the string Italy - Serie A gives me the following 2 closest matches.

Token_sort_ratio: (92, 'Italy - Serie D');(86, 'Italian - Serie A')

The one wanted is obviously the second one, but character by character is closer the first one, which is a different league.

This happens as well with teams. If, let's say I have a string Buchtholz I would obtains Buchtholz II before I get TSV Buchtholz.

My main guess now would be to try and weight the presence and absence of several characters more heavily, like single capital letters at the end of the string, so if there is a difference in the letter or an absence it is weighted as less close. Or for () and special characters.

I don't know if there is a way to take this into account or you guys have a better approach to get the string that really matches.

Answer 1

Similarity matches often require knowledge of the data being analysed. i.e. it is not just a blind single round of matching. I recommend that you pass your results through more steps of matching, starting with inclusive/optimistic approaches (like token_set_ratio) with low cut off scores and working toward more exclusive/pessimistic approaches with higher cut off scores until you have a clear winner. If you know more about the text you're analyzing, you can even modify the strings as you progress.

In a case I worked on, I did similarity matches of goods movement descriptions. In the descriptions the numbers sequences were more important than the text. e.g. when looking for a match for "SLURRY VALVE 250MM RAGMAX 2000" the 250 and 2000 part of the string are important, otherwise I get a "SLURRY VALVE 50MM RAGMAX 2000" as the best match instead of "VALVE B/F 250MM,RAGMAX 250RAG2000 RAGON" which is a better result.

I put the similarity match process through two steps: 1. Get a bunch of similar matches using an optimistic matching scorer (token_set_ratio) 2. get the number sequences of these results and pass them through another round of matching with a more strict scorer (token_sort_ratio). Doing this gave me the better result in the example I showed above.

Below is some blocks of code that could be of assistance:

here's a function to get numbers from the sequence. (In your case you might use this to exclude numbers from your string instead?)

def get_numbers_from_string(description):   
    numbers = ''.join((ch if ch in '0123456789.-' else ' ') for ch in description)
    numbers = ' '.join([nr for nr in numbers.split()])
    return numbers

and here is a portion of the code I used to put the description match through two rounds:

try:    
    # get close match from goods move that has material numbers
    df_material = pd.DataFrame(process.extract(description, 
                                               corpus_material,
                                               scorer=fuzz.token_set_ratio), 
                               columns=['Similar Text','Score']
                              )

    if df_material['Score'][df_material['Score']>=cut_off_accuracy_materials].count()>=1:
        similar_text = df_material['Similar Text'].iloc[0]
        score = df_material['Score'].iloc[0]

        if nr_description_numbers>4:
            # if there are multiple matches found, then get best number combination match  
            df_material = df_material[df_material['Score']>=cut_off_accuracy_materials]
            new_corpus = list(df_material['Similar Text'])
            new_corpus = np.vectorize(get_numbers_from_string)(new_corpus)
            df_material['numbers'] = new_corpus
            df_numbers = pd.DataFrame(process.extract(description_numbers, 
                                                       new_corpus,
                                                       scorer=fuzz.token_sort_ratio), 
                                      columns=['numbers','Score']
                                     )

            similar_text = df_material['Similar Text'][df_material['numbers']==df_numbers['numbers'].iloc[0]].iloc[0]
            nr_score = df_numbers['Score'].iloc[0]

hope it helps, and good luck