CODEWITHSUNDEEP
algorithmdata-structures
Hi There
Hi There

Reputation: 1966

Best Data Structure For Text Processing

Given a sentence like this, and i have the data structure (dictionary of lists, UNIQUE keys in dictionary):

{'cat': ['feline', 'kitten'], 'brave': ['courageous', 'fearless'], 'little': ['mini']}

A courageous feline was recently spotted in the neighborhood protecting her mini kitten

How would I efficiently process these set of text to convert the word synonyms of the word cat to the word CAT such that the output is like this:

A fearless cat was recently spotted in the neighborhood protecting her little cat

The algorithm I want is something that can process the initial text to convert the synonyms into its ROOT word (key inside dictionary), the keywords and synonyms would get longer as well. Hence, first, I want to inquire if the data structure I am using is able to perform efficiently and whether there are more efficient structure. For now, I am only able to think of looping through each list inside the dictionary, searching for the synonym's then mapping it back to its keyword

edit: Refined the question

Upvotes: 0

Views: 121

Answers (1)

trincot
trincot

Reputation: 350310

Your dictionary is organised in the wrong way. It will allow you to quickly find a target word, but that is not helpful when you have an input that does not have the target word, but some synonyms of it.

So organise your dictionary in the opposite sense:

d = {
    'feline': 'cat',
    'kitten': 'cat'
}

To make the replacements, you could create a regular expression and call re.sub with a callback function that will look up the translation of the found word:

import re

regex = re.compile(rf"\b(?:{ '|'.join(map(re.escape, d)) })\b")

s = "A feline was recently spotted in the neighborhood protecting her little kitten"
print(regex.sub(lambda match: d[match[0]], s))

The regular expression makes sure that the match is with a complete word, and not with a substring -- "cafeline" as input will not give a match for "feline".

Upvotes: 1

Related Questions