CODEWITHSUNDEEP
pythonlistinverted-index
Wolf
Wolf

Reputation: 399

Inverted index given a list of document tokens using python?

I'm a newbie to python. I need to create an inverted index function given a list of document tokens. The index maps each unique word to a list of document ids, sorted in increasing order.

My code:

def create_index(tokens):
    inverted_index = {}
    wordCount = {}
    for k, v in tokens.items():
        for word in v.lower().split():
            wordCount[word] = wordCount.get(word,0)+1
            if inverted_index.get(word,False):
                if k not in inverted_index[word]:
                    inverted_index[word].append(k)
            else:
                inverted_index[word] = [k]
    return inverted_index, wordCount

Note: This works fine when the input argument is of the form {1:"Madam I am Adam",2: "I have never been afraid of him"}

output that i get for the above example:

{'madam': [1], 'afraid': [2], 'i': [1, 2], 'of': [2], 'never': [2], 'am': [1], 'been': [2], 'adam': [1], 'have': [2], 'him': [2]}

As per my code K,v correspond to Key and value of the list

Desired output when we call the create_index function with an argument:

index = create_index([['a', 'b'], ['a', 'c']])
>>> sorted(index.keys())
['a', 'b', 'c']
>>> index['a']
[0, 1]
index['b']
[0]
index['c']
[1]

Upvotes: 2

Views: 18195

Answers (1)

poke
poke

Reputation: 388273

Something like this?

>>> from collections import defaultdict
>>> def create_index (data):
        index = defaultdict(list)
        for i, tokens in enumerate(data):
            for token in tokens:
                index[token].append(i)
        return index

>>> create_index([['a', 'b'], ['a', 'c']])
defaultdict(<class 'list'>, {'b': [0], 'a': [0, 1], 'c': [1]})
>>> index = create_index([['a', 'b'], ['a', 'c']])
>>> index.keys()
dict_keys(['b', 'a', 'c'])
>>> index['a']
[0, 1]
>>> index['b']
[0]

Upvotes: 2

Related Questions