CODEWITHSUNDEEP
pythonregextext-processingemoticons
David Moreno García
David Moreno García

Reputation: 4523

Extract emoticons from a text

I need to extract text emoticons from a text using Python and I've been looking for some solutions to do this but most of them like this or this only cover simple emoticons. I need to parse all of them.

Currently I'm using a list of emoticons that I iterate for every text that I have process but this is so inefficient. Do you know a better solution? Maybe a Python library that can handle this problem?

Upvotes: 10

Views: 2645

Answers (1)

Luka Rahne
Luka Rahne

Reputation: 10447

One of most efficient solution is to use Aho–Corasick string matching algorithm and is nontrivial algorithm designed for this kind of problem. (search of multiple predefined strings in unknown text)

There is package available for this.
https://pypi.python.org/pypi/ahocorasick/0.9
https://hkn.eecs.berkeley.edu/~dyoo/python/ahocorasick/

Edit: There are also more recent packages available (haven tried any of them) https://pypi.python.org/pypi/pyahocorasick/1.0.0

Extra:
I have made some performance test with pyahocorasick and it is faster than python re when searching for more than 1 word in dict (2 or more).

Here it is code:

import re, ahocorasick,random,time

# search N words from dict
N=3

#file from http://norvig.com/big.txt
with open("big.txt","r") as f:
    text = f.read()

words = set(re.findall('[a-z]+', text.lower())) 
search_words = random.sample([w for w in words],N)

A = ahocorasick.Automaton()
for i,w in enumerate(search_words):
    A.add_word(w, (i, w))

A.make_automaton()
#test time for ahocorasic
start = time.time()
print("ah matches",sum(1 for i in A.iter(text))) 
print("aho done in ", time.time() - start)


exp = re.compile('|'.join(search_words))
#test time for re
start = time.time()
m = exp.findall(text)
print("re matches",sum(1 for _ in m))
print("re done in ",time.time()-start)

Upvotes: 4

Related Questions