CODEWITHSUNDEEP
pythonnltk
user2896468
user2896468

Reputation: 721

How to keep certain entities as one word using nltk tokenize in python?

How does one keep certain strings together in the following? For example, 

sentence = "?!a# .see"
tokens = nltk.word_tokenize(sentence)  
tokens

gives

['!', '?', 'a', '#', '.see'] rather than keeping '?!a#' as one entity.

Upvotes: 2

Views: 707

Answers (1)

Faruk Sahin
Faruk Sahin

Reputation: 8756

Seems like what you want to do is to split the string with whitespace. So just calling split would suffice:

>>> sentence.split()
['?!a#', '.see']

However if you really want to use a tokenizer, you can use a Regexp tokenizer:

>>> word_tokenizer = RegexpTokenizer('[\S]+') 
>>> word_tokenizer.tokenize(sentence)
['?!a#', '.see']

'\S' matches any non-whitespace character.

Upvotes: 2

Related Questions