CODEWITHSUNDEEP
javanlptokentokenizeinformation-retrieval
Adam_G
Adam_G

Reputation: 7879

Java NLP: Extracting Indicies When Tokenizing Text

When tokenizing a string of text, I need to extract the indexes of the tokenized words. For example, given:

"Mary didn't kiss John"

I would need something like:

[(Mary, 0), (did, 5), (n't, 8), (kiss, 12), (John, 17)]

Where 0, 5, 8, 12 and 17 correspond to the index (in the original string) where the token began. I cannot rely on just whitespace, since some words become 2 tokens. Further, I cannot just search for the token in the string, since the word likely will appear multiple times.

One giant obstacle is that I'm working with "dirty" text. Here is a real example from the corpus, and its tokenization:

String:

The child some how builds a boaty  c capable of getting scrtoacross the sea, even after findingovercoming many treachrous rous obsittalcles.

Tokens:

The, child, some, how, builds, a, boaty, , , c, , capable, of, getting, scrto, , across, the, sea, ,, even, after, finding, , , , , overcoming, many, treachrous, rous, obsittalcles, .

I'm currently using OpenNLP to tokenize the text, but am ambivalent about which API to utilize for tokenization. It does need to be Java, though, so (unfortunately) Python's NLTK is out of the picture.

Any ideas would be greatly appreciated! Thanks!

Upvotes: 0

Views: 1196

Answers (3)

Amit Kumar Gupta
Amit Kumar Gupta

Reputation: 7413

The same you can do with BreakIterator instead of using any external API.

Upvotes: 0

aab
aab

Reputation: 11494

OpenNLP will return the offsets using the method Tokenizer.tokenizePos(String s), see the OpenNLP API for TokenizerME as an example for one the implemented tokenizers. Each Span returned contains the start and end positions of the token.

Whether you decide to use UIMA is really a separate question, but OpenNLP does provide UIMA annotators for their tokenizers that use tokenizePos(). However, if you just want to tokenize a string, UIMA is definitely overkill...

Upvotes: 1

Himanshu Gahlot
Himanshu Gahlot

Reputation: 235

You can use OpenNLP Tokenizer with UIMA. The token annotator in UIMA will create a type for Token which will include the start and end indices of the token. You can also attach features like part-of-speech tag, stem, lemma, etc. to the token. UIMA has Java and C++ APIs.

Upvotes: 1

Related Questions