CODEWITHSUNDEEP
pythonspacystringtokenizer
Milos Cuculovic
Milos Cuculovic

Reputation: 20223

Spacy: how to add the colon character in the list of special case tokenization rules

I have the following sentence:

'25) Figure 9:“lines are results of two-step adsorption model” -> What method/software was used for the curve fitting?'

I would like to separate the colon from the rest of the words.

By default, here is what Spacy returns:

print([w.text for w in nlp('25) Figure 9:“lines are results of two-step adsorption model” -> What method/software was used for the curve fitting?')])

['25', ')', 'Figure', '9:“lines', 'are', 'results', 'of', 'two', '-', 'step', 'adsorption', 'model', '”', '-', '>', 'What', 'method', '/', 'software', 'was', 'used', 'for', 'the', 'curve', 'fitting', '?']

What I would like to get is:

['25', ')', 'Figure', '9', ':', '“', lines', 'are', 'results', 'of', 'two', '-', 'step', 'adsorption', 'model', '”', '-', '>', 'What', 'method', '/', 'software', 'was', 'used', 'for', 'the', 'curve', 'fitting', '?']

I was trying to do:

# Add special case rule
special_case = [{ORTH: ":"}]
nlp.tokenizer.add_special_case(":", special_case)

But no results, the print shows the same value.

Upvotes: 2

Views: 456

Answers (2)

Sergey Bushmanov
Sergey Bushmanov

Reputation: 25249

Try modifying nlp.tokenizer.infix_finditer with compile_infix_regex:

import spacy
from spacy.util import compile_infix_regex

text = "'25) Figure 9:“lines are results of two-step adsorption model” -> What method/software was used for the curve fitting?'"
 
nlp = spacy.load("en_core_web_md")
infixes = (":",) + nlp.Defaults.infixes
infix_regex = spacy.util.compile_infix_regex(infixes)
nlp.tokenizer.infix_finditer = infix_regex.finditer

doc = nlp(text)

for tok in doc:
    print(tok, end =", ")

', 25, ), Figure, 9, :, “lines, are, results, of, two, -, step, adsorption, model, ”, -, >, What, method, /, software, was, used, for, the, curve, fitting, ?, ',

Upvotes: 3

Teo
Teo

Reputation: 1

Simply use word_tokenize

from nltk.tokenize import word_tokenize 
string_my='25) Figure 9:“lines are results of two-step adsorption model” -> What method/software was used for the curve fitting?'
word_tokenize(string_my) 

['25', ')', 'Figure', '9', ':', '“', 'lines', 'are', 'results', 'of', 'two-step', 'adsorption', 'model', '”', '-', '>', 'What', 'method/software', 'was', 'used', 'for', 'the', 'curve', 'fitting', '?']

Upvotes: 0

Related Questions