CODEWITHSUNDEEP
pythonregexnltktokenize
floer32
floer32

Reputation: 2270

RegEx Tokenizer: split text into words, digits, punctuation, and spacing (do not delete anything)

I almost found the answer to this question in this thread (samplebias's answer); however I need to split a phrase into words, digits, punctuation marks, and spaces/tabs. I also need this to preserve the order in which each of these things occurs (which the code in that thread already does).

So, what I've found is something like this:

    from nltk.tokenize import *
    txt = "Today it's   07.May 2011. Or 2.999."
    regexp_tokenize(txt, pattern=r'\w+([.,]\w+)*|\S+')
    ['Today', 'it', "'s", '07.May', '2011', '.', 'Or', '2.999', '.']

But this is the kind of list I need to yield:

    ['Today', ' ', 'it', "'s", ' ', '\t', '07.May', ' ', '2011', '.', ' ', 'Or', ' ', '2.999', '.']

Regex has always been one of my weakpoints so after a couple hours of research I'm still stumped. Thank you!!

Upvotes: 8

Views: 4993

Answers (3)

Andrew Clark
Andrew Clark

Reputation: 208475

I think that something like this should work for you. There is probably more in that regex than there needs to be, but your requirements are somewhat vague and don't exactly match up with the expected output you provided.

>>> txt = "Today it's \t07.May 2011. Or 2.999."
>>> p = re.compile(r"\d+|[-'a-z]+|[ ]+|\s+|[.,]+|\S+", re.I)
>>> slice_starts = [m.start() for m in p.finditer(txt)] + [None]
>>> [txt[s:e] for s, e in zip(slice_starts, slice_starts[1:])]
['Today', ' ', "it's", ' ', '\t', '07', '.', 'May', ' ', '2011', '.', ' ', 'Or', ' ', '2', '.', '999', '.']

Upvotes: 4

Savino Sguera
Savino Sguera

Reputation: 3572

Not fully compliant with the expected output you provided, some more details in the question would help, but anyway:

>>> txt = "Today it's   07.May 2011. Or 2.999."
>>> regexp_tokenize(txt, pattern=r"\w+([.',]\w+)*|[ \t]+")
['Today', ' ', "it's", ' \t', '07.May', ' ', '2011', ' ', 'Or', ' ', '2.999']

Upvotes: 0

MRAB
MRAB

Reputation: 20654

In the regex \w+([.,]\w+)*|\S+, \w+([.,]\w+)* captures words and \S+ captures other non-whitespace.

In order to capture spaces and tabs as well, try this: \w+([.,]\w+)*|\S+|[ \t].

Upvotes: 0

Related Questions