How to split on multiple unicode delimiters but still keeping the delimiter in the list?

Question

Given the string:

老師說：「你們要記住國父說的『青年要立志做大事，不要做大官』這句話。」

The task is to split the string based on a set of delimiter punctuations, i.e.

puncts = [u'!', u'"', u'#', u'$', u'%', u'&', u"'", u'(', u')', u'*', u'+', u',', u'-', u'.', u'/', u':', u';', u'<', u'=', u'>', u'?', u'@', u'[', u'\\', u']', u'^', u'_', u'`', u'{', u'|', u'}', u'~', u'\u2022', u'\u2026', u'\u3001', u'\u3002', u'\u300a', u'\u300b', u'\u300c', u'\u300d', u'\u300e', u'\u300f', u'\uff01', u'\uff08', u'\uff09', u'\uff0c', u'\uff1a', u'\uff1b', u'\uff1f']

The desired output is:

[u'\u8001\u5e2b\u8aaa', u'\uff1a', u'\u300c', u'\u4f60\u5011\u8981\u8a18\u4f4f\u570b\u7236\u8aaa\u7684', u'\u300e', u'\u9752\u5e74\u8981\u7acb\u5fd7\u505a\u5927\u4e8b', u'\uff0c', u'\u4e0d\u8981\u505a\u5927\u5b98', u'\u300f', u'\u9019\u53e5\u8a71', u'\u3002', u'\u300d']

I've looked at Python: Split string with multiple delimiters and the solution using re.split is pretty neat:

>>> x = u'\u8001\u5e2b\u8aaa\uff1a\u300c\u4f60\u5011\u8981\u8a18\u4f4f\u570b\u7236\u8aaa\u7684\u300e\u9752\u5e74\u8981\u7acb\u5fd7\u505a\u5927\u4e8b\uff0c\u4e0d\u8981\u505a\u5927\u5b98\u300f\u9019\u53e5\u8a71\u3002\u300d'
>>> [i for i in re.split(u"[{}]".format("|".join(puncts)), x, re.U)]
[u'\u8001\u5e2b\u8aaa', None, u'', None, u'\u4f60\u5011\u8981\u8a18\u4f4f\u570b\u7236\u8aaa\u7684', None, u'\u9752\u5e74\u8981\u7acb\u5fd7\u505a\u5927\u4e8b', None, u'\u4e0d\u8981\u505a\u5927\u5b98', None, u'\u9019\u53e5\u8a71', None, u'', None, u'']

Note: Sorry, for some reason SO thinks that the printed strings are spam, so you'll have to bare with the byte numbers =(

But the result from re.split threw away the delimiters which is needed.

Is there a way to keep the delimiters from `re.split`?

Are there other ways to split the string using the `puncts` list as multiple delimiters and achieved the desired output?

---

I've also tried to first pad all punctuations with spaces and then split based on spaces:

>>> y = x
>>> for p in puncts:
...     y = y.replace(p, u' {} '.format(p))
... 
>>> y
u'\u8001\u5e2b\u8aaa    \uff1a       \u300c   \u4f60\u5011\u8981\u8a18\u4f4f\u570b\u7236\u8aaa\u7684   \u300e   \u9752\u5e74\u8981\u7acb\u5fd7\u505a\u5927\u4e8b    \uff0c    \u4e0d\u8981\u505a\u5927\u5b98   \u300f   \u9019\u53e5\u8a71    \u3002       \u300d   '
>>> y.split()
[u'\u8001\u5e2b\u8aaa', u'\uff1a', u'\u300c', u'\u4f60\u5011\u8981\u8a18\u4f4f\u570b\u7236\u8aaa\u7684', u'\u300e', u'\u9752\u5e74\u8981\u7acb\u5fd7\u505a\u5927\u4e8b', u'\uff0c', u'\u4e0d\u8981\u505a\u5927\u5b98', u'\u300f', u'\u9019\u53e5\u8a71', u'\u3002', u'\u300d']

Is there a simpler way to achieve the same desired output?

Answer 1

You could convert your puncts list into a regular expression to split on as follows:

import re

text = u"老師說：「你們要記住國父說的『青年要立志做大事，不要做大官』這句話。」"
puncts = [u'!', u'"', u'#', u'$', u'%', u'&', u"'", u'(', u')', u'*', u'+', u',', u'-', u'.', u'/', u':', u';', u'<', u'=', u'>', u'?', u'@', u'[', u'\\', u']', u'^', u'_', u'`', u'{', u'|', u'}', u'~', u'\u2022', u'\u2026', u'\u3001', u'\u3002', u'\u300a', u'\u300b', u'\u300c', u'\u300d', u'\u300e', u'\u300f', u'\uff01', u'\uff08', u'\uff09', u'\uff0c', u'\uff1a', u'\uff1b', u'\uff1f']
puncts = [re.escape(x) for x in puncts]
my_re = re.compile(u'({})'.format(u'|'.join(puncts)))

print [x for x in my_re.split(text) if len(x)]

Giving you:

[u'\u8001\u5e2b\u8aaa', u'\uff1a', u'\u300c', u'\u4f60\u5011\u8981\u8a18\u4f4f\u570b\u7236\u8aaa\u7684', u'\u300e', u'\u9752\u5e74\u8981\u7acb\u5fd7\u505a\u5927\u4e8b', u'\uff0c', u'\u4e0d\u8981\u505a\u5927\u5b98', u'\u300f', u'\u9019\u53e5\u8a71', u'\u3002', u'\u300d']

The final list comprehension is used to remove any empty matches.

Answer 2

Document:

>>> re.split('(\W+)', 'Words, words, words.')
['Words', ', ', 'words', ', ', 'words', '.', '']