CODEWITHSUNDEEP
pythonregex
Alicia
Alicia

Reputation: 1161

Unicode classes in regex with Python2

Is it possible?

This code works in Python3:

In [1]: import re

In [2]: re.split(r'\W+', 'Les Misérables')
Out[2]: ['Les', 'Misérables']

But it does not work in Python2:

In [1]: import re

In [2]: re.split(r'\W+', u'Les Misérables')
Out[2]: [u'Les', u'Mis', u'rables']

This does not work either (tested on Linux with es_ES.UTF-8 locale):

In [1]: import locale

In [2]: locale.setlocale(locale.LC_ALL, 'es_ES.UTF-8')
Out[2]: 'es_ES.UTF-8'

In [3]: import re

In [4]: re.split(ur'\W+', u'Les Misérables', re.U|re.L)
Out[4]: [u'Les', u'Mis', u'rables']

Is there any way to get regex working with Unicode in Python2?

Note: The question is about getting Unicode-aware matches. I know I can rewrite the above regex to separate words using only ASCII classes.

Upvotes: 3

Views: 111

Answers (1)

georg
georg

Reputation: 214949

Your mistake is that you're adding flags on the wrong place (flags should be the 4th param).

>>> import re
>>> re.split(r'(?u)\W+', u'Les Misérables')
[u'Les', u'Mis\xe9rables']
>>> re.split(ur'\W+', u'Les Misérables', 0, re.U)
[u'Les', u'Mis\xe9rables']

To avoid these issues I'd recommend using inline flags (like (?u) above).

Upvotes: 4

Related Questions