CODEWITHSUNDEEP
pythonunicode
Sirius
Sirius

Reputation: 736

Special characters problems using Python unicode

#!/usr/bin/env python
# -*- coding: utf_8 -*-

def splitParagraphIntoSentences(paragraph):

''' break a paragraph into sentences
    and return a list '''
    import re
# to split by multile characters

#   regular expressions are easiest (and fastest)
    sentenceEnders = re.compile('[.!?][\s]{1,2}(?=[A-Z])')
    sentenceList = sentenceEnders.split(paragraph, re.UNICODE)
    return sentenceList


if __name__ == '__main__':
p = "While other species (e.g. horse mango, M. foetida) are also grown ,Mangifera indica – the common mango or Indian mango – Sheffield’s only mango tree is valued at £9.2 billion."

sentences = splitParagraphIntoSentences(p)
for s in sentences:
    print s.strip()

Expected Output: While other species (e.g. horse mango, M. foetida) are also grown ,Mangifera indica – the common mango or Indian mango – Sheffield’s only mango tree is valued at £9.2 billion."

Output Recieved: While other species (e.g. horse mango, M. foetida) are also grown ,Mangifera ind ica – the common mango or Indian mango – Sheffield’s only mango tree is va lued at £9.2 billion.

Ignore the meaning of the sentence, the main point is it isn't able to acess special characters such as " - ", " £ ", " ’ " and others. I tried setting sitecustomize.py file and this code with other encodings such as ascii, utf-32, cp-500, iso8859_15 and utf-8 but wasn`t able to solve it. Sorry I am new to python. Thanx in advance for the help.

Upvotes: 2

Views: 10374

Answers (4)

Sirius
Sirius

Reputation: 736

Have found the solution to this.

The following piece of code, works just fine.

p = p.encode('utf-8') if isinstance(p,unicode)  else p

Upvotes: 2

Mark Tolonen
Mark Tolonen

Reputation: 177610

Using Unicode string literals as Nam suggested is correct, but if your terminal is using the cp437 code page as your output suggests, it will not be able to display some of the Unicode characters you want to use. The Windows console doesn't support UTF-8, which is what you are sending if you declare coding: utf-81 in your source file and do not use Unicode literals. coding: utf-8 declares the encoding of your source file, so make sure you are actually saving your source in UTF-8 encoding.

When you use a Unicode literal, Python interprets the source string in the declared encoding, and converts it to a Unicode string. When printing a Unicode string, Python will encode the string in the terminal encoding, or lacking a terminal encoding, use a default encoding of ascii for Python 2.

An example:

# coding: utf8

print '£9.2 billion'  # Sends UTF-8 to cp437 terminal (gibberish)
print u'£9.2 billion' # Correctly prints on cp437 terminal.
print 'Sheffield’s'   # Sends UTF-8 to cp437 terminal (gibberish)

# Replaces Unicode characters that are unsupported in cp437.
print u'Sheffield’s £9.2 billion'.encode('cp437','xmlcharrefreplace')

print u'Sheffield’s'  # UnicodeEncodeError.

Output

£9.2 billion
£9.2 billion
SheffieldΓÇÖs
Sheffield’s £9.2 billion
Traceback (most recent call last):
  File "C:\Documents and Settings\metolone\Desktop\x.py", line 10, in <module>
    print u'SheffieldΓÇÖs'  # UnicodeEncodeError.
  File "C:\dev\python27\lib\encodings\cp437.py", line 12, in encode
    return codecs.charmap_encode(input,errors,encoding_map)
UnicodeEncodeError: 'charmap' codec can't encode character u'\u2019' in position 9: character maps to <undefined>

So, don't expect things to print all Unicode correctly on a Windows console. Use a Python IDE that supports UTF-8, such as PythonWin (available in the pywin32 extension).

You need two things to display Unicode characters properly in the Windows console: An encoding that maps the Unicode characters you want to display, and a font that supports the correct glyph for those characters. For your example, if you change the console code page to Windows-1252 (chcp 1252) and change the console font to Consolas or Lucida Console instead of Raster Fonts, your original program will work if you use Unicode literals (p = u"...").

Upvotes: 3

Nam Nguyen
Nam Nguyen

Reputation: 1793

p = "While other species..."

should be changed to

p = u"While other species..."

Notice the u in front of the quote.

What you need is a so-called Unicode literals. In Python 2, string literals is not Unicode by default.

Upvotes: 0

wberry
wberry

Reputation: 19347

That looks like cp437. Try this:

import codecs, sys
sys.stdout = codecs.getwriter('UTF-8')(sys.stdout)
print u"valued at £9.2 billion."

This works for me in Python 2.6.

Upvotes: 1

Related Questions