CODEWITHSUNDEEP
pythonregex
Kamilla
Kamilla

Reputation: 11

String with special characters don't work with Python and regex

I'm working in this project and the special characters are driving me crazy! I've searched a lot of solutions around the foruns but they didn't fix my problem.

I have this string with special characters:

['{"response":{"startRow":0,"endRow":5,"totalRows":5,"data":   [{"CODIGO":"72","DESCRICAO":"RECEITA INTRA-ORÇÁMENTÁRIAS DE CONTRIBUÇÕES","PREVISTA":225847716.0,"REALIZADA":165311075.58,"DIFERENCA":60536640.42,"R___":1.0},{"CODIGO":"76","DESCRICAO":"RECEITA  INTRA-ORÇAMENTÁRIAS DE SERVIÇOS","PREVISTA":22367493.0,"REALIZADA":3435363.08,"DIFERENCA":18932129.92,"R___":2.0},{"CODIGO":"77","DESCRICAO":"TRANSFERÊNCIAS  INTRA-ORÇAMENTÁRIAS CORRENTES","PREVISTA":1218252.0,"REALIZADA":0.0,"DIFERENCA":1218252.0,"R___":3.0},{"CODIGO":"71","DESCRICAO":"RECEITA TRIBUTÁRIA INTRA-ORÇAMENTÁRIA","PREVISTA":12000.0,"REALIZADA":0.0,"DIFERENCA":12000.0,"R___":4.0},{"CODIGO":"79","DESCRICAO":"OUTRAS RECEITAS INTRA-ORÇAMENTÁRIAS CORRENTES","PREVISTA":0.0,"REALIZADA":311785.30,"DIFERENCA":-311785.30,"R___":5.0}]}}']

And I have to find some specifics strings using regex but I have to mantain the special characters.

I've tried some things:

nkfd_form = unicodedata.normalize('NFKD', unicode(html))
print u"".join([c for c in nkfd_form if not unicodedata.combining(c)])

print ' '.join(re.findall(r'(?:\w{3,}|-(?=\s))', html))
print ' '.join(''.join([i if ord(i) < 128 else ' ' for i in html]).split())

And a lot of others things...

But when I search using my pattern:

result = re.findall('(:\"[\w\-r"/" ]+"|:[\w\s.\-r"/" ]+)', html, re.U)

The special characters aren't correct. The result is something like this:

[':0', ':2', ':2', ':"94"', ':"DEDU', ':0.0', ':-2748373.25', ':2748373.25', ':1.0', ':"95"', ':"DEDU', ':-1421484000.0', ':-1062829156.22', ':-358654843.78', ':2.0']
[':0', ':5', ':5', ':"72"', ':"RECEITA INTRA-OR', ':225847716.0', ':165311075.58', ':60536640.42', ':1.0', ':"76"', ':"RECEITA  INTRA-OR', ':22367493.0', ':3435363.08', ':18932129.92', ':2.0', ':"77"', ':"TRANSFER', ':1218252.0', ':0.0', ':1218252.0', ':3.0', ':"71"', ':"RECEITA TRIBUT', ':12000.0', ':0.0', ':12000.0', ':4.0', ':"79"', ':"OUTRAS RECEITAS INTRA-OR', ':0.0', ':311785.30', ':-311785.30', ':5.0']

It ignores the special characters!

I need it because I'll write data in a CSV files and it wont work with this errors.

A simple test using prompt:

>>> import re
>>> re.findall('\w+', 'Márquez', re.U)
['M\xc3', 'rquez']

What do I have to do to fix this?

Upvotes: 1

Views: 1403

Answers (2)

pillmuncher
pillmuncher

Reputation: 10162

Turning my comment into an answer (kind of, since it doesn't contain regex):

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import json
import csv
import cStringIO
import codecs
import types


class UnicodeDictWriter(csv.DictWriter):
    """
    A CSV DictWriter which will write rows to CSV file "f",
    which is encoded in the given encoding.
    """

    def __init__(self, f, fields, dialect=csv.excel, encoding="utf-8", **kwds):
        # Redirect output to a queue
        self.queue = cStringIO.StringIO()
        self.writer = csv.DictWriter(
            self.queue, fields, dialect=dialect, **kwds)
        self.stream = f
        self.encoder = codecs.getincrementalencoder(encoding)()

    def writerow(self, row):
        self.writer.writerow(dict(
            (f, v.encode("utf-8") if isinstance(v, types.StringTypes) else v)
                for f, v in row.iteritems()))
        # Fetch UTF-8 output from the queue ...
        data = self.queue.getvalue()
        data = data.decode("utf-8")
        # ... and reencode it into the target encoding
        data = self.encoder.encode(data)
        # write to the target stream
        self.stream.write(data)
        # empty queue
        self.queue.truncate(0)

    def writerows(self, rows):
        for row in rows:
            self.writerow(row)


data = '{"response":{"startRow":0,"endRow":5,"totalRows":5,"data":   [{"CODIGO":"72","DESCRICAO":"RECEITA INTRA-ORÇÁMENTÁRIAS DE CONTRIBUÇÕES","PREVISTA":225847716.0,"REALIZADA":165311075.58,"DIFERENCA":60536640.42,"R___":1.0},{"CODIGO":"76","DESCRICAO":"RECEITA  INTRA-ORÇAMENTÁRIAS DE SERVIÇOS","PREVISTA":22367493.0,"REALIZADA":3435363.08,"DIFERENCA":18932129.92,"R___":2.0},{"CODIGO":"77","DESCRICAO":"TRANSFERÊNCIAS  INTRA-ORÇAMENTÁRIAS CORRENTES","PREVISTA":1218252.0,"REALIZADA":0.0,"DIFERENCA":1218252.0,"R___":3.0},{"CODIGO":"71","DESCRICAO":"RECEITA TRIBUTÁRIA INTRA-ORÇAMENTÁRIA","PREVISTA":12000.0,"REALIZADA":0.0,"DIFERENCA":12000.0,"R___":4.0},{"CODIGO":"79","DESCRICAO":"OUTRAS RECEITAS INTRA-ORÇAMENTÁRIAS CORRENTES","PREVISTA":0.0,"REALIZADA":311785.30,"DIFERENCA":-311785.30,"R___":5.0}]}}'
field_order = [
    'CODIGO', 'DESCRICAO', 'PREVISTA', 'REALIZADA', 'DIFERENCA', 'R___']

with open('jsontest.csv', 'w') as csvfile:
    writer = UnicodeDictWriter(csvfile, field_order)
    writer.writerows(json.loads(data)['response']['data'])

Then jsontest.csv looks like this:

72,RECEITA INTRA-ORÇÁMENTÁRIAS DE CONTRIBUÇÕES,225847716.0,165311075.58,60536640.42,1.0
76,RECEITA  INTRA-ORÇAMENTÁRIAS DE SERVIÇOS,22367493.0,3435363.08,18932129.92,2.0
77,TRANSFERÊNCIAS  INTRA-ORÇAMENTÁRIAS CORRENTES,1218252.0,0.0,1218252.0,3.0
71,RECEITA TRIBUTÁRIA INTRA-ORÇAMENTÁRIA,12000.0,0.0,12000.0,4.0
79,OUTRAS RECEITAS INTRA-ORÇAMENTÁRIAS CORRENTES,0.0,311785.3,-311785.3,5.0

I used Python 2.6.8.

BTW: I adapted the UnicodeDictWriter class from here. Just scroll two or three screens down and you'll find the original UnicodeWriter class.

Upvotes: 1

Gung Foo
Gung Foo

Reputation: 13558

Your input seems to be unicode, but the regex isn't..

try changing ALL your regex patterns to unicode, like this:

print ' '.join(re.findall(ur'(?:\w{3,}|-(?=\s))', html))

Reference:

http://docs.python.org/tutorial/introduction.html#unicode-strings

Upvotes: 0

Related Questions