CODEWITHSUNDEEP
djangounicode
EagleOne
EagleOne

Reputation: 238

Django request with unicode strings wrongly UTF-8 encoded

I'm having some trouble making my website compatible with accented characters (french website).

I have a form where some field values can be with accented chars: "Coupé" for instance.

My URL looks like this:

http://localhost:8080/recherches/s?marque=Audi&modeles=A5+Coup%C3%A9

In my django view I do something like this:

def search(request):
  logger = logging.getLogger('custom')
  criteria_form = CriteriaForm(request.GET or None)
  logger.debug("search")
  logger.debug(request.GET)

And what I get in my logs is:

<QueryDict: {u'marque': [u'Audi'], u'modeles': [u'A5 Coup\xc3\xa9']}>

If I query my database with this variable "modeles", I get an error:

>>> mo = u'A5 Coup\xc3\xa9'
>>> Vehicule.objects.filter(valid=True, modele=mo)[0].marque.name
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/Library/Python/2.7/site-packages/django/db/models/query.py", line 211, in __getitem__
    return list(qs)[0]
IndexError: list index out of range

Things work if I query the database with the utf-8 version:

>>> mo = 'A5 Coup\xc3\xa9'
>>> Vehicule.objects.filter(valid=True, modele=mo)[0].marque.name
u'Audi'

So I think (but I might be wrong) that my problem comes from the fact that my variable is utf8 and then encoded with unicode.

How comes this is encoded that way?

UPDATE AFTER 1st RESPONSE:

On the header of the page that sends the form there is:

<meta http-equiv="Content-type" content="text/html;charset=UTF-8"/>

However if I print in my views.py the encoding:

logger.debug(request.encoding)

Then I get None.

But I don't know how to setup this encoding. I thought it would be from the header like I did above...

Also I have that in my HTTP_ACCEPT_CHARSET:

HTTP_ACCEPT_CHARSET ISO-8859-1,utf-8;q=0.7,*;q=0.3

Can that come from here? If yes, how should I change that?

Upvotes: 1

Views: 5046

Answers (2)

EagleOne
EagleOne

Reputation: 238

I have finally found the problem. I tested this problem with a brand new django application with nearly nothing in it. No DB, a simple view which displayed the content of the form in a page. Something like this:

# -*- coding: utf-8 -*-
from django.shortcuts import render_to_response
from django.template import RequestContext


def test(request):
  found = request.GET.get('modeles')
  print found
  return render_to_response('test.html',
                            {"found":found},
                            context_instance=RequestContext(request))

And if I opened the url 

localhost:8080/mysite?modeles=Coupé

I was getting the wrongly formatted Coupé

Here for sure I could not blame any ajax, or db call or anything I could have done in python.

So I just tried it with Django 1.4... And it worked like a charm! Then I tried with the latest Django 1.5a1 version and it also worked...

I guess I should not use the beta for now. I will stick with the 1.4!

Hope this saves time for somebody else.

Upvotes: 1

zxq9
zxq9

Reputation: 13164

With all the validation magic happening in the background, it may be easier to pinpoint the problem by bypassing all that. In your view, what does the following produce?

Vehicule.objects.filter(valid=True, modele=request.GET['mo'])[0].marque.name

If all you are doing is passing a query in, then your huge Ajaxariffic validation set is going to be doing weird Python things to your unicode before you get the value back, and possibly knocking your special characters off the end.

EDIT: After playing with this myself, it looks like the urlified data within a GET is the problem. Instead of doing that and string.encode('iso8859-1').decode('utf8'), it is easiser to change your form to submit POST data instead and use that.

Upvotes: 0

Related Questions