local dev server and app engine python utf 8

Question

In building a website in GAE. I did most of the work in the dev server where I have a database. In my dev server I inserted the entities in UTF-8 because if I don't the values are not writen. However, this makes me have to use {{ nom.titre.encode("utf-8")}} in order to display them correctly with jinja2. I know that GAE datastore only uses UNICODE so I could anticipate the problem that I'm facing now.

So, I used appcfg.py download_data --url=http://localhost:8080/_ah/remote_api/ --filename=filename.csv --kind=-kind- to get a backup of my dev datastore.

Now the problem is when I try to update to the server.

Either I use :

• appcfg.py upload_data --url=http://as.appspot.com/_ah/remote_api --kind=kind --filename=filename.csv But the {{ nom.titre.encode("utf-8")}} makes an error on GAE with jinja

OR

• appcfg.py create_bulkloader_config --filename=G:\Backup\datastore\bulkloader.yaml --url=http://toolsetdocs.appspot.com/_ah/remote_api with

  transformers:
  - kind: CISP
    connector: csv
    connector_options:
      encoding: utf-8
  

  and I get an error with UnicodeDecodeError : 'utf-8' codec can't decode ..... and so on

So, what is the most direct way to upload the info of my dev server datastore into the GAE datastore keeping the UTF-8 AND not changing the template encoding?

Answer 1

When you store the strings, you should use decode('utf-8') and then encode for display purposes. You are currently doing this in the template, but you should be decoding the expected UTF-8 string when you are storing it.