CODEWITHSUNDEEP
pythondjangopostgresqlutf-8pgadmin
Mathias.r
Mathias.r

Reputation: 41

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 460: invalid continuation byte

I've got a problem with the command loaddata of my Django app.

When I run this command, this UnicodeDecodeError appears because of the "é" that are in my database table.

i've try with many type of database on pgAdmin with different encode type like "UTF-8", "latin-1" or "win1252" or different "character type" or "collation" settings like "C" or "french_switzerland.1252" but every time there is this error.

Here is more information about my work space.

Let me know if you need more informations, I don't give you too much, I don't want to give useless information.

I know there are other questions which look similar to mine but I don't understand the answers. There are explanations about why there is the error but not about how to solve it.

the complete error :

(venv) C:\Users\Mathias\PycharmProjects\yufindProject>python manage.py loaddata ask/dumps/ask.json
Traceback (most recent call last):
  File "manage.py", line 22, in <module>
    main()

  File "manage.py", line 18, in main
    execute_from_command_line(sys.argv)

  File "C:\Users\Mathias\PycharmProjects\yufindProject\venv\lib\site-packages\django\core\management\__init__.py", line 401, in execute_from_command_line
    utility.execute()

  File "C:\Users\Mathias\PycharmProjects\yufindProject\venv\lib\site-packages\django\core\management\__init__.py", line 395, in execut
    self.fetch_command(subcommand).run_from_argv(self.argv)

  File "C:\Users\Mathias\PycharmProjects\yufindProject\venv\lib\site-packages\django\core\management\base.py", line 330, in run_from_argv
    self.execute(*args, **cmd_options)

  File "C:\Users\Mathias\PycharmProjects\yufindProject\venv\lib\site-packages\django\core\management\base.py", line 371, in execute
    output = self.handle(*args, **options)

  File "C:\Users\Mathias\PycharmProjects\yufindProject\venv\lib\site-packages\django\core\management\commands\loaddata.py", line 72, in handle
    self.loaddata(fixture_labels)

  File "C:\Users\Mathias\PycharmProjects\yufindProject\venv\lib\site-packages\django\core\management\commands\loaddata.py", line 114, in loaddata
    self.load_label(fixture_label)

  File "C:\Users\Mathias\PycharmProjects\yufindProject\venv\lib\site-packages\django\core\management\commands\loaddata.py", line 172, in load_label
    for obj in objects:

  File "C:\Users\Mathias\PycharmProjects\yufindProject\venv\lib\site-packages\django\core\serializers\json.py", line 67, in Deserializer
    stream_or_string = stream_or_string.decode()

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 460: invalid continuation byte

Upvotes: 2

Views: 2469

Answers (2)

Gray
Gray

Reputation: 11

I don't know can it help you or not. I had other operation, but error at your. I maked migration DB from Sqlite3 to Postgres in Django. First i make dump

py -Xutf8 manage.py dumpdata --indent=2 --output=mysite_data.json

After i change settings in DATABASES setting.py to Postgres db. When i begin load data from my created early file mysite_data.json i use command

python manage.py loaddata mysite_data.json

I got error as your.

Me helped command

python manage.py loaddata --exclude auth.permission --exclude contenttypes mysite_data.json

Upvotes: 0

Ron
Ron

Reputation: 1508

Found the solution in Django dumpdata fails on special characters

To save json data in django the TextIOWrapper is used:

The default encoding is now locale.getpreferredencoding(False) (...)

In documentation of locale.getpreferredencoding fuction we can read:

Return the encoding used for text data, according to user preferences. User preferences are expressed differently on different systems, and might not be available programmatically on some systems, so this function only returns a guess.

Here I found "hacky" but working method to overwrite these settings:

In file settings.py of your django project add these lines:

import _locale
_locale._getdefaultlocale = (lambda *args: ['en_US', 'utf8'])

Upvotes: 0

Related Questions