'utf8' codec can't decode byte 0xb5 in position 0: invalid start byte

Question

Before marking this as a duplicate, I want to make it clear that I have tried countless solutions to make this go away, by using from __future__ import unicode_literals to every pertumations and combinations of str.encode('utf8') and str.decode('utf8'), putting # -*- coding: utf-8 -*- at the start of the file and what not. I know I'm getting something wrong so I'll be as specific as possible, I'm converting a dictionary into a JSON Array/Object and showing it in it's raw string form on the webpage.

The unicode string which I'm having issue is the one starting with "µ" in a file name, thus the error occurs at the last fourth line in the below code. The files array is showing the value at the index of that string as \xb5Torrent.lnk.

if os.path.isdir(finalDirPath):
        print "is Dir"
        for (path,dir,files) in os.walk(finalDirPath):

            if dir!=[]:
                for i in dir:
                    if not hidden(os.path.join(path,i)):
                        # Here
                        JSONarray.append({"ext":"dir","path":b64(os.path.join(path,i)),"name":i})

            if files!=[]:
                for i in files:
                    if not hidden(os.path.join(path,i)):
                        # Here
                        JSONarray.append({"ext":i.split('.')[-1],"path":b64(os.path.join(path,i)),"name":i})
            break
        jsonStr = {"json":json.dumps(JSONarray)}
        return render(request,"json.html",jsonStr)

Here's the traceback:

Traceback (most recent call last):
File "C:\Python27\lib\site-packages\django\core\handlers\exception.py", line 39, in inner
response = get_response(request)
File "C:\Python27\lib\site-packages\django\core\handlers\base.py", line 249, in _legacy_get_response
response = self._get_response(request)
File "C:\Python27\lib\site-packages\django\core\handlers\base.py", line 187, in _get_response
response = self.process_exception_by_middleware(e, request)
File "C:\Python27\lib\site-packages\django\core\handlers\base.py", line 185, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "E:\ICT\Other\Python\Django\trydjango18\src\newsletter\views.py", line 468, in getJSON
JSONarray.append({"ext":i.split('.')[-1],"path":b64(os.path.join(path.encode('utf8'),i)),"name":i})
UnicodeDecodeError: 'utf8' codec can't decode byte 0xb5 in position 0: invalid start byte

Answer 1

A shorter example that demonstrates your problem:

>>> json.dumps('\xb5Torrent.lnk')

Traceback (most recent call last):
  File "<pyshell#17>", line 1, in <module>
    json.dumps('\xb5Torrent.lnk')
  File "C:\Python27\lib\json\__init__.py", line 243, in dumps
    return _default_encoder.encode(obj)
  File "C:\Python27\lib\json\encoder.py", line 201, in encode
    return encode_basestring_ascii(o)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xb5 in position 0: invalid start byte

Your array files contains byte strings, but json.dumps() wants any strings in the data to be unicode. It assumes that any byte strings are utf-8 encoded but your strings are using a different encoding: possibly latin1 or possibly something else. You need to find out the encoding used by your filesystem and decode all of the filenames to unicode before you add them to your JSONarray structure.

First thing is to check your filesystem encoding:

import sys
print sys.getfilesystemencoding()

should tell you the encoding used for the filenames and then you just make sure that all your path manipulations use unicode:

import sys
fsencoding = sys.getfilesystemencoding()
if os.path.isdir(finalDirPath):
    print "is Dir"
    for (path,dir,files) in os.walk(finalDirPath):
        path = path.decode(fsencoding)
        for i in dir:
            i = i.decode(fsencoding)
            if not hidden(os.path.join(path,i)):
                # Here
                JSONarray.append({
                    "ext": "dir",
                    "path": b64(os.path.join(path,i)),
                    "name": i})
                })

        for i in files:
            i = i.decode(fsencoding)
            if not hidden(os.path.join(path,i)):
                # Here
                JSONarray.append({
                    "ext": i.split('.')[-1],
                    "path": b64(os.path.join(path,i)),
                    "name":i
                })
        break
    jsonStr = {"json":json.dumps(JSONarray)}
    return render(request,"json.html",jsonStr)