CODEWITHSUNDEEP
pythonutf-8pytorchpickleutf
Mona Jalal
Mona Jalal

Reputation: 38225

UnicodeDecodeError: ‘utf-8’ codec can’t decode byte 0xbe in position 2: invalid start byte

Do you know how I can fix this problem in PyTorch 1.9?

File "main.py", line 138, in main
    checkpoint = torch.load(args.resume)
  File "/scratch3/venv/fashcomp/lib/python3.8/site-packages/torch/serialization.py", line 608, in load
    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
  File "/scratch3/venv/fashcomp/lib/python3.8/site-packages/torch/serialization.py", line 787, in _legacy_load
    result = unpickler.load()
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xbe in position 2: invalid start byte

I have:

$ pip freeze
h5py==3.3.0
joblib==1.0.1
numpy==1.21.2
Pillow==8.3.1
scikit-learn==0.24.2
scipy==1.7.1
sklearn==0.0
threadpoolctl==2.2.0
torch==1.9.0
torchaudio==0.9.0
torchvision==0.10.0
typing-extensions==3.10.0.0

Upvotes: 2

Views: 5182

Answers (1)

Corralien
Corralien

Reputation: 120479

Can you try something like this:

>>> torch.load('model.pt', encoding='ascii')  # or latin1, or other encoding

By default, we decode byte strings as utf-8. This is to avoid a common error case UnicodeDecodeError: 'ascii' codec can't decode byte 0x... when loading files saved by Python 2 in Python 3. If this default is incorrect, you may use an extra encoding keyword argument to specify how these objects should be loaded, e.g., encoding='latin1' decodes them to strings using latin1 encoding, and encoding='bytes' keeps them as byte arrays which can be decoded later with byte_array.decode(...).

Read this, the 2nd note.

Upvotes: 3

Related Questions