UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 2247: character maps to <undefined>

Question

When I run my code (Python 3) I keep getting this error:

Traceback (most recent call last):
  File "country.py", line 16, in <module>
    for row in csv_reader:
  File "C:\Users\benny\Anaconda3\lib\csv.py", line 112, in __next__
    row = next(self.reader)
  File "C:\Users\benny\Anaconda3\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 2247: character maps to <undefined>

I have tried these solutions but none work.

The code only prints one line if I fix the encoding problem by adding encoding='UTF-8. If I leave the encoding problem in place it prints almost 700 rows before it throws an error. Either way, it still won't work.

import csv
import country_converter as coco

with open('Interpol.csv', 'r') as csv_file, open('Interpol_Extra.csv', 'w', newline='') as new_file:

    csv_reader = csv.DictReader(csv_file)

    fieldnames = ['Case Happened - UN Region', 'Case Happened - Continent', 
    'Recovered - UN Region', 'Recovered - Continent'] + csv_reader.fieldnames

    csv_writer = csv.DictWriter(new_file, fieldnames)

    csv_writer.writeheader()

    for row in csv_reader:
        case_country_name = row['Case happened - Country']
        recovered_country_name = row['Recovered - Country']

        if case_country_name:
            row['Case Happened - UN Region'] = coco.convert(names=case_country_name, to='UNregion')
            row['Case Happened - Continent'] = coco.convert(names=case_country_name, to='Continent')

        if recovered_country_name:
            row['Recovered - UN Region'] = coco.convert(names=recovered_country_name, to='UNregion')
            row['Recovered - Continent'] = coco.convert(names=recovered_country_name, to='Continent')

    csv_writer.writerow(row)

Answer 1

This is the code I used which finally worked.

As suggested by Arun in the comments, if you're having a similar problem you should read all the answers on this question. It has the most succinct and helpful info on stack exchange for this problem.

And then re-check your code to make sure it is valid. In my case, it was some wrong indentation that finally fixed it.

import csv
import country_converter as coco

with open('Interpol.csv', 'r', encoding="utf-8") as csv_file, open('Interpol_Extra.csv', 'w', newline='', encoding="utf-8") as new_file:

    csv_reader = csv.DictReader(csv_file)

    fieldnames = ['Case Happened - UN Region', 'Case Happened - Continent', 
    'Recovered - UN Region', 'Recovered - Continent'] + csv_reader.fieldnames

    csv_writer = csv.DictWriter(new_file, fieldnames)

    csv_writer.writeheader()

    for row in csv_reader:
        case_country_name = row['Case happened - Country']
        recovered_country_name = row['Recovered - Country']

        if case_country_name:
            row['Case Happened - UN Region'] = coco.convert(names=case_country_name, to='UNregion')
            row['Case Happened - Continent'] = coco.convert(names=case_country_name, to='Continent')

        if recovered_country_name:
            row['Recovered - UN Region'] = coco.convert(names=recovered_country_name, to='UNregion')
            row['Recovered - Continent'] = coco.convert(names=recovered_country_name, to='Continent')

        csv_writer.writerow(row)