CODEWITHSUNDEEP
pythonpandasexport-to-csvpython-unicode
collarblind
collarblind

Reputation: 4739

Unicode Encode Error when writing pandas df to csv

I cleaned 400 excel files and read them into python using pandas and appended all the raw data into one big df.

Then when I try to export it to a csv:

df.to_csv("path",header=True,index=False)

I get this error:

UnicodeEncodeError: 'ascii' codec can't encode character u'\xc7' in position 20: ordinal not in range(128)

Can someone suggest a way to fix this and what it means?

Thanks

Upvotes: 35

Views: 62517

Answers (3)

Mohsen Rajabi Golmehr
Mohsen Rajabi Golmehr

Reputation: 39

Use "utf_8_sig"

    df.to_csv('df.csv', encoding='utf_8_sig')

Upvotes: 0

tangfucius
tangfucius

Reputation: 444

Adding an answer to help myself google it later:

One trick that helped me is to encode a problematic series first, then decode it back to utf-8. Like:

df['crumbs'] = df['crumbs'].map(lambda x: x.encode('unicode-escape').decode('utf-8'))

This would get the dataframe to print correctly too.

Upvotes: 19

unutbu
unutbu

Reputation: 879471

You have unicode values in your DataFrame. Files store bytes, which means all unicode have to be encoded into bytes before they can be stored in a file. You have to specify an encoding, such as utf-8. For example, 

df.to_csv('path', header=True, index=False, encoding='utf-8')

If you don't specify an encoding, then the encoding used by df.to_csv defaults to ascii in Python2, or utf-8 in Python3.

Upvotes: 70

Related Questions