CODEWITHSUNDEEP
pythonpython-3.xutf-8web-scraping
Chae
Chae

Reputation: 193

Python - UnicodeEncodeError: 'charmap' codec can't encode characters in position 85-89: character maps to <undefined>

I am trying to see if I can transfer the output of urllib.request.urlopen() to a text file just to look at it. I tried decoding the output into a string so I can write into a file, but apparently the original output included some Korean characters that are not translating properly into the string.

So far I have:

from urllib.request import urlopen

openU = urlopen(myUrl)
pageH = openU.read()
openU.close()
stringU = pageH.decode("utf-8")

f=open("test.txt", "w+")
f.write(stringU)

I do not get any errors until the last step at which point it says: 

Traceback (most recent call last):  
  File "<stdin>", line 1, in <module>  
  File "C:\Users\Chae\AppData\Local\Programs\Python\Python36\lib\encodings\cp1252.py", line 19, in encode  
  return codecs.charmap_encode(input,self.errors,encoding_table)[0] 
UnicodeEncodeError: 'charmap' codec can't encode characters in position 85-89: character maps to `<undefined>`

Is there a way to get the string to also include Korean or if not, how do I skip the characters causing problems and write the rest of the string into the file?

Upvotes: 5

Views: 16563

Answers (1)

Robᵩ
Robᵩ

Reputation: 168636

Does it matter to you what the file encoding is? If not, then use utf-8 encoding:

f=open("test.txt", "w+", encoding="utf-8")
f.write(stringU)

If you want the file to be cp1252-encoded, which apparently is the default on your system, and to ignore unencodable values, add errors="ignore": 

f=open("test.txt", "w+", errors="ignore")
f.write(stringU)

Upvotes: 18

Related Questions