CODEWITHSUNDEEP
pythonjsonweb-scrapingpython-requests
JEJS
JEJS

Reputation: 101

UnicodeEncodeError Converting Requests Text to JSON in Python

I am trying to scrape website but with certain searches it gives me error: 

File "C:\Users\user\AppData\Local\Programs\Python\Python37\lib\encodings\cp1252.py", line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode characters in position 5399-5400: character maps to <undefined>

My Python script is: 

import requests
import json

url = 'https://www.protocols.io/api/v3/protocols?filter=%20public%20&order_field=relevance&key=%20gel%22electrophoresis%20'

r = requests.get(url)
text = r.text 
jason = json.loads(text)
print (jason)

I have tried to change the encoding as suggested here by 

text = text.encode('utf-8')

But still recieve the same error. It seems to be using the cp1252 encoding not utf-8?

Any suggestions will be greatly appreciated!

Update: This seems to be a issue in git bash or atom as works without error in Spyder or windows cmd

Upvotes: 0

Views: 133

Answers (2)

Edouard Yonga
Edouard Yonga

Reputation: 507

My OUTPUT: as you can see is a json of items

enter image description here

Upvotes: 0

Edouard Yonga
Edouard Yonga

Reputation: 507

Maybe is because you haven't imported sys,

like this:

import sys
import codecs

import requests
import json

url = 'https://www.protocols.io/api/v3/protocols?filter=%20public%20&order_field=relevance&key=%20gel%22electrophoresis%20'

r = requests.get(url)
text = r.text 
text = text.encode('utf-8') 
jason = json.loads(text)
print (jason)

Upvotes: 1

Related Questions