CODEWITHSUNDEEP
pythonjsonscrapy
Svarto
Svarto

Reputation: 643

Scrapy on a JSON response

I am having a problem scraping a json response in Scrapy. I am getting the following error:

TypeError: the JSON object must be str, not 'Response'

I have tried to decode it using the following by importing codecs and decoding the response to utf-8, but it isn't working for some reason. The code:

import scrapy
import json

class SrealitkyBuySpiderSpider(scrapy.Spider):
    name = "srealitky_buy"
    allowed_domains = ["https://www.sreality.cz/"]
    start_urls = ['https://www.sreality.cz/api/cs/v2/estates?category_main_cb=1&category_type_cb=1&per_page=20&region=Praha']

    def parse(self, response):
        jsonresponse = json.loads(response)
        print(jsonresponse)

I also tried adding .text to the response when using json.loads, but then I get the error that "response has no attribute 'text'".

jsonresponse = json.loads(response.text)

Does anyone know what is wrong?

Upvotes: 2

Views: 4950

Answers (1)

Tarun Lalwani
Tarun Lalwani

Reputation: 146510

If you look at type(response) it would be scrapy.http.response.text.TextResponse

And response.body will give b'{"meta_description": "5192 realit v nab\\u00eddce prodej byt\\u016f, .... "collective"}}'

So you need to use

jsonresponse = json.loads(response.body)
print(jsonresponse["meta_description"])
'5192 realit v nabídce prodej bytů, Praha. Vyberte si novou nemovitost na sreality.cz s hledáním na mapě a velkými náhledy fotografií nabízených bytů.'

Edit-1

If the above doesn't work for you because of Python version try below

jsonresponse = json.loads(response.body.decode("utf-8"))

Upvotes: 4

Related Questions