CODEWITHSUNDEEP
pythonjsonpython-2.7dictionary
Volatil3
Volatil3

Reputation: 14998

Unable to convert returned JSON data into Dict in Scrapy

I am using scrapy to get a certain piece of data from here. As I was suggested here, I used the following code in my script:

pattern = re.compile(r"qubit_product_list = (.*?);", re.M)
        script = hxs.select("//script[contains(., 'qubit_product_list')]/text()").extract()[0]
        data = pattern.search(script).group(1)

        j_data = json.loads(data)
        self.log('After calling LOAD Begins')
        self.log(j_data) #It is not printing ANYTHING!!!!
        self.log('After calling LOAD Ends')

        self.log('\n---------------------------------\n')

Which outputs following from variable data:

{
    "9102-DBL-sprung slat base": {
        "id": "9102",
        "name": "Imperial Bedstead",
        "url": "/p/Imperial_Bedstead.htm",
        "description": "Double - Sprung Slat Base",
        "unit_price": 429.99,
        "unit_sale_price": 429.99,
        "currency": "GBP",
        "sku_code": "BENT:1320B-Beech",
        "category": "Bed Frames",
        "stock": 100
    },
    "9102-KS-sprung slat base": {
        "id": "9102",
        "name": "Imperial Bedstead",
        "url": "/p/Imperial_Bedstead.htm",
        "description": "Kingsize - Sprung Slat Base",
        "unit_price": 439.98996,
        "unit_sale_price": 439.98996,
        "currency": "GBP",
        "sku_code": "BENT:1326B-Beech",
        "category": "Bed Frames",
        "stock": 100
    }
}

Now, I want to convert this json like structure to python dict. I tried following but it returns unicode type.

j_data = json.loads(data)

So, how do I get Array/Dict in Python 2.7? Ironically same loads method is returning of type dict when using scrapy shell.

Upvotes: 0

Views: 610

Answers (1)

sid1408
sid1408

Reputation: 424

Try this:

#typecasting the JSON to string for json.loads to work
data = str(data)
#returning type dict from json
j_data = json.loads(data)
#typecasting the dict to string before writing to log
self.log(str(j_data))

Upvotes: 1

Related Questions