CODEWITHSUNDEEP
pythonjsonweb-scrapingscrapy
emon
emon

Reputation: 35

Scraping json response using scrapy

JSON RESPONSE FROM WEBSITE I am new to python scrapy and json . I am trying to scrape json response from 78751 . But it is showing error . The code i used is 

import scrapy
import json 
class BlackSpider(scrapy.Spider):
    name = 'black'
    start_urls = ['https://appworld.blackberry.com/cas/content/2360/reviews/2.17.2?page=1&pagesize=100&sortby=newest&callback=_content_2360_reviews_2_17_2&_=1499161778751']





    def parse(self, response):
         data = re.findall('(\{.+\})\);', response.body_as_unicode())
         a=json.loads(data[0])

         item = MyItem()
         item["Reviews"] = a["reviews"][4]["review"]           

         return item

The error it is showing is ValueError("No JSON object could be decoded")ERROR

Upvotes: 0

Views: 1124

Answers (1)

Granitosaurus
Granitosaurus

Reputation: 21436

The response you are getting is javascript function with some json in it:

_content_2360_reviews_2_17_2(\r\n{"some":"json"}]});\r\n

To extract the data from this you can use simple regex solution:

import re
import json
data = re.findall('(\{.+\})\);', response.body_as_unicode())
json.loads(data[0])

It trasnslates to: select everything between {} that ends with );

edit: results I'm getting with this: 

{'platform': None,
 'reviews': [{'createdDate': '2017-07-04',
   'model': 'London',
   'nickname': 'aravind14-92362',
   'rating': 6,
   'review': 'Very bad ',
   'title': 'My WhatsApp no update '}],
 'totalReviews': 569909,
 'version': '2.17.2'}

Upvotes: 1

Related Questions