How to convert response.text to json in Scrapy

Question

I am scraping the financial data from below link using Scrapy:

Financial Data from Tencent

The reponse.body is like below:

I have tried to split the response using regular regression then convert it to json but it shows no json object, here is my code:

import scrapy
import re
import json

class StocksSpider(scrapy.Spider):
    name = 'stocks'
    allowed_domains = ['web.ifzq.gtimg.cn']
    start_urls = ['http://web.ifzq.gtimg.cn/appstock/hk/HkInfo/getFinReport?type=3&reporttime_type=1&code=00001&startyear=1990&endyear=2016&_callback=jQuery11240339550$']

    def start_requests(self):
        for url in self.start_urls:
            yield scrapy.Request(url=url, callback=self.parse,
            #endpoint='render.json', # optional; default is render.html
            #splash_url='<url>',     # optional; overrides SPLASH_URL
            #slot_policy=scrapy_splash.SlotPolicy.PER_DOMAIN,  # optional
            )

    def parse(self, response):
        try:
            json_data = re.search('\{\"data\"\:(.+?)\}\}\]', response.text).group(1)
        except AttributeError:
            json_data = ''
        #print json_data
        loaded_json = json.loads(json_data)

        print loaded_json

It throws an error saying that no json object can be decoded:

    Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/scrapy/utils/defer.py", line 102, in iter_errback
    yield next(it)
  File "/usr/local/lib/python2.7/dist-packages/scrapy_splash/middleware.py", line 156, in process_spider_output
    for el in result:
  File "/usr/local/lib/python2.7/dist-packages/scrapy/spidermiddlewares/offsite.py", line 30, in process_spider_output
    for x in result:
  File "/usr/local/lib/python2.7/dist-packages/scrapy/spidermiddlewares/referer.py", line 339, in <genexpr>
    return (_set_referer(r) for r in result or ())
  File "/usr/local/lib/python2.7/dist-packages/scrapy/spidermiddlewares/urllength.py", line 37, in <genexpr>
    return (r for r in result or () if _filter(r))
  File "/usr/local/lib/python2.7/dist-packages/scrapy/spidermiddlewares/depth.py", line 58, in <genexpr>
    return (r for r in result or () if _filter(r))
  File "/root/finance/finance/spiders/stocks.py", line 25, in parse
    loaded_json = json.loads(json_data)
  File "/usr/lib/python2.7/json/__init__.py", line 339, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python2.7/json/decoder.py", line 364, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python2.7/json/decoder.py", line 382, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
2018-06-09 23:54:26 [scrapy.core.engine] INFO: Closing spider (finished)

My goal is to convert it to json so that I can easily iterate the content. Is it necessary to convert it to json and how to convert in this case? The response is in unicode format so that I need to convert it to utf-8 as well? Is there any other good way to do iteration?

Answer 1

import re
import scrapy


class StocksSpider(scrapy.Spider):
    name = 'stocks'
    allowed_domains = ['gtimg.cn']
    start_urls = ['http://web.ifzq.gtimg.cn/appstock/hk/HkInfo/getFinReport?type=3&reporttime_type=1&code=00001&startyear=1990&endyear=2016&_callback=jQuery1124033955090772971586_1528569153921&_=1528569153953']

    def parse(self, response):
        try:
            json = eval(re.findall(r'jQuery\d+_\d+(\(\{.+\}\))', response.body)[0])
            print json
        except:
            self.log('Response couldn\'t be parsed, seems like it is having different format')

instead of converting in json use eval because at the end you're going to use it as a dict of lists and etc

may be like,

import re
import scrapy


class StocksSpider(scrapy.Spider):
    name = 'stocks'
    allowed_domains = ['gtimg.cn']
    start_urls = ['http://web.ifzq.gtimg.cn/appstock/hk/HkInfo/getFinReport?type=3&reporttime_type=1&code=00001&startyear=1990&endyear=2016&_callback=jQuery1124033955090772971586_1528569153921&_=1528569153953']

    def parse(self, response):
        data = eval(re.findall(r'jQuery\d+_\d+(\(\{.+\}\))', response.body)[0])
        items = data.get('data', {}).get('data', [])

        for item in items:
            yield item

or may be you can use json load instead eval it is also fine

Answer 2

The problem seems to be that the actual data is inside jQuery1124033955090772971586_1528569153921(). I was able to get rid of it by removing a parameter in the request url. If you absolutely needs it, this may do the trick:

>>> import json
>>> url = 'http://web.ifzq.gtimg.cn/appstock/hk/HkInfo/getFinReport?type=3&reporttime_type=1&code=00001&startyear=1990&endyear=2016&_callback=jQuery1124033955090772971586_1528569153921&_=1528569153953'
>>> fetch(url)
2018-06-09 21:55:13 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://web.ifzq.gtimg.cn/appstock/hk/HkInfo/getFinReport?type=3&reporttime_type=1&code=00001&startyear=1990&endyear=2016&_callback=jQuery1124033955090772971586_1528569153921&_=1528569153953> (referer: None)
>>> data = response.text.strip('jQuery1124033955090772971586_1528569153921()')
>>> parsed_data = json.loads(data)

If you prefer to remove the _callback parameter from the url, simply:

>>> import json
>>> url = 'http://web.ifzq.gtimg.cn/appstock/hk/HkInfo/getFinReport?type=3&reporttime_type=1&code=00001&startyear=1990&endyear=2016&_=1528569153953'
>>> fetch(url)
2018-06-09 21:53:36 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://web.ifzq.gtimg.cn/appstock/hk/HkInfo/getFinReport?type=3&reporttime_type=1&code=00001&startyear=1990&endyear=2016&_=1528569153953> (referer: None)
>>> parsed_data = json.loads(response.text)

Answer 3

As bla said without &_callback=jQuery1124033955090772971586_1528569153921 the data is vaild json, callback is not required also is not static, for example http://web.ifzq.gtimg.cn/appstock/hk/HkInfo/getFinReport?type=3&reporttime_type=1&code=00001&startyear=1990&endyear=2016&_callback=test is gives the same results