CODEWITHSUNDEEP
pythonjsonscrapy
pep
pep

Reputation: 41

Scrapy parsing JSON output

I am using Scrapy to crawl a website. Some pages use AJAX so I got the AJAX requests to get the actual data. so far so good. The output of those AJAX requests is JSON outputs. Now I would like to parse the JSON but Scrapy just provides HtmlXPathSelector. Has anybody transformed successfully a JSON output into HTML and been able to parse it with HtmlXPathSelector?

Thank you very much in advance

Upvotes: 4

Views: 3081

Answers (2)

Sravan
Sravan

Reputation: 61

Slightly complicated, still works.

If you're interested in working with xpaths on JSON outputs..

Disclaimer : May not be the optimal soln. +1 if someone improves this approach.

install dicttoxml package (pip recommended)

-Download the output using scrapy's traditional Request module

in spider: 

from scrapy.selector import XmlXPathSelector
import lxml.etree as etree

request = Request(link, callback=self.parse_resp)
yield request

def parse_resp(self,response):
     json=response.body
     #Now load the contents using python's JSON module
     json_dict = json.loads(json)
     #transform the contents into xml using dicttoxml
     xml = dicttoxml.dicttoxml(json_dict)
     xml = etree.fromstring(xml)
     #Apply scrapy's XmlXPathSelector module,and start using xpaths
     xml = XmlXPathSelector(text=xml)
     data = xml.select(".//*[@id='count']/text()").extract()
     return data

I did this because, i'm maintaining all the xpaths of all the spiders in one place (config-files)

Upvotes: 0

Ionut Hulub
Ionut Hulub

Reputation: 6326

import json

response = json.loads(jsonResponse)

The code above will decode the json you receive. Afterwards, you should be able to process it any way you want.

(Replace jsonResponse with the json that you get from the ajax request)

Upvotes: 5

Related Questions