CODEWITHSUNDEEP
pythonjsonweb-scrapingscrapyweb-crawler
Cristian Calin
Cristian Calin

Reputation: 27

Extract from dynamic JSON response with Scrapy

I want to extract the 'avail' value from the JSON output that look like this. 

{
    "result": {
        "code": 100,
        "message": "Command Successful"
    },
    "domains": {
        "yolotaxpayers.com": {
            "avail": false,
            "tld": "com",
            "price": "49.95",
            "premium": false,
            "backorder": true
        }
    }
}

The problem is that the ['avail'] value is under ["domains"]["domain_name"] and I can't figure out how to get the domain name.

You have my spider below. The first part works fine, but not the second one. 

import scrapy
import json
from whois.items import WhoisItem

class whoislistSpider(scrapy.Spider):
    name = "whois_list"
    start_urls = []
    f = open('test.txt', 'r')
    global lines
    lines = f.read().splitlines()
    f.close()
    def __init__(self):
        for line in lines:
            self.start_urls.append('http://www.example.com/api/domain/check/%s/com' % line)

    def parse(self, response):
        for line in lines:
            jsonresponse = json.loads(response.body_as_unicode())
            item = WhoisItem()
            domain_name = list(jsonresponse['domains'].keys())[0]
            item["avail"] = jsonresponse["domains"][domain_name]["avail"]
            item["domain"] = domain_name
            yield item

Thank you in advance for your replies.

Upvotes: 1

Views: 584

Answers (3)

vikas0713
vikas0713

Reputation: 596

To get the domain name from above json response you can use list comprehension , e.g:

domain_name = [x for x in jsonresponse.values()[0].keys()]

To get the "avail" value use same method, e.g:

avail = [x["avail"] for x in jsonresponse.values()[0].values() if "avail" in x]

to get the values in string format you should call it by index 0 e.g:

domain_name[0] and avail[0] because list comprehension results stored in list type variable.

More info on list comprehension

Upvotes: 0

Dean Fenster
Dean Fenster

Reputation: 2395

Assuming you are only expecting one result per response:

domain_name = list(jsonresponse['domains'].keys())[0]
item["avail"] = jsonresponse["domains"][domain_name]["avail"]

This will work even if there is a mismatch between the domain in the file "test.txt" and the domain in the result.

Upvotes: 0

alecxe
alecxe

Reputation: 473763

Currently, it tries to get the value by the "('%s.com' % line)" key.

You need to do the string formatting correctly:

domain_name = "%s.com" % line.strip()
item["avail"] = jsonresponse["domains"][domain_name]["avail"]

Upvotes: 1

Related Questions