CODEWITHSUNDEEP
pythonjsonweb-scrapingscrapy
johncsmith427
johncsmith427

Reputation: 83

Scrapy JSON get value

Using this link I am trying to get the following data after every instance of this which would be 100 data sets

contributor-limited-meta":{"data":{"id":"4103089"

The expected result would be:

768919, 4103089, 193246966, 976367, 508762, and more

I have tried response.xpath('//script').re(r'author":"([0-9.]+?)"') via the scrapy shell but I cannot get it to work.

Upvotes: 0

Views: 151

Answers (1)

QHarr
QHarr

Reputation: 84465

You could use requests then use json from response and extract in a loop

import requests
res = requests.get('https://www.shutterstock.com/sstk/api/footage/images/search?site=image&image_type=vector&q=&page%5Bnumber%5D=1&studio=1&include=contributor-limited-meta').json()
results  = []
for item in res['data']:
    id = item['relationships']['contributor-limited-meta']['data']['id']
    results.append(id)
print(results)

With scrapy use json 

jsonres = json.loads(response.body_as_unicode()

Upvotes: 2

Related Questions