CODEWITHSUNDEEP
pythonjsonpython-requests
Bigmoose70
Bigmoose70

Reputation: 473

Multiple GET Requests, merge json response into one array or json object

I am running one get request that returns some data in json format while also providing me the next url for the next page of data. I run a while loop grabbing all the data but want to append each new pages data to the existing object. Ultimately, I want one big json object or array.

Here is my code so far, I do not think append is the right move here, as it's creating an index within the array for each page. Rather I want one index or one json object with all the data combined.

host = 'https://xxxxx.com/api/v1/users'

headers = {'Accept': 'application/json', 'Content-Type': 'application/json',
           'Authorization': 'xxxxx'}

response = requests.get('{}'.format(host), 
                            headers = headers)
alist = []
keep_running = True
while keep_running:

    json_response = response.json()
    alist.append(json_response)
    host = response.links['next']['url']
    response = requests.get('{}'.format(host), 
                        headers = headers)    
    keep_running = response.status_code == requests.codes.ok and 'next' in response.links.keys()

There are 200 objects per page and about 18 pages. I am getting an array of length 18 with 200 objects within each index. Ideally, an array of length 18*200 = 3,600 would be want I'd want.

I can achieve creating a dataframe/table using pandas however, I'd also like it in raw json. Any ideas or help is appreciated.

Upvotes: 0

Views: 5612

Answers (2)

hurlenko
hurlenko

Reputation: 1425

Assuming your json_response is a list, it looks like list.extend() is what you're looking for. So instead of doing alist.append(json_response) use alist.extend(json_response).

append - adds an item to the end of the list.

extend - extends the list by appending all the items from the iterable.

Example:

x = [1, 2]

x.append([3, 4]) # gives [1, 2, [3, 4]]

x.extend([3, 4]) # gives [1, 2, 3, 4]

Upvotes: 1

S.D.
S.D.

Reputation: 2941

I'm not sure what you data actually looks like. But is sounds you can load the json-objects as a python-object. Append all of the data, and turn it back into a json-string afterwards.

Like so:

import json


...
alist = []
keep_running = True
while keep_running:

    json_response = response.json()
    alist.append(json.loads(json_response))
    host = response.links['next']['url']
    response = requests.get('{}'.format(host), 
                        headers = headers)    
    keep_running = response.status_code == requests.codes.ok and 'next' in response.links.keys()


json_string = json.dumps(alist)

Upvotes: 0

Related Questions