How do I write multiple API requests to a valid JSON file?

Question

I am trying to requests multiple api calls in a python script. As my code is set up now I am using requests to making connections and looping through the API urls. The JSON response is written to a file so I can manipulate the data externally. I am able to make the connection and write the response to a file without a problem. However my problem arises when I try to validate the completed file in jsonlint which tells me that I have multiple top level fields, which look to be the response headers. So the question is how can I loop through my APIs to write a single valid JSON file?

So far I have tried changing the JSON response into a python dictionary and now I am at a bit of a loss on what to try next..

Here is my request/file writing snippet:

for x, y in sites[data[z]].items():
    url = "".join(y['host'] + endpoint + 'customer_id=' + y['customer_id'] +
                  '&requestor_id=' + y['requestor_id'] + '&api_key=' + y['api_key'])
    urls = url + "&begin_date=" + begin + "&end_date=" + end

    r = requests.get(urls)  # make connections to vendors
    print("Connection Status: ", r.status_code)  # print http response code

    try:
        r.json()  # get data from vendors
    except json.decoder.JSONDecodeError:
        print(urls, "This is not a JSON format..")  # catch vendor JSON errors

    print("I'm saving your usage statistics to a text file.")
    with open(reportName + '-' + begin + '-' + end + '.json', 'a') as datafile:
        json.dump(r.json(), datafile)  # write api resp to .JSON file

    print("I'\'m done writing your usages to file:" + reportName + '-' + begin
          + '-' + end + ".json.")

Here is the api response

{  
   "Report_Header":{  },
   "Report_Items":[  ]
}{  
   "Report_Header":{  },
   "Report_Items":[  ]
}

Answer 1

Instead of collecting all the responses in a list and then writing them, you could construct a JSON file of this form,

{"responses":[
   {  
   "Report_Header":{  },
   "Report_Items":[  ]
   },
   {  
   "Report_Header":{  },
   "Report_Items":[  ]
   }
 ]
}

Which is actually a valid json object. You could achieve it by having the following modifications to your code:

with open(fileName, 'a') as datafile:
    datafile.write('{"responses":[')

for x, y in sites[data[z]].items():
    url = "".join(y['host'] + endpoint + 'customer_id=' + y['customer_id'] +
                  '&requestor_id=' + y['requestor_id'] + '&api_key=' + y['api_key'])
    urls = url + "&begin_date=" + begin + "&end_date=" + end

    r = requests.get(urls)  # make connections to vendors
    print("Connection Status: ", r.status_code)  # print http response code

    try:
        r.json()  # get data from vendors
    except json.decoder.JSONDecodeError:
        print(urls, "This is not a JSON format..")  # catch vendor JSON errors

    print("I'm saving your usage statistics to a text file.")
    with open(fileName, 'a') as datafile:
        json.dump(r.json(), datafile)  # write api resp to .JSON file
        datafile.write(",") # add comma for JSON array element

with open(fileName, 'a') as datafile:
    datafile.seek(0, os.SEEK_END) # Move to last 
    datafile.seek(datafile.tell() - 1, os.SEEK_SET) # back One character
    datafile.truncate() # Delete the last comma ","
    datafile.write(']}')

print("I'\'m done writing your usages to file:" + fileName)

Now you can parse the JSON file for your external uses however you want.

Answer 2

Here:

with open(reportName + '-' + begin + '-' + end + '.json', 'a') as datafile:
    json.dump(r.json(), datafile)

you're appending json snippets to the same file, which indeed does NOT produce valid json - ie this:

# cat youfile.json
{'foo': 'bar'}{'foo': 'baaz'}{'foo': 42}

is NOT valid.

if you want to save all the collected data in the same file, you have to first collect them into a list (or dict but you'll have to provide keys for each json snippet) and only then dump the result to a file opened in write mode (to make sure it's blank).