Create directories from a json

Question

I am working with a list of nested dictionaries loaded into a public Singapore API that automatically updates the data every 15 seconds. First, I am only extracting the nested data I need with the extract_values function. Nevertheless, my main goal is to create one directory with 87 directories of each camera id. Then store the image property in the correspondent directory of each camera id.

This is a part of the JSON

{
  "items": [
    {
      "timestamp": "2020-06-12T00:56:28+08:00",
      "cameras": [ {
          "timestamp": "2020-06-12T00:55:28+08:00",
          "image": "https://images.data.gov.sg/api/traffic-images/2020/06/ce03a29a-7c9e-4e08-aadb-2afcc25db586.jpg",
          "location": {
            "latitude": 1.323604823,
            "longitude": 103.8587802
          },
          "camera_id": "1701",
          "image_metadata": {
            "height": 480,
            "width": 640,
            "md5": "75c9ab0b027e5e8aca9de09ef0d1b7b1"
          }
        },
        {
          "timestamp": "2020-06-12T00:55:28+08:00",
          "image": "https://images.data.gov.sg/api/traffic-images/2020/06/ee537eb0-c7dd-474e-88a2-2615df74d28a.jpg",
          "location": {
            "latitude": 1.34355015,
            "longitude": 103.8601984
          },
          "camera_id": "1702",
          "image_metadata": {
            "height": 480,
            "width": 640,
            "md5": "1de3dfc757a5f79d137769e0c5fe75ef"
          }
        },
        {
          "timestamp": "2020-06-12T00:55:28+08:00",
          "image": "https://images.data.gov.sg/api/traffic-images/2020/06/e3840170-d8d5-41fd-8cdd-c6afcea96249.jpg",
          "location": {
            "latitude": 1.28569398886979,
            "longitude": 103.837524510188
          },
          "camera_id": "1704",
          "image_metadata": {
            "height": 480,
            "width": 640,
            "md5": "9cfd60fd0025b2ef6ee44177d50ad892"
          }
        },.....]
    }
  ],
  "api_info": {
    "status": "healthy"
  }
}

Here's my code

import json 
import requests


"""path = "C:/Users/Issstezac1/Desktop" """
endpoint = "https://api.data.gov.sg/v1/transport/traffic-images"

"""response from API URL"""
get_req = requests.get(endpoint)

"""Transform json input to python objects""" 
data = json.loads(get_req.text)

##Prints all the json data
##print(data.get('items'))

def extract_values(obj, key):
    """Pull all values of the specified key from nested JSON."""
    arr = []

    def extract(obj, arr, key):
        """Recursively search for values of key in JSON tree."""
        if isinstance(obj, dict):
            for k, v in obj.items():
                if isinstance(v, (dict, list)):
                    extract(v, arr, key)
                elif k == key:
                    arr.append(v)
        elif isinstance(obj, list):
            for item in obj:
                extract(item, arr, key)
        return arr

    results = extract(obj, arr, key)
    return results

names = extract_values(data, 'camera_id')
print(names)

Could somebody guide me on how to achieve this? Any further help will be appreciated

Answer 1

It's simple actually. os module allow us to run operations on system base. You just need a loop for handing all of cameras, create a dictionary for data if not exists and then download the image inside this dictionary. I just created a simple example for your endpoind/data.

import os
import ntpath
import shutil
import requests

endpoint = "https://api.data.gov.sg/v1/transport/traffic-images"
data = requests.get(endpoint).json()

output_data_path = "./output_data"

if not os.path.exists(output_data_path): # check for is directory exists
    os.mkdir(output_data_path) # create the directory if not exists

for record in data["items"][0]["cameras"]:
    camera_id = record["camera_id"]
    image_location = record["image"]
    image_name = ntpath.basename(image_location)

    if not os.path.exists(f"{output_data_path}/{camera_id}"):
        os.mkdir(f"{output_data_path}/{camera_id}")
    
    image_data = requests.get(image_location, stream=True) # get the source of image
    with open(f"{output_data_path}/{camera_id}/{image_name}", 'wb') as image_file:
        shutil.copyfileobj(image_data.raw, image_file) # write the source into a file

    print("Downloaded an image for camera id:", camera_id)

Answer 2

If you are looking for a method of creating directories, you can use os.mkdir()