CODEWITHSUNDEEP
pythonjson
Ronald Leenstra
Ronald Leenstra

Reputation: 13

iterating through json with python

I have learning python for a few months now, doing several online courses a now started on my own project. I have found an interesting website which provided data in a json-format which I import using the urllib library, parse using the json library and eventually want to insert this into an mysql database. The first and the last part of this project work, but not the parsing part. This is a part of the full json file:

{
"total": 24555,
"results": {
    "type": "FeatureCollection",
    "features": [
        {
            "type": "Feature",
            "geometry": null,
            "id": "556897c8b9a2143b1187cf79",
            "properties": {
                "Expedition": "okavango_15",
                "SpeciesName": "African Hoopoe",
                "Count": 1,
                "t_utc": 1325383950,
                "t_created": 1432917960.344582,
                "Member": "Maans",
                "Taxonomy": {
                    "Kingdom": "Animalia",
                    "Species": "Upupa epops",
                    "Family": "Upupidae",
                    "Genus": "Upupa",
                    "Phylum": "Chordata",
                    "Order": "Upupiformes",
                    "Class": "Aves"
                },
                "FeatureType": "sighting",
                "Images": [],
                "DateTime": "2012-01-01T04:12:30+0200",
                "EstimatedGeometry": null,
                "Behavior": "Flying",
                "Habitat": "Mioambo Woodland"
            }
        },
        {
            "type": "Feature",
            "geometry": null,
            "id": "556897b2b9a2143b1d14a807",
            "properties": {
                "Expedition": "okavango_15",
                "SpeciesName": "Peregrine Falcon",
                "Count": 1,
                "t_utc": 1325406263,
                "t_created": 1432917938.878641,
                "Member": "Maans",
                "Taxonomy": {
                    "Kingdom": "Animalia",
                    "Species": "Falco peregrinus",
                    "Family": "Falconidae",
                    "Genus": "Falco",
                    "Phylum": "Chordata",
                    "Order": "Falconiformes",
                    "Class": "Aves"
                },
                "FeatureType": "sighting",
                "Images": [],
                "DateTime": "2012-01-01T10:24:23+0200",
                "EstimatedGeometry": null,
                "Behavior": "Flying",
                "Habitat": "Mioambo Woodland"
            }
        },
        {
            "properties": {
                "SpeciesName": "Reed Cormorant",
                "Count": 1,
                "Accuracy": 8.0,
                "Expedition": "okavango_13",
                "t_created": 1430929467.352914,
                "Member": null,
                "FeatureType": "sighting",
                "Altitude": 973.2047942238661,
                "t_utc": 1378449600,
                "DateTime": "2013-09-06T08:40:00+0200",
                "Activity": "L",
                "SightingId": 0
            },
            "geometry": {
                "type": "Point",
                "coordinates": [
                    22.345429148371103,
                    -19.018043140479318
                ]
            },
            "id": "554a403bb9a21401a231284f",
            "type": "Feature"
        }
    ]
},
"resolution": "full",
"filter": {
    "FeatureType": "sighting"
},
"returned": 3,
"order": 1,
"limit": 3

}

For parsing the json file I use this code:

import json

def remove_nulls(d):
    return {k: v for k, v in d.iteritems() if v is not None}

with open('okavango.json') as fhand:
    data = json.loads(fhand.read(), object_hook=remove_nulls)

The level which I want to get the data from is in "features", which contains animal sightings with a lot of other keys like species name, timestamp, location. The various sightings don't always contain the same keys, location for example is not always present. I do want to parse the data sighting by sighting thought since I want it to be a single rows in my database. I am able to get every single bit of data I want by calling each individual item using this kind of code:

results = data["results"]

for item in results["features"]:
    value1 = item["id"]
    value2 = item["properties"]["t_created"]

But this makes my code very long and it should be easier to iterate i guess. However, I am not able to get single key/value results for each single . I have used the .get function, .values etc. but it turns out that not every level in my json file is a dictionary, but something it is a list. If I run this code for example:

print type(data)
data2 = data["results"]
print type(data2)
data3 = data2["features"]
print type(data3)

my result is this: type 'dict' type 'dict' type 'list'

How do I iterate through the original json load and parse the data by sighting?

Upvotes: 1

Views: 3318

Answers (1)

Ben
Ben

Reputation: 7124

I don't understand what you mean by "parse data by sighting", but the following code will iterate through your JSON. If you adjust your question to include what your desired output would be given the example, I could help more. 

data = json.loads(json_data)


def iterate_item(item):
    result_str = ""
    if type(item) == dict:
        for key in item:
            result_str = result_str + key + ": " + iterate_item(item[key])
    elif type(item) == list:
        for i in item:
            result_str = result_str + iterate_item(i) + ", "
    else:
        if not item:
            item = "None"
        result_str = result_str + str(item) + ". "

    return result_str

for sighting in data["results"]["features"]:
    print iterate_item(sighting) + "\n\n"

Upvotes: 2

Related Questions