Python 'list' object has no attribute 'keys' when trying to write a row in CSV file

Question

I am trying to write a new row into a CSV file and I can't because I get an error in Python Shell.

Below is the code I am using (I am reading JSON from API and want to put data into CSV file)

# import urllib library
from urllib.request import Request, urlopen
c=1
# import json
import json
# store the URL in url as 
# parameter for urlopen
import pandas as pd
import csv
headerList = ['name','id','order','height','weight','speed','special_defense','special_attack','defense','attack','hp']
  
# open CSV file and assign header
with open("pokemon_stats.csv", 'w') as file:
    dw = csv.DictWriter(file, delimiter=',', 
                        fieldnames=headerList)
    dw.writeheader()
  
# display csv file
fileContent = pd.read_csv("pokemon_stats.csv")
for r in range(1,3):
    req = Request('https://pokeapi.co/api/v2/pokemon/'+str(r)+'/', headers={'User-Agent': 'Chrome/32.0.1667.0'})
  
# store the response of URL
    response = urlopen(req)

# storing the JSON response 
# from url in data
    data_json = json.loads(response.read())

#print(data_json)

    for key, value in data_json.items():
        if key=='name':
            name=value
        elif key=='id':
            id=value
        elif key=='order':
            order=value
        elif key=='height':
            height=value
        elif key=='weight':
            weight=value
        elif key == 'stats':
            for sub in data_json['stats']:
                for i in sub:
                    if i=='base_stat':
                        base_stat=sub[i]
                    if i=='stat':
                        for j in sub[i]:
                            if j=='name':
                                stat_name=sub[i][j]
                                if stat_name=='hp':
                                    hp=base_stat
                                elif stat_name=='attack':
                                    attack=base_stat
                                elif stat_name=='defense':
                                    defense=base_stat
                                elif stat_name=='special-attack':
                                    special_attack=base_stat
                                elif stat_name=='special-defense':
                                    special_defense=base_stat
                                elif stat_name=='speed':
                                    speed=base_stat
    data = [name,id,order,height,weight,speed,special_defense,special_attack,defense,attack,hp]
    dw.writerow(data)

After I try the execution of this code I get an error as it follows:

Traceback (most recent call last):
  File "C:/Users/sbelcic/Desktop/NANOBIT_API.py", line 117, in <module>
    dw.writerow(data)
  File "C:\Users\sbelcic\AppData\Local\Programs\Python\Python37\lib\csv.py", line 155, in writerow
    return self.writer.writerow(self._dict_to_list(rowdict))
  File "C:\Users\sbelcic\AppData\Local\Programs\Python\Python37\lib\csv.py", line 148, in _dict_to_list
    wrong_fields = rowdict.keys() - self.fieldnames
AttributeError: 'list' object has no attribute 'keys'*

Can somebody pls help and tell me what I am doing wrong.

I don't have working experience of manipulating JSON response with Python so any comments are welcome. If someone sees a better way to do this he is welcome to share.

Answer 1

I'm going to ignore the actual error that got you here, and instead propose a radical restructure: I think your code will be simpler and easier to reason about.

I've looked at the JSON returned from that Pokemon API and I can see why you started down the path you did: there's a lot of data, and you only need a small subset of it. So, you're going through a lot of effort to pick out exactly what you want.

The DictWriter interface can really help you here. Consider this really small example:

header = ['name', 'id', 'order']
with open('output.csv', 'w', newline='') as f:
    writer = csv.DictWriter(f, fieldnames=header)
    writer.writeheader()

    writer.writerow({'name': 'bulbasaur', 'id': 1, 'order': 1, 'species': {}})

Maybe you've run something like this before and got this error:

ValueError: dict contains fields not in fieldnames: 'species'

because the JSON you loaded has keys you didn't include when you created your writer... because you don't want them. And then, maybe you figured, "well, that means I've got to be very selective about what I put in the dict before passing to writerow()?

Since you've already defined which keys you care about for the header, use those keys to pull out what you want from the JSON:

header = ['name', 'id', 'order', 'height', 'weight',
          'speed', 'special-defense', 'special-attack', 
          'defense', 'attack', 'hp']

all_data = json.load(open('1.json'))  # bulbasaur, I downloaded this from the API URL

my_data = {}
for key in header:
    my_data[key] = all_data.get(key)  # will return None for sub-stats keys, which is okay for now

writer = csv.DictWriter(sys.stdout, fieldnames=header)
writer.writeheader()
writer.writerow(my_data)

The get(key_name) method on a dict (the JSON data) will try to find that key in the dict and return that key's value. If the key isn't found, None is returned. Running that I get the following CSV (the sub-stat columns are empty, as expected):

name,id,order,height,weight,speed,special_defense,special_attack,defense,attack,hp
bulbasaur,1,1,7,69,,,,,,

This has the same effect as your "if this key, then this value" statements, but it's driven by the data (header names) you already defined.

On to the sub-stats...

I think it's safe to assume that if there is a stats key in the JSON, each "stat object" in the list of stats will have the data you want. It's important to make sure you're only copying the stats you've specified in header; and again, you can use your data to drive the process:

for stat in all_data['stats']:
    stat_name = stat['stat']['name']
    if stat_name not in header:
        continue  # skip this sub-stat, no column for it in the CSV

    base_stat = stat['base_stat']
    my_data[stat_name] = base_stat

When I insert that loop, I now get this for my CSV output:

name,id,order,height,weight,speed,special_defense,special_attack,defense,attack,hp
bulbasaur,1,1,7,69,45,,,49,49,45

Some stats are populated, but some, the "special" stats are blank? That's because in your header you've named them like special_attack (with underscore) but in reality they're like special-attack (with hyphen). I fixed your header, and now I get:

name,id,order,height,weight,speed,special-defense,special-attack,defense,attack,hp
bulbasaur,1,1,7,69,45,65,65,49,49,45

Those are all the pieces you need. To put it together, I recommend the following structure... I'm a big fan of breaking up a process like this into distinct tasks: get all the data, then process all the data, then write all the processed data. It makes debugging easier, and less indentation of code:

# Make all API calls and record their JSON
all_datas = []
# loop over your API calls:
    # make the request
    # get the JSON data
    # append JSON data to all_datas


# Process/transform the API JSON into what you want
my_data_rows = []
for all_data in all_datas:
    my_data_row = {}

    for key in header:
        my_data_row[key] = all_data.get(key)
    
    for stat in all_data['stats']:
        stat_name = stat['stat']['name']
        if stat_name not in header:
            continue  # skip this sub-stat

        base_stat = stat['base_stat']
        my_data[stat_name] = base_stat


# Write your transformed data to CSV
writer = csv.DictWriter(sys.stdout, fieldnames=header)
writer.writeheader()
writer.writerows(my_data_rows)

Answer 2

Check the example for using the DictWriter. You need to pass a dictionary to writerow instead of a list, so your last line should be

data =['name':name,'id': id,'order':order,'height': height,'weight':weight,'speed':speed,'special_defense':special_defense,'special_attack':special_attack,'defense':defense,'attack':attack,'hp':hp]
 dw.writerow(data)

Note that your whole code can also be simplified if you populate the data dictionary instead of all your if/else:

data={} #empty dictionary

#First extract everything that is on the main level of your dict
for key in ("name", "id", "order", "height", "weight":
    if key in data_json:
        data[key]=data_json[key]
        
#Check if the "stats" dict exists in your JSON data
if 'stats' in data_json:    
    if 'base_stat' in data_json['stats']:
       data['base_stat']=data_json['stats']['base_stat']
    if 'stat' in data_json['stats']:
        statDict = data_json['stats']['stat']
        for key in ['hp', 'attack', 'defense', 'special-attack', 'special-defense', 'speed']:
            if key in statDict:
                data[key]=statDict[key]

Notes:

1. I did not test this code, check it carefully, but I hope you get the idea
2. You could add else to all if key in checks to include an error message if a stat is missing
3. If you are sure that all keys will always be present, then you can skip a few of the if checks

Answer 3

Since dw is a DictionaryWriter, data needs to be a dictionary (currently it's a list) as seen in the documentation.

Convert data to a dictionary with your headers

data = [name,id,order,height,weight,speed,special_defense,special_attack,defense,attack,hp]
data = dict(zip(headerList, data))
dw.writerow(data)