CODEWITHSUNDEEP
pythonpandascsv
Malkovitch John
Malkovitch John

Reputation: 41

Python: append dict to existing csv

I want to append dictionary data to an existing csv, without writing again headers.

I tried every thing from this issue: append new row to old csv file python

I tried pandas and csv solutions.

   try:
       if os.path.isfile(csv_filepath) is False:
          df = pandas.DataFrame.from_dict(dict_data, orient="index")
          df.to_csv(csv_filepath)
       else:
           list_values = []
           list_keys = []
           for key, value in dict_data.items():
               list_values.append(value)
               list_keys.append(key)
           with open(csv_filepath, 'a', newline='') as fd:
               writer = csv.DictWriter(fd, fieldnames=list_keys)
               writer.writerow(dict_data)
   except Exception as e:
       raise e

I also tried

with open(csv_filtepath, 'a') as f:
    writer = csv.writer(f)
    writer.writerow(dict_data)

but both erase and write the new data without appending.

I also tried to to load the both csv in dataframes then append the second to the first but it added headers two times

EDIT:

For exemple, I first create a csv with: {'toto': 1, 'tata': 2, 'titi': 3}

Then I want to append the created csv {'toto': 2, 'tata': 1, 'titi': 5}

After this operation, I want to have 1 and 2 for in toto column, 2 and 1 in the tata column and 3 and 5 in the titi column.

EDIT 2

I tried this:

 df1 = pd.read_csv(csv_filepath)
 df2 = pd.DataFrame(dict_data)
 df3 = pd.concat([df1, df2], axis=0)
 df3.to_csv(csv_filepath)

but I have an error message : 'Error tokenizing data. C error: Expected 2 fields in line 20, saw 9'

EDIT 3

list_keys = []
for key, value in dict_data.items():
    list_keys.append(key)
with open(csv_filepath, 'a', newline='') as fd:
    writer = csv.DictWriter(fd, fieldnames=list_keys)
    writer.writerow([data['toto'], data['tata'], data['titi']])

And I have this error message: 'list' object has no attribute 'keys'

I want to add to an existing csv file, data from a dictionary with same keys.

Upvotes: 1

Views: 5452

Answers (2)

9000
9000

Reputation: 40884

The open(csv_filtepath, 'a') approach works; in fact, it just worked for me without truncation.

Note that you create a writer, not DictWriter; the former accepts lists, not dicts.

But if you're appending to an existing file, you have to follow its order of columns explicitly anyway, so a DictWriter won't work. You will have to do something like below, assuming that it's the column order of the file you're appending to.

import csv

with open(csv_file, 'a') as f:
  writer = csv.writer(f)  # Note: writes lists, not dicts.
  for data in rows_to_append:  # Maybe your df, or whatever iterable.
    writer.writerow([data['toto'], data['tata'], data['titi']])

Knowing the column order may be tricky, too. If you're not sure about it, you can e.g. read only the first line of the (presumably huge) first CSV file if that line contains column names. You can use the normal CSV reader for it:

with open(csv_file) as f:
  reader = csv.reader(f)
  column_names = next(reader)
# Here the file is closed after reading just one line.

Upvotes: 3

geekzeus
geekzeus

Reputation: 895

df1 = pd.DataFrame({'toto': [1], 'tata': [2], 'titi': [3]})
df2 = pd.DataFrame({'toto': [2], 'tata': [1], 'titi': [5]})

DF = pd.concat([df1, df2], axis=0)

Is this what you are looking for

EDIT: Memory Efficient snippet for your example

filepaths = ['pathtocsvfolder/df1.csv','pathtocsvfolder/df2.csv']
only_head = True

for i in filepaths:
    df = pd.read_csv(i)
    if only_head:
        df.to_csv('final.csv', index=False)
        only_head = False
    else:
        df.to_csv('final.csv', index=False, header=False, mode='a')

Upvotes: 0

Related Questions