CODEWITHSUNDEEP
pythoncsvfield
DLo9
DLo9

Reputation: 31

Python CSV writer - writing columns in new csv file up to maximum number of fields in csv files

I have 200 CSV files in my folder. What I am trying to do is read first row of each files and write in new csv. And on top, I want to write [file,field1,field2,...fieldn] n is maximum number of fields.

import csv
import glob 
list=[]
hel=[]
files=glob.glob('C:/dataset/*.csv')
with open('test.csv', 'w',newline='') as testfile:
    csv_writer = csv.writer(testfile)
    for file in files:
        with open(file, 'r') as infile:
            file=file[file.rfind('\\')+1:]
            file=file.strip('.csv')
            reader = csv.reader(infile)
            headers = next(reader)
            hel.append((len(headers)))
            max(hel)
            lst = [file] + headers
            csv_writer.writerow(lst)

It came out that maximum number of fields of 200 files are 255. So on top of new csv file, I want to write file, field1, field2 ... field 255. How can I do this?

import csv
import glob 
list=[]
hel=[]
files=glob.glob('C:/dataset/*.csv')
with open('test.csv', 'w',newline='') as testfile:
    csv_writer = csv.writer(testfile)
    for file in files:
        with open(file, 'r') as infile:
            file=file[file.rfind('\\')+1:]
            file=file.strip('.csv')
            reader = csv.reader(infile)
            headers = next(reader)
            hel.append((len(headers)))
            b=['field{}'.format(i) for i in range(1,max(hel)+1)]
            lst = [file] + headers
            csv_writer.writerow(lst)

Now b is list that looks like this ['field1','field2'...'field255'] I need to insert 'file' before 'field1' and write that row on the top of new csv file. Writing code after csv_writer.writerow(lst) gives me csv file with 'field1','field2'.. every other line. How can I fix this problem

Upvotes: 0

Views: 1158

Answers (2)

frogcoder
frogcoder

Reputation: 1003

Read the field count and first line from each file before writing the file.

import glob
from itertools import chain
import os
from os.path import splitext, basename

def first_line(filepath):
    with open(filepath) as f:
        return next(f)


def write_test_file(dest_file_path, source_path_name):
    source_paths = glob.glob(source_path_name)
    first_lines = list(map(first_line, source_paths))

    max_count = max(l.count(",") for l in first_lines)
    field_names = map("field{}".format, range(1, max_count + 2))
    header = ",".join(chain(["file"], field_names)) + os.linesep

    file_names = (splitext(basename(p))[0] for p in source_paths)
    content = chain([header], map(",".join, zip(file_names, first_lines)))

    with open(dest_file_path, 'w') as testfile:
        testfile.write("".join(content))


write_test_file('test.csv', 'C:/dataset/*.csv')

Upvotes: 0

John Zwinck
John Zwinck

Reputation: 249303

You first need to read all your input files to determine the maximum number of fields is 255. Then you need to construct a list of field names to write into the output file (just once, not in a loop):

['field{}'.format(i) for i in range(1, 256)]

You can pass that list to the csv module to write it.

Upvotes: 0

Related Questions