CODEWITHSUNDEEP
pythoncsvpandasdata-cleaning
madik_atma
madik_atma

Reputation: 799

How to read multiple csv files into pandas and output in one csv file

I want to import in path structured csv files and output as a one CSV. My code just works with a path and a manually typed csv file.

import csv
import pandas as pd
import numpy as np
import glob

cols = ['Date', 'Time', 'Duration', 'IP', 'Request']
pd.DataFrame(columns=cols).to_csv('out9.csv', index=False, sep=';')

for df in pd.read_csv('query.csv', sep='\s',  header=None, chunksize=6):
    df.reset_index(drop=True, inplace=True)
    df.fillna('', inplace=True)
    d = pd.DataFrame([df.loc[3,0], df.loc[3,1], ' '.join(df.loc[3,4:8]), ' '.join(df.loc[4,4:6]), ' '.join(df.loc[5,4:])])
    d.T.to_csv('out.csv', index=False, header=False, mode='a', sep=';')

I know there are some topics how to read many csv files, but in my case have not helped unfortunately.

I would like to read about it:

: C\Desktop\Files\*.csv

Information about the csv files: All are built the same, that is, no header, same structures. And I would like at a start of my code all in a folder read in and as a formatted again give out.

Therefore if possible to change the code as little as possible, I would only read several csv, instead of these a 'query.csv'

Thanks!

Upvotes: 2

Views: 1805

Answers (1)

jezrael
jezrael

Reputation: 863611

I think you can use glob:

import glob

cols = ['Date', 'Time', 'Duration', 'IP', 'Request']
pd.DataFrame(columns=cols).to_csv('out9.csv', index=False, sep=';')

for file in glob.glob('C:/Desktop/Files/*.csv'):
    for df in pd.read_csv(file, sep='\s',  header=None, chunksize=6):
        df.reset_index(drop=True, inplace=True)
        ...
        ...

Upvotes: 3

Related Questions