CODEWITHSUNDEEP
pythonpandas
Marcosffn
Marcosffn

Reputation: 23

How to read various csv files into a dataframe with a for loop and append it all together

I am want to open and read many csv files at once, open each one as a DataFrame then put them all together in a single dataframe. All csv/DataFrames have the same numbers of columns. What I tried to do was this:

import os
import pandas as pd

df = pd.DataFrame()

paths = "C:\\Users\\Host02\\Documents\\analise2\\archives\\ms"

arr = os.scandir(paths)

for file in arr:
    df2 = pd.read_csv(file.path, sep=";", header=8)
    df.append(df2)

The problem is, in the end the first Dataframe (df) stays empty, seems like append has no effect.

Edit:I have solved my problem that way. I appreciate all the asnwers.

import pandas as pd
import os

arquivos_path = os.scandir("/home/marcos/Python/pesquisa/arquivos/ms/bissexto")

j = pd.DataFrame()

for arquivo in arquivos_path:
    df = pd.read_csv(arquivo.path, skiprows=8, sep=";")
    j = j.append(df)

j.to_csv("/home/marcos/Python/pesquisa/arquivos/ms_novo/bissexto/teste.csv", index=False)

Upvotes: 2

Views: 1609

Answers (2)

Youyoun
Youyoun

Reputation: 338

Load each file in a dataframe and add them to a list. Then use pd.concat to merge all those dataframes together.

Here's an example using the same code structure as yours:

import os
import pandas as pd

df = []

paths = "C:\\Users\\Host02\\Documents\\analise2\\archives\\ms"

arr = os.scandir(paths)

for file in arr:
    df2 = pd.read_csv(file.path, sep=";", header=8)
    df.append(df2)
df = pd.concat(df)

Upvotes: 1

Alex S
Alex S

Reputation: 239

If your csv files are the same width you should be able to run the following:

paths = "C:\\Users\\Host02\\Documents\\analise2\\archives\\ms"
arr = os.scandir(paths)

for i,file in enumerate(arr):
    load_df = pd.read_csv(file.path, sep=";", header=8)
    if i==0:
        df=load_df
    if i>0:
        df=pd.concat([df, load_df])

Upvotes: 0

Related Questions