CODEWITHSUNDEEP
pythonpandascsvglob
PV8
PV8

Reputation: 6260

Load multiple csv files from a folder with a condition

I have a very unstructured folder, where a lot of files have no entries (just the row headers), but there is no data inside. I know that i can include them and they will not change anything, but the problem is that the headers are not the same everywhere, so every file includes some extra manual work for me.

Until now I now how to load all files in a specific folder with the following code:

import glob

path = r'C:/Users/...'
all_files = glob.glob(path+ "/*.csv")

li = []

for filename in all_files:
    frame = pd.read_csv(filename, index_col=None, header=0, sep=';', encoding='utf-8', low_memory=False)
    li.append(frame)

df = pd.concat(li, axis=0, ignore_index=True, sort=False)

How can I skip every file, which only has one row?

Upvotes: 0

Views: 368

Answers (1)

U13-Forward
U13-Forward

Reputation: 71580

Modify this loop from:

for filename in all_files:
    frame = pd.read_csv(filename, index_col=None, header=0, sep=';', encoding='utf-8', low_memory=False)
    li.append(frame)

To:

for filename in all_files:
    frame = pd.read_csv(filename, index_col=None, header=0, sep=';', encoding='utf-8', low_memory=False)
    if len(frame) > 1:
        li.append(frame)

That's what if statements are for.

Upvotes: 1

Related Questions