CODEWITHSUNDEEP
pythonpandascsvheader
AC1
AC1

Reputation: 73

Searching for specific headers in csv file

I want to read a csv file in python and grab all the headers, and then search different csv files to check if they have any extra headers that the first csv file didn't have.

I currently have a program to merge csv files but I need to handle if one of the files has an extra (or multiple extra) headers.

import glob, os
import pandas as pd

file_list = glob.glob("*.csv")
pd_list = []

for file in file_list:
    pd_list.append(pd.read_csv(file))

combined = pd.concat(pd_list)
final = combined.drop_duplicates()

if not os.path.exists('output'):
    os.makedirs('output')

final.to_csv("output/final.csv", index=False)

Upvotes: 0

Views: 392

Answers (2)

Trenton McKinney
Trenton McKinney

Reputation: 62413

Using pandas accomplishes the same thing with less code:

Data:

csv1:

CustomerID,Gender,Day,SaleAmount
18,Male,Monday,71.55
24,Female,Monday,219.66
112,Male,Friday,150.44

csv2:

CustomerID,Gender,Day,SaleAmount
18,Male,Monday,71.55
24,Female,Monday,219.66
112,Male,Friday,150.44

csv3 - Note 2 extra columns, where 1 is also a duplicate:

CustomerID,Gender,SaleAmount,Day,random,Day
18,Male,71.55,Monday,test1,Monday
24,Female,219.66,Monday,test2,Wednesday
112,Male,150.44,Friday,test3,Friday

Use pathlib not os & glob:

code:

from pathlib import Path

p = Path.cwd() / 'csv_files'
f = p.rglob('*.csv')

df = pd.concat([pd.read_csv(x) for x in f], sort=False)

df.reset_index(inplace=True, drop=True)
df.to_csv('test.csv', index=False)

enter image description here

  • adds the extra columns and fills with NaN where it doesn't exist
  • the duplicate Day is Day.1

Upvotes: 0

Sinan Gok
Sinan Gok

Reputation: 189

If by 'headers' you mean a list of strings, then I believe you can use sets:

A = set(['red', 'blue', 'green', 'white'])
B = set(['red', 'blue', 'green', 'white', 'purple'])

if A.issubset(B):
    if B.difference(A):
        print('B has extra headers')
    else:
        print('B does not have extra headers')
else:
    print('B is missing headers')

'B has extra headers'

Upvotes: 1

Related Questions