CODEWITHSUNDEEP
pythoncsvduplicatesrows
abn
abn

Reputation: 1373

Compare all the CSV files in a folder and print duplicate rows

I have multiple CSV files in a folder, which I want to compare and print the matching rows (where the number of columns could be different). I know how to get duplicates within a file but this case is a little different. Let's say there are two files in a folder and I want to compare them. 

CSV1:
H1,H2,H4
C01,23,F
C2,45,M

CSV2:
H1,H2,H3,H4
C01,23,data,F
C01,23,some other data,M
C4,34,data,M

I need my output to check if all the available data (from the one with the least number of columns) matches exactly in another file in the same folder. My output could be like

CSV1,CSV2 (H1:C01,H2:23,H4:F(H3:data))

Upvotes: 0

Views: 475

Answers (1)

Alex Martelli
Alex Martelli

Reputation: 881685

What about something like:

def duplines(csv_least_cols, csv_most_cols):
    rowset = set()
    with open(csv_least_cols) as csv1:
        r = csv.reader(csv1)
        csv1_cols = next(r)
        for row in r:
            rowset.add(tuple(row))
    with open(csv_most_cols) as csv2:
        dr = csv.DictReader(csv2)
        for drow in dr:
            refcols = tuple(drow[c] for c in csv1_cols)
            if refcols in rowset: yield csv1_cols, refcols, drow

You can call this in a loop and perform whatever formatting you want -- this generator deals with the underlying logic, separating out the formatting task to its caller.

So for example to get your peculiar desired CSV1,CSV2 (H1:C01,H2:23,H4:F(H3:data)) style output you could have...:

def formatit(csv_least, csv_most):
    out_start = '{},{} ('.format(csv_least, csv_most)
    for c1cols, refvals, c2dict in duplines(csv_least, csv_most):
        out_middle = []
        for c, v in zip(c1cols, refvals):
            out_middle.append('{}:{}'.format(c, v))
        out_end = []
        for c in c2dict:
            if c in c1cols: continue
            out_end.append('{}:{}'.format(c, c2dict[c]))
        out = '{}{}({}))'.format(out_start, ','.join(out_middle), ','.join(out_end))
        print(out)

You'll notice that the formatting work is substantially more complex than the actual logic (and hence more likely to hide bugs:-) which is why I call your desired format "peculiar".

But I hope this can at least get you started (and you can try out each function separately, making sure the logic is as you desire it before worrying about the formatting:-).

Upvotes: 1

Related Questions