CODEWITHSUNDEEP
pythonpandasduplicates
Kimberly Clemons
Kimberly Clemons

Reputation: 55

Remove duplicates python/pandas

I have data like this:

d = {'col1': ['1-2-3', '2-2-3'], 'col2': ['1-2-2-3', '4-4-4']}
df = pd.DataFrame(data=d)
df

    col1    col2
0   1-2-3   1-2-2-3
1   2-2-3   4-4-4

I need to remove the duplicates and preserve the order to achieve this:

    col1    col2
0   1-2-3   1-2-3
1   2-3     4

Upvotes: 0

Views: 57

Answers (2)

sam
sam

Reputation: 2311

drop_duplicates is the generic API you can use. But in this special case, you can create a temporary column, put all of the numbers as a list. And then again convert that in the non-duplicated string

df = df.apply(lambda x: x.str.split('-'), axis=1) 
df['col1'] = df['col1'].apply(lambda x: '-'.join((list(dict.fromkeys(x)))) )
df['col2'] = df['col2'].apply(lambda x: '-'.join((list(dict.fromkeys(x)))) )
df

Upvotes: 1

jezrael
jezrael

Reputation: 862741

Use DataFrame.applymap...

if order is important use split with dict.fromkeys for remove duplicates, last join back:

df = df.applymap(lambda x: '-'.join(dict.fromkeys(x.split('-')).keys()))
print (df)
    col1   col2
0  1-2-3  1-2-3
1    2-3      4

if order is not important use sets:

df = df.applymap(lambda x: '-'.join(set(x.split('-'))))
print (df)
    col1   col2
0  3-2-1  3-2-1
1    3-2      4

Upvotes: 1

Related Questions