CODEWITHSUNDEEP
pythonlistitems
alexvassel
alexvassel

Reputation: 10740

Python, remove duplicates from list of tuples

I have the following list:

[('mail', 167, datetime.datetime(2010, 9, 29)) , 
 ('name', 1317, datetime.datetime(2011, 12, 12)), 
 ('mail', 1045, datetime.datetime(2010, 8, 13)), 
 ('name', 3, datetime.datetime(2011, 11, 3))]

And I want to remove items from the list with coinciding first item in a tuple where date is not the latest. In other words I need to get this:

[('mail', 167, datetime.datetime(2010, 9, 29)) , 
 ('name', 1317, datetime.datetime(2011, 12, 12))]

Upvotes: 4

Views: 9165

Answers (5)

Shawn Chin
Shawn Chin

Reputation: 86864

The following approach uses a dictionary to overwrite entries with the same key. Since the list is sorted by the date, older entries get overwritten by newer ones.

temp = {}
for v in sorted(L, key=lambda L: L[2]): # where L is your list
    temp[v[0]] = v
result = temp.values()

Or, for something a lot more compact (but much less readable):

result = dict((v[0],v) for v in sorted(L, key=lambda L: L[2])).values()

Update

This method would be reasonably quick if the list is already (or mostly) sorted by date. If it isn't, and especially if it is a large list, then this may not be the best approach.

For unsorted lists, you will likely get a some performance improvement by sorting by the key first, then the date. i.e. sorted(L, key=lambda L: (L[0],L[2])).

Or, better yet, go for Space_C0wb0y's answer.

Upvotes: 2

Björn Pollex
Björn Pollex

Reputation: 76788

You can use a dictionary to store the highest value found for a given key so far:

temp = {}
for key, number, date in input_list:
    if key not in temp: # we see this key for the first time
        temp[key] = (key, number, date)
    else:
        if temp[key][2] < date: # the new date is larger than the old one
            temp[key] = (key, number, date)
result = temp.values()

Upvotes: 16

utdemir
utdemir

Reputation: 27216

You can do it via sorting the list and getting the highest values by d[2]:

    In [26]: d
    Out[26]: 
    [('mail', 167, datetime.datetime(2010, 9, 29, 0, 0)),
     ('name', 1317, datetime.datetime(2011, 12, 12, 0, 0)),
     ('mail', 1045, datetime.datetime(2010, 8, 13, 0, 0)),
     ('name', 3, datetime.datetime(2011, 11, 3, 0, 0))]

    In [27]: d.sort(key = lambda i: i[2], reverse=True)

    In [28]: d
    Out[28]: 
    [('name', 1317, datetime.datetime(2011, 12, 12, 0, 0)),
     ('name', 3, datetime.datetime(2011, 11, 3, 0, 0)),
     ('mail', 167, datetime.datetime(2010, 9, 29, 0, 0)),
     ('mail', 1045, datetime.datetime(2010, 8, 13, 0, 0))]

    In [29]: [i for pos, i in enumerate(d) if i[0] in [j[0] for j in d[pos+1:]]]
    Out[29]: 
    [('name', 1317, datetime.datetime(2011, 12, 12, 0, 0)),
     ('mail', 167, datetime.datetime(2010, 9, 29, 0, 0))]

Upvotes: 0

Keith
Keith

Reputation: 43024

Here you go.

#!/usr/bin/python2

from pprint import pprint
import datetime

ol = [('mail', 167, datetime.datetime(2010, 9, 29)) , 
     ('name', 1317, datetime.datetime(2011, 12, 12)), 
     ('mail', 1045, datetime.datetime(2010, 8, 13)), 
     ('name', 3, datetime.datetime(2011, 11, 3))]

d = {}

for t in sorted(ol, key=lambda t: (t[0], t[2])):
    d[t[0]] = t
out = d.values()

pprint(out)

That sorts the list using the first and third tuple elements as keys, then removes duplicates by using a hash table.

Upvotes: -1

Patrick
Patrick

Reputation: 1148

d = {}

for item in list:
    if (item[0], item[1]) not in d:
        d[(item[0], item[1])] = item[2]
    else:
        if item[2] > d[(item[0], item[1])]:
            d[(item[0], item[1])] = item[2]

item = [(x[0], x[1], d[x] for x in d.keys()]

Upvotes: 0

Related Questions