CODEWITHSUNDEEP
pythonlistduplicates
Noob Coder
Noob Coder

Reputation: 949

Eliminate Duplicates of a List (Python)

I have a list of lists and need to eliminate all lists that have the same 3rd element, and save 1.

For example:

x=[[1,2,3],[1,3,3],[5,6,3],[2,4,6],[8,5,9],[10,5,9]]

could turn into:

x=[[1,2,3],[2,4,6],[8,5,9]]

I've attempted a solution that eliminates all lists that have the same 2nd index using lambda, but I don't know how to save one of each element like set() does--I just get:

x=[[2,4,6]]

Upvotes: 3

Views: 377

Answers (6)

abarnert
abarnert

Reputation: 366073

The itertools docs come with a nice set of recipes, one of which, unique_everseen does exactly what you want: "List unique elements, preserving order. Remember all elements ever seen."

You can copy it paste it from the recipes, or you can pip install more-itertools and just use it as more_itertools.unique_everseen.

Or you can look at the code, and see that it's essentially just a slightly more flexible and slightly optimized version of exactly what Raymond Hettinger's answer does.*

And that's exactly what the recipes are there for—when you want to do something to an iterable, there's often a recipe for it, which not only does what you want, but is so simple that if you take a couple minutes to work out how it works, you'll know how to do it yourself in the future.

* This shouldn't surprise anyone who knows anything about Python, or just looks at the source to itertools…

Upvotes: 2

John La Rooy
John La Rooy

Reputation: 304443

You can use set if you define a helper class

>>> L = [[1,2,3],[1,3,3],[5,6,3],[2,4,6],[8,5,9],[10,5,9]]
>>> class Element(list):__hash__=lambda s:hash(s[2]);__eq__=lambda s,o:s[2]==o[2]
... 
>>> set(map(Element, L))
set([[8, 5, 9], [1, 2, 3], [2, 4, 6]])

Upvotes: 3

John La Rooy
John La Rooy

Reputation: 304443

>>> L = [[1,2,3],[1,3,3],[5,6,3],[2,4,6],[8,5,9],[10,5,9]]
>>> s = set(i[2] for i in L) 
>>> [s.remove(i[2]) or i for i in L if i[2] in s]
[[1, 2, 3], [2, 4, 6], [8, 5, 9]]

Upvotes: 1

Jon Clements
Jon Clements

Reputation: 142226

You could also use a dict here, which could be a bit more flexible later:

data =[[1,2,3],[1,3,3],[5,6,3],[2,4,6],[8,5,9],[10,5,9]]
one_of = {el[2]:el for el in data}.values()
# [[10, 5, 9], [5, 6, 3], [2, 4, 6]]

Upvotes: 4

roman
roman

Reputation: 117540

You can do this with list comprehension and helper set:

>>> t = set()
>>> y = [l for l in x if l[2] not in t and not t.add(l[2])]
>>> y
[[1, 2, 3], [2, 4, 6], [8, 5, 9]]

Upvotes: 2

Raymond Hettinger
Raymond Hettinger

Reputation: 226664

Python sets should still work for you:

>>> seen = set()
>>> result = []
>>> for s in x:
    e = s[2]
    if e not in seen:
        result.append(s)
        seen.add(e)


>>> result
[[1, 2, 3], [2, 4, 6], [8, 5, 9]]

Upvotes: 8

Related Questions