CODEWITHSUNDEEP
python
Joshua Ellingson
Joshua Ellingson

Reputation: 21

How would I remove all duplicates in Python List?

For example:

newList = [1, 1, 2, 2, 2, 2, 3, 4, 4, 4, 4, 5, 5]

I need the list to be:

newList = [3]

Another example:

tempList = [["cheese", "ham"], ["spinach", "dressing"], ["cheese", "ham"]]

The list would be:

tempList = [["spinach", "dressing"]]

It can't use anything that won't work on unhashable items - the real list will have list stored in it. I also will have over 100,000 objects so speed is the biggest factor. I don't need the list to be in any sort of order.

Here is what I have come up with so far:

newList.sort()

i = 0
while i < newList.__len__()-1:
    x = 1
    while newList[i].__eq__(newList[i + x]):
        x += 1
        if not (i + x <= newList.__len__()- 1):
            break
    if x != 1:
        for j in range(i, i + x):
            del newList[i]
        i-=1
    i+=1

This works but I'm wondering if there is amore efficient way.

Upvotes: 2

Views: 70

Answers (4)

kjmerf
kjmerf

Reputation: 4335

I would recommend generating a new list. A simple loop would work:

lst = [1, 1, 2, 2, 2, 2, 3, 4, 4, 4, 4, 5, 5]

new = []

for x in lst:
    if lst.count(x) == 1:
        new.append(x)

print(new)

#output:

[3]

Upvotes: 1

Ajax1234
Ajax1234

Reputation: 71451

You can try this:

def counter(newList):
   counter_list = {tuple(i) if not any(isinstance(i, b) for b in [int, str, float]) else i:newList.count(i) for i in newList}
   return [a for a, b in counter_list.items() if b == 1]

newList = [1, 1, 2, 2, 2, 2, 3, 4, 4, 4, 4, 5, 5]
print(counter(newList))

Output:

[3]

When newList contains lists:

tempList = [["cheese", "ham"], ["spinach", "dressing"], ["cheese", "ham"]]
print(counter(tempList))

Output:

[('spinach', 'dressing')]

Upvotes: 0

Chetan Kabra
Chetan Kabra

Reputation: 361

Try this :

tempList = [["cheese", "ham"], ["spinach", "dressing"], ["cheese", "ham"]]

Below function will return dictionary having value = 1.

 from collections import Counter
    dict = {}
    for j in newList:
        for i in Counter(j):
            if i in dict:
                del dict[i]
            else:
                dict.update({i:1})
    print dict

Other List = [1, 1, 2, 2, 2, 2, 3, 4, 4, 4, 4, 5, 5]

Simply remove first for loop

Upvotes: 0

Patrick Haugh
Patrick Haugh

Reputation: 60944

You could convert everything to it's repr repersentation, find the unique elements using a Counter and then use ast.literal_eval to transform everything back into objects. This will work for stuff like [1, [1, 23], [1, 23]], but not for everything. 

from collections import Counter
from ast import literal_eval

def unique_elements(iterable):
    c = Counter(repr(e) for e in iterable)
    return [literal_eval(k) for k, v in c.items() if v == 1]

Upvotes: 0

Related Questions