CODEWITHSUNDEEP
pythonlistduplicatesiteration
Alex
Alex

Reputation: 75

Removing duplicate records from a Python list

I am obtaining data from a URL and doing some processing on that data. I have now come to a point where I need to eliminate duplicates. Below is the output of my program. 

{u'Price': 133.84, u'@Number': u'1', u'Mw': 10} 
{u'Price': 139.09, u'@Number': u'2', u'Mw': 15} 
{u'Price': 144.34, u'@Number': u'3', u'Mw': 10} 
{u'Price': 148.53, u'@Number': u'4', u'Mw': 10} 
{u'Price': 152.52, u'@Number': u'5', u'Mw': 9} 
{u'Price': 133.84, u'@Number': u'1', u'Mw': 10} 
{u'Price': 139.09, u'@Number': u'2', u'Mw': 15} 
{u'Price': 144.34, u'@Number': u'3', u'Mw': 10} 
{u'Price': 148.53, u'@Number': u'4', u'Mw': 10} 
{u'Price': 152.52, u'@Number': u'5', u'Mw': 9} 
{u'Price': 133.84, u'@Number': u'1', u'Mw': 10} 
{u'Price': 139.09, u'@Number': u'2', u'Mw': 15} 
{u'Price': 144.34, u'@Number': u'3', u'Mw': 10} 
{u'Price': 148.53, u'@Number': u'4', u'Mw': 10} 
{u'Price': 152.52, u'@Number': u'5', u'Mw': 9}

I want this to be my output:

{u'Price': 133.84, u'@Number': u'1', u'Mw': 10} 
{u'Price': 139.09, u'@Number': u'2', u'Mw': 15} 
{u'Price': 144.34, u'@Number': u'3', u'Mw': 10} 
{u'Price': 148.53, u'@Number': u'4', u'Mw': 10} 
{u'Price': 152.52, u'@Number': u'5', u'Mw': 9}

The @Number can be between 1-7. What is the best way to approach this problem? Do I need to get the max of @Number and then print as many lines as my max value?

Upvotes: 2

Views: 139

Answers (6)

Padraic Cunningham
Padraic Cunningham

Reputation: 180481

use an OrderedDict to maintain order and the u'@Number' as the key only adding if the u'@Number' is not already in out dict. 

l=[{u'Price': 133.84, u'@Number': u'1', u'Mw': 10},
{u'Price': 139.09, u'@Number': u'2', u'Mw': 15},
{u'Price': 144.34, u'@Number': u'3', u'Mw': 10},
{u'Price': 148.53, u'@Number': u'4', u'Mw': 10},
{u'Price': 152.52, u'@Number': u'5', u'Mw': 9},
{u'Price': 133.84, u'@Number': u'1', u'Mw': 10},
{u'Price': 139.09, u'@Number': u'2', u'Mw': 15},
{u'Price': 144.34, u'@Number': u'3', u'Mw': 10},
{u'Price': 148.53, u'@Number': u'4', u'Mw': 10},
{u'Price': 152.52, u'@Number': u'5', u'Mw': 9},
{u'Price': 133.84, u'@Number': u'1', u'Mw': 10},
{u'Price': 139.09, u'@Number': u'2', u'Mw': 15},
{u'Price': 144.34, u'@Number': u'3', u'Mw': 10},
{u'Price': 148.53, u'@Number': u'4', u'Mw': 10},
{u'Price': 152.52, u'@Number': u'5', u'Mw': 9}
]

from collections import OrderedDict
od = OrderedDict()
for d in l:
    num =  d["@Number"]
    if num not in od:
        od[num] = d

print(list(od.values())

[{u'@Number': u'1', u'Mw': 10, u'Price': 133.84},
{u'@Number': u'2', u'Mw': 15, u'Price': 139.09},
{u'@Number': u'3', u'Mw': 10, u'Price': 144.34},
{u'@Number': u'4', u'Mw': 10, u'Price': 148.53},
{u'@Number': u'5', u'Mw': 9, u'Price': 152.52}]

If the order is always guaranteed as in your example you can simply break when you find a repeated "@Number":

for d in l:
    num = d["@Number"]
    if num not in od:
        od[num] = d
    else:
         break

If you want to use max:

from itertools import islice
from operator import itemgetter

# for @Number" > 9 use lambda
#  mx = int(max(l, key=lambda x: int(x["@Number"]))["@Number"])
mx = int(max(l, key=itemgetter("@Number"))["@Number"])
print(list(islice(l,None,mx)))

Upvotes: 2

dawg
dawg

Reputation: 104032

You can maintain a set for the one you have already seen:

seen=set()
data=[]
for e in input:
    if e["@Number"] not in seen:
        seen.add(e["@Number"])
        data.append(e)

>>> data
[{u'Price': 133.84, u'@Number': u'1', u'Mw': 10}, {u'Price': 139.09, u'@Number': u'2', u'Mw': 15}, {u'Price': 144.34, u'@Number': u'3', u'Mw': 10}, {u'Price': 148.53, u'@Number': u'4', u'Mw': 10}, {u'Price': 152.52, u'@Number': u'5', u'Mw': 9}]

If your data is alway in order and your lines will be complete once you print the max, you could do:

>>> input[0:max(int(e) for e in (d['@Number'] for d in input))]
[{u'Price': 133.84, u'@Number': u'1', u'Mw': 10}, {u'Price': 139.09, u'@Number': u'2', u'Mw': 15}, {u'Price': 144.34, u'@Number': u'3', u'Mw': 10}, {u'Price': 148.53, u'@Number': u'4', u'Mw': 10}, {u'Price': 152.52, u'@Number': u'5', u'Mw': 9}]

Upvotes: 3

alexis
alexis

Reputation: 50220

If your data really looks as you describe, and you want to rule out identical triples, the obvious solution would be to convert them to a set. Since order is important, you could use collections.OrderedDict instead with a dummy value True.

You'll first need to transform your data into tuples, since dictionaries cannot be used as dict keys:

mytuples = [ tuple(x["Price"], x["@Number"], x["Mw"]) for x in mydata

(or just build your data as tuples to begin with).

Then:

from collections import OrderedDict
unique = OrderedDict((tup, True) for tup in mytuples)

You can now use unique.keys() to retrieve your triplets in the order they were seen.

Upvotes: 0

Marcin
Marcin

Reputation: 238587

Another possibility, using OrderedDict:

from collections import OrderedDict

l = [{u'Price': 133.84, u'@Number': u'1', u'Mw': 10}, 
{u'Price': 139.09, u'@Number': u'2', u'Mw': 15} ,
{u'Price': 144.34, u'@Number': u'3', u'Mw': 10} ,
{u'Price': 148.53, u'@Number': u'4', u'Mw': 10} ,
{u'Price': 152.52, u'@Number': u'5', u'Mw': 9} ,
{u'Price': 133.84, u'@Number': u'1', u'Mw': 10} ,
{u'Price': 139.09, u'@Number': u'2', u'Mw': 15} ,
{u'Price': 144.34, u'@Number': u'3', u'Mw': 10} ,
{u'Price': 148.53, u'@Number': u'4', u'Mw': 10} ,
{u'Price': 152.52, u'@Number': u'5', u'Mw': 9} ,
{u'Price': 133.84, u'@Number': u'1', u'Mw': 10} ,
{u'Price': 139.09, u'@Number': u'2', u'Mw': 15} ,
{u'Price': 144.34, u'@Number': u'3', u'Mw': 10} ,
{u'Price': 148.53, u'@Number': u'4', u'Mw': 10} ,
{u'Price': 152.52, u'@Number': u'5', u'Mw': 9}]

od = OrderedDict()

for d in l:
    od[d["@Number"]] = d


print(od)

Results is:

OrderedDict([('1', {'@Number': '1', 'Price': 133.84, 'Mw': 10}), ('2', {'@Number': '2', 'Price': 139.09, 'Mw': 15}), ('3', {'@Number': '3', 'Price': 144.34, 'Mw': 10}), ('4', {'@Number': '4', 'Price': 148.53, 'Mw': 10}), ('5', {'@Number': '5', 'Price': 152.52, 'Mw': 9})])

In the dict, @Number is used as a key. And because it is an OrderedDict, the order of Numbers in the original list preserved.

Upvotes: 1

Robᵩ
Robᵩ

Reputation: 168716

This program does what you ask:

data = [
{u'Price': 133.84, u'@Number': u'1', u'Mw': 10} ,
{u'Price': 139.09, u'@Number': u'2', u'Mw': 15} ,
{u'Price': 144.34, u'@Number': u'3', u'Mw': 10} ,
{u'Price': 148.53, u'@Number': u'4', u'Mw': 10} ,
{u'Price': 152.52, u'@Number': u'5', u'Mw': 9} ,
{u'Price': 133.84, u'@Number': u'1', u'Mw': 10} ,
{u'Price': 139.09, u'@Number': u'2', u'Mw': 15} ,
{u'Price': 144.34, u'@Number': u'3', u'Mw': 10} ,
{u'Price': 148.53, u'@Number': u'4', u'Mw': 10} ,
{u'Price': 152.52, u'@Number': u'5', u'Mw': 9} ,
{u'Price': 133.84, u'@Number': u'1', u'Mw': 10} ,
{u'Price': 139.09, u'@Number': u'2', u'Mw': 15} ,
{u'Price': 144.34, u'@Number': u'3', u'Mw': 10} ,
{u'Price': 148.53, u'@Number': u'4', u'Mw': 10} ,
{u'Price': 152.52, u'@Number': u'5', u'Mw': 9},
]

result = []
for item in data:
    if item not in result:
        result.append(item)
for item in result:
    print item

Output:

{u'Price': 133.84, u'@Number': u'1', u'Mw': 10}
{u'Price': 139.09, u'@Number': u'2', u'Mw': 15}
{u'Price': 144.34, u'@Number': u'3', u'Mw': 10}
{u'Price': 148.53, u'@Number': u'4', u'Mw': 10}
{u'Price': 152.52, u'@Number': u'5', u'Mw': 9}

Upvotes: 1

avinash pandey
avinash pandey

Reputation: 1381

You can make a dictionary with Price @Number and Mw as keys.Then check if the new item is already in dict or not.

def combine(L):
    results = {}
    for item in L:
        key = (item["Price"], item["@Number"],item["Mw"])
        if key not in results:  # combine them
            results[key] = item
    return results.values()

The output

{u'Price': 148.53, u'@Number': u'4', u'Mw': 10}
{u'Price': 139.09, u'@Number': u'2', u'Mw': 15}
{u'Price': 152.52, u'@Number': u'5', u'Mw': 9}
{u'Price': 144.34, u'@Number': u'3', u'Mw': 10}
{u'Price': 133.84, u'@Number': u'1', u'Mw': 10}

Upvotes: 0

Related Questions