How to merge two lists? Preserving identical list elements for set manipulation

Question

I have been drawing Venn diagrams, coding loops and different sets(symmetrical_differences, unions, intersection, isdisjoint), enumerating by row numbers for the better part of a day or two trying to figure out how to implement this in code.

a = [1, 2, 2, 3] # <-------------|
b = [1, 2, 3, 3, 4] # <----------| Do not need to be in order.
result = [1, 2, 2, 3, 3, 4] # <--|

OR:

A = [1,'d','d',3,'x','y']
B = [1,'d',3,3,'z']
result =  [1,'d','d',3,3,'x','y','z']

edit:

Not trying to do a + b = [1, 1, 2, 2, 2, 3, 3, 3, 4]

Trying to do something like:

a - b = [2]

b - a = [3, 4]

a ∩ b = [1,2,3]

So

[a - b] + [b - a] + a ∩ b = [1, 2, 2, 3, 3, 4] ?

I am not sure here.

I have two spread sheets, each with several thousand lines. I want to compare both spreadsheets by a column type.

I have created lists from each column to compare/merge.

def returnLineList(fn):
    with open(fn,'r') as f:
        lines = f.readlines()
    line_list = []
    for line in lines:
        line = line.split('\t')
        line_list.append(line)
    return line_list

def returnHeaderIndexDictionary(titles):
    tmp_dict = {}
    for x in titles:
        tmp_dict.update({x:titles.index(x)})
    return tmp_dict

def returnColumn(index, l):
    column = []
    for row in l:
        column.append(row[index])
    return column

def enumList(column):
    tmp_list = []
    for row, item in enumerate(column):
        tmp_list.append([row,item])
    return tmp_list

def compareAndMergeEnumerated(L1,L2):
    less = []
    more = []
    same = []
    for row1,item1 in enumerate(L1):
        for row2,item2 in enumerate(L2):
            if item1 in item2:
                count1 = L1.count(item1)
                count2 = L2.count(item2)
                dif = count1 - count2
                if dif != 0:
                    if dif < 0:
                        less.append(["dif:"+str(dif),[item1,row1],[item2,row2]])
                    if dif > 0:
                        more.append(["dif:"+str(dif),[item1,row1],[item2,row2]])
                else:
                    same.append(["dif:"+str(dif),[item1,row1],[item2,row2]])
                break
    return less,more,same,len(less+more+same),len(L1),len(L2)

def main():
    unsorted_lines = returnLineList('unsorted.csv')
    manifested_lines = returnLineList('manifested.csv')

    indexU = returnHeaderIndexDictionary(unsorted_lines[0])
    indexM = returnHeaderIndexDictionary(manifested_lines[0])

    u_j_column = returnColumn(indexU['jnumber'],unsorted_lines)
    m_j_column = returnColumn(indexM['jnumber'],manifested_lines)

    print(compareAndMergeEnumerated(u_j_column,m_j_column))

if __name__ == '__main__':
    main()

update:

from collections import OrderedDict
A = [1,'d','d',3,'x','y']
B = [1,'d',3,3,'z']
M = A + B
R = [1,'d','d',3,3,'x','y','z']


ACount = {}
AL = lambda x: ACount.update({str(x):A.count(x)})
[AL(x) for x in A]

BCount = {}
BL = lambda x: BCount.update({str(x):B.count(x)})
[BL(x) for x in B]

MCount = {}
ML = lambda x: MCount.update({str(x):M.count(x)})
[ML(x) for x in M]


RCount = {}
RL = lambda x: RCount.update({str(x):R.count(x)})
[RL(x) for x in R]


print('^sym_difAB',set(A) ^ set(B)) # set(A).symmetric_difference(set(B))
print('^sym_difBA',set(B) ^ set(A)) # set(A).symmetric_difference(set(B))
print('|union    ',set(A) | set(B)) # set(A).union(set(B))
print('&intersect',set(A) & set(B)) # set(A).intersection(set(B))
print('-dif AB   ',set(A) - set(B)) # set(A).difference(set(B))
print('-dif BA   ',set(B) - set(A)) 
print('<=subsetAB',set(A) <= set(B)) # set(A).issubset(set(B))
print('<=subsetBA',set(B) <= set(A)) # set(B).issubset(set(A))
print('>=supsetAB',set(A) >= set(B)) # set(A).issuperset(set(B))
print('>=supsetBA',set(B) >= set(A)) # set(B).issuperset(set(A))

print(sorted(A + [x for x in (set(A) ^ set(B))]))
#[1, 3, 'd', 'd', 'x', 'x', 'y', 'y', 'z']

print(sorted(B + [x for x in (set(A) ^ set(B))]))
#[1, 3, 3, 'd', 'x', 'y', 'z', 'z']
cA = lambda y: A.count(y)
cB = lambda y: B.count(y)
cM = lambda y: M.count(y)
cR = lambda y: R.count(y)
print(sorted([[y,cA(y)] for y in (set(A) ^ set(B))]))
#[['x', 1], ['y', 1], ['z', 0]]

print(sorted([[y,cB(y)] for y in (set(A) ^ set(B))]))
#[['x', 0], ['y', 0], ['z', 1]]

print(sorted([[y,cA(y)] for y in A]))
print(sorted([[y,cB(y)] for y in B]))
print(sorted([[y,cM(y)] for y in M]))
print(sorted([[y,cR(y)] for y in R]))
#[[1, 1], [3, 1], ['d', 2], ['d', 2], ['x', 1], ['y', 1]]
#[[1, 1], [3, 2], [3, 2], ['d', 1], ['z', 1]]
#[[1, 2], [1, 2], [3, 3], [3, 3], [3, 3], ['d', 3], ['d', 3], ['d', 3], ['x', 1], ['y', 1], ['z', 1]]
#[[1, 1], [3, 2], [3, 2], ['d', 2], ['d', 2], ['x', 1], ['y', 1], ['z', 1]]

cAL = sorted([[y,cA(y)] for y in A])

update: 2

Basically I think it is time for me to learn:

• Pandas

It looks like a combination of aggregation, groupby, and summing.

Answer 1

After further review (and experimentation with a Python interpreter now that I'm home), I see what you're trying to do, but it contradicts your title about removing duplicates. I see you're viewing each additional element as a new indexed unique item.

This is conceptually similar to the decorate, sort, undecorate pattern, just substitute the term "sort" with "join" or "set operation".

So here's a setup, first import itertools so we can group each like element and enumerate them into a set:

import itertools

def indexed_set(a_list):
    '''
    assuming given a sorted list, 
    groupby like items, 
    and index from 0 for each group
    return a set of tuples with like items and their index for set operations
    '''
    return set((like, like_index) for _like, like_iter in itertools.groupby(a_list)
                          for like_index, like in enumerate(like_iter))

Later we'll need to convert the set with indexes back to a list:

def remove_index_return_list(an_indexed_set):
    '''
    given a set of two-length tuples (or other iterables)
    drop the index and 
    return a sorted list of the items 
    (sorted by str() for comparison of mixed types)
    '''
    return sorted((item for item, _like_index in an_indexed_set), key=str)

Finally, we need our data (taken from the data you provided):

a = [1, 2, 2, 3] 
b = [1, 2, 3, 3, 4] 
expected_result = [1, 2, 2, 3, 3, 4]

And here's my proposed usage:

a_indexed = indexed_set(a)
b_indexed = indexed_set(b)
actual_result = remove_index_return_list(a_indexed | b_indexed)

assert expected_result == actual_result

does not raise an AssertionError, and

print(actual_result)

prints:

[1, 2, 2, 3, 3, 4]

---

EDIT: Since I made the functions handle mixed cases, I figured I'd demo:

c = [1,'d','d',3,'x','y']
d = [1,'d',3,3,'z']
expected_result =  [1,'d','d',3,3,'x','y','z']
c_indexed = indexed_set(c)
d_indexed = indexed_set(d)
actual_result = remove_index_return_list(c_indexed | d_indexed)
assert actual_result == expected_result

And we see we don't have exactly what we expect, but pretty close due to sorting:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AssertionError
>>> actual_result
[1, 3, 3, 'd', 'd', 'x', 'y', 'z']
>>> expected_result
[1, 'd', 'd', 3, 3, 'x', 'y', 'z']

Answer 2

No need to learn pandas yet! (Although it's a really excellent library.) I'm not sure if I understand your problem exactly but the collections.Counter data type is designed to act as a bag/multiset. One of the operators implemented is "or" which might be what you need. Read the comments in this code example and see if it fits your needs:

a = [1, 2, 2, 3]
b = [1, 2, 3, 3, 4]

from collections import Counter

# A Counter data type counts the elements fed to it and holds
# them in a dict-like type.

a_counts = Counter(a) # {1: 1, 2: 2, 3: 1}
b_counts = Counter(b) # {1: 1, 2: 1, 3: 2, 4: 1}

# The union of two Counter types is the max of each value
# in the (key, value) pairs in each Counter. Similar to
# {(key, max(a_counts[key], b_counts[key])) for key in ...}

result_counts = a_counts | b_counts

# Return an iterator over the keys repeating each as many times as its count.

result = list(result_counts.elements())

# Result:
# [1, 2, 2, 3, 3, 4]

Answer 3

I think the test case in the problem statement is not enough, for example, suppose

a = [1, 2, 2, 3, 2, 2, 3] b = [1, 2, 2, 3, 3, 4, 3, 3, 5]

shall we merge the two into [1, 2, 2, 2, 2, 3, 3, 4, 3, 3, 5], or [1, 2, 2, 3, 3, 4, 5]? This will definitely change the algorithm you are going to implement.

Answer 4

So you're asking how to remove duplicate elements and keep unique ones? You definitely need sets for that:

When you say this:

(a - b) + (b - a)

What you want is this

set(a) ^ set(b)

Which is the symmetric difference of the two.

If your elements are lists, you won't be able to hash them (a prerequisite for a set element), so you need to convert them to tuples:

set(tuple(i) for i in a) ^ set(tuple(i) for i in b)

---

EDIT

Now that you've edited your question, you appear to be looking for this:

(a - b) + (b - a) + a ∩ b

Which is the union of the two sets (assuming you mean the union of the sets by +, else you would mean the intersection, which would be the null set, and this ambiguity is the reason sets do not support the + operator):

set(tuple(i) for i in a) | set(tuple(i) for i in b)

The above returns the equivalent to the end result of my_set using the in-place function, union:

my_set = set(tuple(i) for i in a) 
my_set.union(tuple(i) for i in b)