Algorithm to add new, update existing and remove missing data

Question

I have done lots of searching, but been unable to find a satisfactory answer to the most efficient approach to achieve the following.

Say my App contains a list of Products. At the end of every day an external service is called that returns another list of Products from a master data source.

• If the list of Products from master data contains any Products not in my App, add the Product to the App.

• If the Product in the master data is already in my App, and no changes have been made, do nothing.

• If the Product in the master data is already in my App, but some data has changed (the Product's name for instance), update the Product.

• If a Product is available in my App, but no longer in the master data source, flag it as "Unavailable" in the App.


At the moment, I do a loop on each list, looping through the other list for each Product:

• For each Product in the master data list, loop through the Products in the App, and update as needed. If no Product was found, then add the Product to the App. 

• Then, for each Product in the App, loop through the Products in the master data list, and if not found, flag as "Unavailable" in the App.


I'm wondering if there is a more efficient method to achieve this? Or any algorithms or patterns that are relevant here?

In each case the Products are represented by objects in a Python list.

Answer 1

First of all I'd suggest to use dicts with the Product code (or name or whatever) as key and the Product object as value. This should make your loops faster by at least a 100x factor on a thousand entries.

Then especially for the second search it may be worth exploring the possibility of converting the keys of the first dict to a set and looping on the difference as in

for i in set(appDict.keys()).difference(masterDict.keys()):
    ##update unavailable Product data