CODEWITHSUNDEEP
pythondictionary
personalt
personalt

Reputation: 860

Can you iterate over a dictionary while data is being added to the dictionary by another thread?

I have a Python application where thread1 calls an API to see 'what reports are ready to download' and sends that report_id to thread2 which 'downloads/processes those reports. I am trying to determine what happens if thread1 is adding items to the dict if thread2 is iterating over it. Right now I do a copy before working in thread2

Two questions

  1. Can I iterate over a changing dictionary? I currently a) make a copy of dict before I iterate, b) iterate over a copy of the dict, c) for items that are 'processed' by the loop on the copy of the dict I delete the key from the 'original' dict so on the next loop it doesn't reprocess the same item

  2. If I can't iterate over a changing dict do I need to use a lock to make a copy like I am doing below. Is that the right way to do it?

lock = threading.Lock()
while True:

        with lock:  #copy dict to prevent contenion
            reports_to_call_copy = self.reports_to_call.copy()

        for line in reports_to_call_copy:
              #do stuff and delete items from original dict so on next loop it doesn't process twice. 


        is_killed = self._kill.wait(180)
        if is_killed:
            print("Killing - Request Report")
            break
              del self.reports_to_call[user, report_name]

Upvotes: 0

Views: 52

Answers (1)

Marc Sances
Marc Sances

Reputation: 2614

  1. No, it is not possible to iterate a changing dictionary, even when changed from the same thread. Simplest reproducible example:
>>> d = dict()
>>> d['a'] = 10
>>> for k, v in d.items():
...     del d['a']
...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: dictionary changed size during iteration
  1. Any code that manipulates the dictionary (such as the del statements below) must acquire the lock as well, otherwise, it may delete items at any time while your thread is creating the deep copy, potentially causing the same trouble, particularly if the dictionary size is large enough that the deep copy could be interrupted by the deletion process.

Upvotes: 1

Related Questions