Multi Threading: Multiple threads collecting information, One does work

Question

I am writing code which should have the following structure:

• multiple threads are collecting data. The data is put in a buffer.
• simultaneously does one thread pop data from that buffer, does some work on it, and stores it in a different buffer (only this thread has access to).
• the worker thread only terminates if all the "collector" threads are done and all data in the buffer has been processed.

Could someone give me some code/pseudo code on how to do something like this. Mainly on what kind of buffer that I should use & how the threads communicate such that all data is collected and processed correctly.

I have read up on the "Queue" and "threading" class in python. However, I am still not sure on how to do this properly. I have tried multiple easy examples with the threading.Event() object - however results are inconsistent.

Thanks alot!

Answer 1

If your worker can see the collectors being done, then it's as simple as checking while the threads are not done OR the queue is not empty.

If your worker cannot see the collectors state or even how many there are, you can have the collectors communicate that state in the queue itself, along the data.

Imagine the data object has an origin field, and a boolean field to mark as open or closed. For example, collector 1 would send something like:

data-1-opened, data-1-opened ... data-1-closed.

The worker only has to maintain a set of collector ids, adding new entries as they appear, and removing them when closed. So the exit condition is that the set is empty AND the data queue is empty.