Reputation: 297
I'm unable to paste my code snippet, so I'll explain the scenario 1. I have a list of around 50000 stocks that need some valuation 2. These stocks are in a MultiProcess.Queue 3. I create multiple processes, each process will take a batch of 50 from the queue and do something. 4. in the main thread, I have a check which looks like this
anymore_to_process = True
while anymore_to_process:
if (stock_queue.qsize() == 0):
anymore_to_process = False
for jobs in stock_jobs:
jobs.join()
what am I doing wrong? Why are the processes not joining when I process a lot of stocks.
I know this is difficult to answer without looking at my code.....but if you can give me some pointers, that would be very helpful.
Upvotes: 11
Views: 9947
Reputation: 297
Problem resolved - took a cue from @dano's question. I was indeed writing to another queue, which was blocking the processes. I took it out and problem resolved.
Upvotes: 9