CODEWITHSUNDEEP
pythonpython-2.7multiprocessingpython-multithreading
user2097496
user2097496

Reputation: 297

Python Processes not joining

I'm unable to paste my code snippet, so I'll explain the scenario 1. I have a list of around 50000 stocks that need some valuation 2. These stocks are in a MultiProcess.Queue 3. I create multiple processes, each process will take a batch of 50 from the queue and do something. 4. in the main thread, I have a check which looks like this

 anymore_to_process = True

 while anymore_to_process:
     if (stock_queue.qsize() == 0):
         anymore_to_process = False

for jobs in stock_jobs:
    jobs.join()
  1. however, this doesn't seem to work when i process 50000 records. If I process 500 stocks, this works fine.

what am I doing wrong? Why are the processes not joining when I process a lot of stocks.

I know this is difficult to answer without looking at my code.....but if you can give me some pointers, that would be very helpful.

Upvotes: 11

Views: 9947

Answers (1)

user2097496
user2097496

Reputation: 297

Problem resolved - took a cue from @dano's question. I was indeed writing to another queue, which was blocking the processes. I took it out and problem resolved.

Upvotes: 9

Related Questions