CODEWITHSUNDEEP
pythonmultiprocessing
BigBadMe
BigBadMe

Reputation: 1852

Start new process without waiting for all other processes to finish

If I start multiple processes like this:

        procs = []
        for agent_id in agents:
            proc = Process(target=beginmultiproc, args=(id))
            procs.append(proc)
            proc.start()

        # wait for processes to finish
        for proc in procs:
            proc.join()

Is there a way I can start a new process as soon as one has finished, rather than needing to wait for all to finish? I want to run 8 processes at a time as that's how many cores my CPU has, but I have more than 8 jobs that I need to run, so as soon as one process has finished I want to start the next process running.

I'm aware of pool.map but I can't use it in this case.

Upvotes: 2

Views: 1097

Answers (2)

new-dev-123
new-dev-123

Reputation: 411

You could also take a look at ProcessPoolExecutor which, if I remember correctly, is built on top of multiprocessing and provides a somewhat simpler way of doing things.

Alternatively, check concurrent futures which has an "as_completed" method which will yield results as each process completes allowing you to work with those values immediately.

Upvotes: 2

Sıddık Açıl
Sıddık Açıl

Reputation: 967

You do not have more processes than 8. You have more "work-items/tasks" than 8. Use a multiprocessing.queue to schedule your work. If a process is idle get something from this queue and process it until the queue is finished. You do not have to kill processes and spawn another one.

Upvotes: 1

Related Questions