CODEWITHSUNDEEP
pythonmultithreadingconcurrencydependenciesmultiprocessing
Mike
Mike

Reputation: 386

Python multi-threading/processing module for tasks with dependencies requiring sequencing

I'm in the process of implementing a Python module that provides a thread (or process) pool that can handle processing tasks concurrently that are potentially dependent on each other. For example, in the case of an order management system, you could place an order, place another order, cancel the second order, place a third order, and then cancel the first order - all order placements can be processed at the same time, but the cancellations can't happen at the same time as their respective orders and must happen after their order placements are completed. I have come up with a good general purpose solution to problems like this and wanted to use it as my first open source contribution.

Before spending too much time on this I just wanted to know if something like this already exists. Does anyone know of an existing Python package that does this? Would this be useful to anyone?

Upvotes: 2

Views: 898

Answers (1)

Anton
Anton

Reputation: 6537

Yes, take a look at . It has different schedulers for multithreaded, multiprocessing, and distributed computing which reuse threads and processes through TaskPool and Pool respectively. You can build any task dependencies (direct acyclic graphs - DAGs) with it. Though, I'm not sure about cancellation support but you can implement it manually in your tasks.

Upvotes: 1

Related Questions