How to work with queues for file processing in python?

Question

I'm currently writing an application in python that mainly works with files. So the algorithm works like this:

A user submits a file through an API, imagine a post request with the file and some data.

Then the program works with the file and extracts some conclusions.

After that, those conclusions are stored inside a DB.

Then the user is able to query the db and ask for conclusions.

As the user is able to submit a file through an API and this can be done simultaneously by many users in many systems and the process of file processing may take some time. I want to explore a way to implement a work queue such as:

Only one file can be processed at a time, so when a user submits a file, that file is put inside a work queue and so has to wait before getting inside the "processing function".

How can I do that, any reference or tutorial?

Thanks

Answer 1

Checkout Celery, there are many good tutorials online. It works with workers so it also doesn't block you api listening. Also it could provide the option to process multiple files at once concurently if you'd want.