Kubernetes number of Pods according to tasks

Question

I have a certain processing task that I want to solve with kubernetes. The basic concept is that there is a certain number of items in a work queue that I want to process. Items can be added to the queue and are deleted as soon as a pod finished processing one. The preferred work flow would be:

• Defining a maximum number of pods(e.g.40)
• push items to queue(e.g. 20)
• number of pods is created according to number of items in queue(=>20)
• while the pods are still processing on the 20 items another 40 items are pushed to the queue resulting in 20 more pods that are created(maximum number is reached), and as soon as the first ones finish additional pods are created until the end of the queue is reached.

Is there any build in solution using kubectl? Using the job pattern I can define the number of parallel Pods but those are running all the time until success and not scaled up upon other criteria.

Thanks for your help!

Answer 1

Use a Horizontal Pod Autoscaler. You may have to define Custom Metrics for getting the number of items in the Queue and use it in HPA.