How to use DeployerPartitionHandler with MessageChannelPartitionHandler

Question

I have a Spring Batch job using Spring Integration for messaging middleware (Kafka). It uses RemotePartitioningManagerStepBuilder and RemotePartitioningWorkerStepBuilder for scaling the workers to handle the load created by manager.

I want to use Spring Cloud Data Flow to register and schedule the job. I see that the worker instance can be launched using DeployerPartitionHandler from Spring Cloud Task, so that when the manager instance is started, the worker step is executed using the partitionHandler – DeployerPartitionHandler.

The RemotePartitioningManagerStepBuilder creates its own MessageChannelPartitionHandler for Kafka messages to be consumed by workers. Is it possible to use DeployerPartitionHandler in such a scenario. Most of the examples for remote partitioning expect the worker instance to be launched manually or running all the time. But a batch job (manager and workers) in a Kubernetes environment should only be running and shut down after the batch job is complete.

The answer to Spring batch Partitioning execute slave step in other pods in Kubernetes says that workers can be packaged like regular app and be a long-running process. Is it possible for these workers to be short-lived?

The DeployerPartitionHandler works with a StepBuilder but not with a RemotePartitioningManagerStepBuilder. Is there a way to deploy worker instances using Spring Cloud Data Flow for a batch job using remote partitioning for scaling?

It seems like the RemotePartitioningManagerStepBuilder is not meant to work with any other partitionHandler. It has this unsupported operation exception - When configuring a manager step for remote partitioning using the RemotePartitioningManagerStepBuilder, the partition handler will be automatically set to an instance of MessageChannelPartitionHandler. The partition handler must not be provided in this case.