How to get the execution id of a restarted job in Spring Cloud Data Flow

Question

We have created a Spring Batch job to be executed in Spring Cloud Data Flow through Spring Cloud Task (a simple task, it only executes the job). The execution has been checked both with the UI and the REST API, and everything is OK in an ideal case. The problem comes when we try to stop and restart a job. Following the REST API Guide:

1. Launch the task: https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#api-guide-resources-task-executions-launching
2. Get information of the task execution using the task execution id returned in the previous point: https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#api-guide-resources-task-executions-detail
3. Here comes the first issue: if we stop the task with the task execution id (https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#api-guide-resources-task-executions-stopping) the job associated to the task stops its execution (in the logs) but in de database the job stays as "STARTED". And then it is impossible to restart the job execution as it never reachs the "STOPPED" status --> For the SCDF gurus... isn't that a bug or there is a technical reason?
4. If we want to stop the job, it is necessary to get the jobExecutionIds first param from the response of the point 2, and make the request with the job execution id (https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#api-guide-resources-job-executions-stop)
5. Waiting the appropiate time to complete the running step execution, the job reaches the "STOPPED" status. If now we restart the job (https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#api-guide-resources-job-executions-restart) a new task execution is created, linked with the new job execution, and it completes the processing.

And now the million dollar question: if we want to integrate SCDF with a planifier (like Control-M) how can we make polling requests to check the after-restart execution's status if the restart request does not return any execution id? If there is just a normal execution, we can send polling requests with the job execution id (step 4) associated to the task execution id (step 2), but after the restart we are "blind"

Answer 1

Well, I don't know if there is a directer way to do this, but this is a possible solution for the integration with an external system:

1. Launch the task: https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#api-guide-resources-task-executions-launching this returns the task execution id
2. Recover the job execution id associated to the previous task execution id: https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#api-guide-resources-task-executions-detail being job execution id the field "executionId"
3. Use the job execution id to stop the execution (optionally, it could have been finished by an error): https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#api-guide-resources-job-executions-stop
4. Use the job execution id to restart/resume the execution: https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#api-guide-resources-job-executions-restart
5. Use the "old" job execution id to get the job instance id: https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#api-guide-resources-job-executions-detail the job instance id is the field "jobId"
6. Use the job instance id to get the "new" job execution id: https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#api-guide-resources-job-instances-detail where the field "jobExecutions" is an array of execution where the first position is the last execution, so we can get the new job execution id from the field "executionId"

Answer 2

Here comes the first issue: if we stop the task with the task execution id (https://docs.spring.io/spring-cloud-dataflow/docs/current/reference/htmlsingle/#api-guide-resources-task-executions-stopping) the job associated to the task stops its execution (in the logs) but in de database the job stays as "STARTED". And then it is impossible to restart the job execution as it never reachs the "STOPPED" status

This looks like a bug and feel free to report it at https://github.com/spring-cloud/spring-cloud-dataflow/issues. When reporting, please add any additional information such as the SCDF server log etc., to investigate it better.

if we want to integrate SCDF with a planifier (like Control-M) how can we make polling requests to check the after-restart execution's status if the restart request does not return any execution id? If there is just a normal execution, we can send polling requests with the job execution id (step 4) associated to the task execution id (step 2), but after the restart we are "blind"

SCDF server exposes REST endpoints to check the status of Job Executions. You can either hit the server endpoints or use the REST client to check. For more information on the REST client, you can check [here].1