CODEWITHSUNDEEP
spring-xd
Peter Rietzler
Peter Rietzler

Reputation: 481

How does Spring-XD handle job execution

I can't get the information out of the documentation. Can anyone tell me how Spring-XD executes jobs? Does it assign a job to a certain container and is this job only executed on the container it is deployed to, or is each job execution assigned to another container? Can I somehow control that a certain job may be executed in parallel (with different arguments) and others may not ?

Thanks! Peter

Upvotes: 0

Views: 2096

Answers (1)

Ilayaperumal Gopinathan
Ilayaperumal Gopinathan

Reputation: 4179

I am sure you would have seen some of the documentation here: https://github.com/spring-projects/spring-xd/wiki/Batch-Jobs

To answer your questions:

Can anyone tell me how Spring-XD executes jobs? Does it assign a job to a certain container and is this job only executed on the container it is deployed to, or is each job execution assigned to another container?

After you create a new job definition using this:

xd>job create dailyfeedjob --definition "myfeedjobmodule" --deploy

the batch job module myfeedjobmodule gets deployed into the XD container. Once deployed, there is a job launching queue setup in the message broker: redis, rabbit or local. The name of the queue is job:dailyfeedjob in the message broker. Since this queue is bound to the job module deployed in the XD container, a request message sent to this queue is picked by the job module deployed inside that specific container.

Now, you can send the job launching request message (with job parameters) into the job:dailyfeedjob queue by simply setting up a stream that sends a message into this queue. For example: a trigger (fixed-delay, cron, date triggers) could do that. This also a job launch command from the shell which launches job only once. This section would explain it more: https://github.com/spring-projects/spring-xd/wiki/Batch-Jobs#launching-a-job

Hence, the job is launched (every time it receives the job launching request) only inside the container where the job module is deployed and you can expect original the spring batch flow when the job is executed. (refer to shell doc for all the job related commands)

Can I somehow control that a certain job may be executed in parallel (with different arguments) and others may not ?

If it is for the different job parameters for the same job definition, then it would go to the same container where the job module is deployed. But, you can still create a new job definition with the same batch job module. 

xd>job create myotherdailyfeedjob --definition "myfeedjobmodule" --deploy

The only difference being it will be under that namespace. and, the job launching queue name would job:myotherdailyfeedjob. It all depends on how do you want to organize running your batch jobs.

Also, for parallel processing batch jobs you can use: http://docs.spring.io/spring-batch/reference/html/scalability.html

and, XD provides single step partitioning support for running batch jobs:

Include this in your job module:

<import resource="classpath:/META-INF/spring-xd/batch/singlestep-partition-support.xml"/>

with partitioner and tasklet beans defined.

You can try out some of the XD batch samples from here: https://github.com/spring-projects/spring-xd-samples

Upvotes: 1

Related Questions