CODEWITHSUNDEEP
kubernetesspring-batchbatch-processing
Daniel Setiawan
Daniel Setiawan

Reputation: 123

Batch Processing on Kubernetes

Anyone here have experience about batch processing (e.g. spring batch) on kubernetes ? Is it good idea ? How to prevent batch processing process same data if we use kubernetes auto scaling feature ? Thank you.

Upvotes: 12

Views: 11975

Answers (1)

Mahmoud Ben Hassine
Mahmoud Ben Hassine

Reputation: 31600

Anyone here have experience about batch processing (e.g. spring batch) on kubernetes ? Is it good idea ?

For Spring Batch, we (the Spring Batch team) do have some experience on the matter which we share in the following talks:

Running batch jobs on kubernetes can be tricky:

  • pods may be re-scheduled by k8s on different nodes in the middle of processing
  • cron jobs might be triggered twice
  • etc

This requires additional non-trivial work on the developer's side to make sure the batch application is fault-tolerant (resilient to node failure, pod re-scheduling, etc) and safe against duplicate job execution in a clustered environment.

Spring Batch takes care of this additional work for you and can be a good choice to run batch workloads on k8s for several reasons:

  • Cost efficiency: Spring Batch jobs maintain their state in an external database, which makes it possible to restart them from the last save point in case of job/node failure or pod re-scheduling
  • Robustness: Safe against duplicate job executions thanks to a centralized job repository
  • Fault-tolerance: Retry/Skip failed items in case of transient errors like a call to a web service that might be temporarily down or being re-scheduled in a cloud environment

I wrote a blog post in which I explain all these aspects in details with code examples. You can find it here: Spring Batch on Kubernetes: Efficient batch processing at scale

How to prevent batch processing process same data if we use kubernetes auto scaling feature ?

Making each job process a different data set is the way to go (a job per file for example). But there are different patterns that you might be interested in, see Job Patterns from k8s docs.

Upvotes: 18

Related Questions