CODEWITHSUNDEEP
kubernetesnextflow
David Parks
David Parks

Reputation: 32111

Running nextflow as a job vs. a pod in kubernetes

I'm new to nextflow. We would like to build our workflow using nextflow and have nextflow deploy the workflow to a large mulit-institution Kubernetes cluster that we use.

In this cluster we don't have admin permission, we have a namespace we work in. Also, pods in our cluster have limited resources, but jobs have unlimited resources.

In looking at the documentation for nextflow + kubernetes and it says that the workflow runs under a Kubernetes pod, which raises red flags for me because of the limitation on pods in our cluster.

Is there a way to execute nextflow workflows as a kubernetes jobs instead of a pod? What are my options in this area?

Upvotes: 1

Views: 754

Answers (2)

Carambakaracho
Carambakaracho

Reputation: 315

There might be a new feature with nextflow 22.04. Quoting Ben Sherman's nextflow k8s best practices page:

In Nextflow v22.04 and later, the k8s executor can be configured to use Jobs instead of Pods directly.

Upvotes: 1

David Parks
David Parks

Reputation: 32111

Based on a conversation on https://gitter.im/nextflow-io/nextflow, Nextflow cannot run a Job, it only supports spawning Pods. One pod is spawned for workflow control, and that Pod spawns pods for the individual tasks.

Upvotes: 1

Related Questions