CODEWITHSUNDEEP
dockerkubernetesairflowdagster
Manrique Vargas
Manrique Vargas

Reputation: 63

Run a new task in Airflow Kubernetes with it's own isolated Dockerfile?

How do I run a new pipeline with it's own isolated Dockerfile in Airflow Kubernetes ?

I've been using Dagster and I can run new pipelines on their own Dockerfile, but can't figure out how to do this in Airflow

Upvotes: 2

Views: 655

Answers (1)

Hussein Awala
Hussein Awala

Reputation: 5088

If you want to run a docker container task on Kubernetes using Airflow, regardless the executor you are using and how you deployed Airflow server, you can use KubernetesPodOperator.

You can specify the docker image by providing the argument image, you can also override the image entrypoint and provide extra args (cmds and arguments). And you can configure your pod as you need (labesl, volumes, secrets, configMaps, ...).

Upvotes: 3

Related Questions