CODEWITHSUNDEEP
pythonkubernetesairflow
user430953
user430953

Reputation: 355

Airflow - using multiple pod templates

We have deployed airflow using the official helm chart and we are using KubernetesExecutor with git-sync. I already managed to launch a worker pod with a custom pod template using the executor_config parameter as mentioned in https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/executor/kubernetes.html#pod-override. However I am unable to override the default worker docker image. How do I need to setup the helm values file to be able to override the docker image? Currently I have this in my values file

 images:
   airflow:
     repository: <custom-docker-image>
     tag: webserver
 
 config:
   kubernetes_executor:
     worker_container_repository: ~
     worker_container_tag: ~

The pod template is exact copy paste of https://github.com/apache/airflow/blob/main/chart/files/pod-template-file.kubernetes-helm-yaml with only the image changed to custom-docker-image:test_template . My worker pods are still using the default airflow image custom-docker-image:webserver. If I set the kubernetes_executor keys to not null values it will use those, but I am still unable to override those.
I know how to override the docker image of the default pod template, but my idea is to have multiple pod templates, then use the executor_config in my task decorator to pick a pod template from a folder. This way I would basically have multiple worker environments to choose from.
I am also aware I can do that with pod overrides as mentioned in the docs, I am interested how to achieve this using the templates though.

Upvotes: 0

Views: 599

Answers (1)

user430953
user430953

Reputation: 355

I dont really know why, but it started working once I specified a random docker image under

images:
  pod_template:
    repository: MY_CUSTOM_DOCKER_IMAGE
    tag: latest

Upvotes: 0

Related Questions