CODEWITHSUNDEEP
kubernetesairflow
AlanPear
AlanPear

Reputation: 737

Airflow When to use KubernetesExecutor vs KubernetesPodOperator?

Does the Executor automatically send tasks to pods, or do I need the operator to do that?

Each operator has the "executor_config" param so I'm not sure when to use either.

Upvotes: 6

Views: 7158

Answers (1)

brandondtb
brandondtb

Reputation: 311

They serve different purposes.

Briefly:

KubernetesExecutor: You need to specify one of the supported executors when you set up Airflow. The executor controls how all tasks get run. In the case of the KubernetesExecutor, Airflow creates a pod in a kubernetes cluster within which the task gets run, and deletes the pod when the task is finished.

Basically, you would use this instead of something like Celery.

KubernetesPodOperator: This allows you to, essentially, run a container as a task, and that container will be run inside a pod on a kubernetes cluster.

You'd use this if you if you have containerized workloads that you need to schedule in Airflow, or if you have non-python code you want to execute as Airflow tasks.

Hope that is helpful.

Upvotes: 15

Related Questions