CODEWITHSUNDEEP
airflowamazon-eksdbt
LeoC
LeoC

Reputation: 41

Connect dbt to Airflow using EKS

We currently deployed helm Airflow in AWS EKS and want to trigger dbt models from it.

A few questions: 1. What would be the ideal way to deploy dbt? I am thinking about deploying another container for dbt only or installing dbt in the same container running Airflow via pip or brew.

2. If the ideal way to run dbt is in its own container, how do I connect airflow to dbt?

Please feel free to add any relevant information!

Upvotes: 2

Views: 1321

Answers (1)

Jarek Potiuk
Jarek Potiuk

Reputation: 20097

  1. I think you should consider switching to the Official chart that he Apache Airflow community released recently: https://airflow.apache.org/docs/helm-chart/stable/index.html - it is prepared and maintained by the same community that builds Airflow.

    1. I think one of the best descriptions about how to integrate dbt you can find in this Astronomer's blog: https://www.astronomer.io/blog/airflow-dbt-1

Summarising it - if you do not want to use dbt cloud, you can install dbt as pip package, and either run it via Bash script, or using dedicated DBT operators. If you already use Airflow through image, connecting dbt image to it when it should be invoked in another image, while technically possible, is a bit challenging and likely not worth the hassle.

You should simply extend the Airflow Image and add dbt as pip package. You can learn how to extend or customize the Airflow Image here: https://airflow.apache.org/docs/docker-stack/build.html

Small follow-up. Not sure if you have seen the talk from last week's Airflow Summit but I highly recommend it: https://airflowsummit.org/sessions/2021/building-a-robust-data-pipeline-with-the-dag-stack/ - it might give you a bit more answers :)

Upvotes: 1

Related Questions