CODEWITHSUNDEEP
architectureairflowpipeline
Adelin
Adelin

Reputation: 18991

Airflow grouping DAGs by common id or params

In Airflow most pipeline consists of a group of DAGs each DAG contains tightly related tasks. Does Airflow have some construct/object which can group DAGs together? So that one can track the success or failure of the whole pipeline?

Something like in the picture:

enter image description here

Upvotes: 2

Views: 761

Answers (1)

Hussein Awala
Hussein Awala

Reputation: 5100

To create a relationship between two separate dags, airflow provides ExternalTaskSensor to add a dependency to a task in another dag. When you use this sensor, you can access the graph of DAG dependencies available on <AIRFLOW_SERVER_URI>/dag-dependencies.

Also, Airflow provide an experimental lineage support based on OpenLineage, which allows you to use an external lineage platforms (ex: Marquez). By using a lineage tool, you can define the dependencies between the different airfow dags to create bigger business/functional dags, and track the success or failure of the whole pipeline.

Upvotes: 1

Related Questions