CODEWITHSUNDEEP
airflow-schedulerairflow
palamuGuy
palamuGuy

Reputation: 357

Add job to existing Airflow DAG without dependency on any job

I am creating an Airflow job that will run as part of an existing dag that has n number of jobs. I have to add this new job as independent job

My current job dependency is as below

accountable_job >> dq_check >> dq_a1_validaton_job >> data_aggregation_job >> sync_job

I have to add another job dq_b1_validaton_job that will be independent but jobs after dq_a1 validation will be dependent on dq_b1_validaton_job. In sort dq_a1_validaton_job and dq_b1_validaton_job will be in parallel but dq_b1_validaton_job will be independent of any job.

Upvotes: 0

Views: 544

Answers (1)

Jarek Potiuk
Jarek Potiuk

Reputation: 20097

You just add it to dag. Either with context manager:

with Dag(...):
    independent_task = YourOperator()

or by passing dag as parameter:

your_dag = Dag(....)

independent_task = YourOperator(..., dag=your_dag)

See https://airflow.apache.org/docs/apache-airflow/stable/concepts/dags.html#declaring-a-dag

Upvotes: 1

Related Questions