CODEWITHSUNDEEP
google-cloud-composer
cryanbhu
cryanbhu

Reputation: 5254

Clear cloud composer all Successful dag run states

I've read a question on clearing DAG run states in Composer but its not obvious,
How I can clear all the DAG run states including those which are Successful?
i.e. I have a backfill job which I would like to start from scratch.

I also read the Airflow clear CLI Docs, it doesn't seem to show how I can only select Successful ones too.

Upvotes: 0

Views: 516

Answers (1)

cryanbhu
cryanbhu

Reputation: 5254

I found out, you need to add the --upstream and --downstream parts into the command in gcloud, and then it'll clear the task states too. For example:

gcloud composer environments run jido --location=asia-northeast1 clear -- --dag_regex 'val' -c -s 2018-12-31 -e 2020-01-01 --upstream --downstream

$ gcloud composer environments run <environment> \
    --location=asia-northeast1 clear -- <DAG_ID> \
      -c \
      -s <dag run start date> \
      -e <dag run end date> \
      --upstream \
      --downstream

Example with fields filled in:

$ gcloud composer environments run mycomposerenvironment \
    --location=asia-northeast1 clear -- my_important_dag \
      -c \
      -s 2019-08-20 \
      -e 2019-08-31 \
      --upstream \
      --downstream

You can also use dag_regex which is very useful if you have dynamic dags: 

$ gcloud composer environments run mycomposerenvironment \
    --location=asia-northeast1 clear -- --dag_regex '_val_' \
      -c \
      -s 2019-08-20 \
      -e 2019-08-31 \
      --upstream \
      --downstream

Upvotes: 3

Related Questions