CODEWITHSUNDEEP
google-cloud-platformairflowgoogle-cloud-composer
cryanbhu
cryanbhu

Reputation: 5254

Debugging broken dags in GCP Composer

I have read the question for vanilla Airflow.

How can broken DAGs be debugged effectively in Google Cloud Composer?
How can i see the full logs of a broken DAG?

Right now I can only see one line of trace in Airflow UI main page.

EDIT:
Answers seem to be not understanding my question.
I am looking for fixing broken DAGs i.e. the DAG does not even appear in the DAGs list and of course there are no tasks running and no task logs to view.

Upvotes: 2

Views: 3095

Answers (2)

LEC
LEC

Reputation: 247

As hexacynide pointed out, you can look at the task logs - there's details in the Composer docs about doing that specifically found here. You can also use Stackdriver logging, which is enabled by default in Composer projects. In Stackdriver logs, you can filter your logs on many variables, including by time, by pod (airflow-worker, airflow-webserver, airflow-scheduler, etc.) and by whatever keywords you suspect might appear in the logs.

EDIT: Adding screenshots and more clarity in response to question update

In Airflow, when there's a broken DAG, there is usually some form of error message at the top. (Yes, I know this error message is helpful and I don't need to debug further, but I'm going to just to show how to)

In the message, I can see that my DAG bq_copy_across_locations is broken.

Airflow Web UI with "Broken DAG" Error message

To debug, I go to Stackdriver, and search for the name of my DAG. I limit the results to the logs from this Composer environment. You can also limit the time frame if needed.

Stackdriver logs with DAG name in search bar

I looked through the error logs and found a the Traceback error for the broken DAG.

Logs with traceback line highlighted

Alternatively, if you know you only want to search for the stack traceback, you can run an advanced filter looking for your DAG name and the word "traceback". To do so, click the arrow at the right side of the Stackdriver logging bar and hit "convert to advance filter"

Stackdriver logs with arrow pointing to convert to advance filter

Then enter your advanced filter

resource.type="cloud_composer_environment"
resource.labels.location="YOUR-COMPOSER-REGION"
resource.labels.environment_name="YOUR-ENV-NAME"
("BROKEN-DAG-NAME" AND
"Traceback")

This is what my advanced search looked like

Advanced filter for traceback logs

The only logs that will be returned will be the stack Traceback logs for that DAG.

Tracebacks in Stackdriver logs

Upvotes: 3

hexacyanide
hexacyanide

Reputation: 91769

To determine run-time issues that occur when a DAG is triggered, you can always look at task logs as you would for any typical Airflow installation. These can be found using the web UI, or by looking in the associated logs folder in your Cloud Composer environment's associated Cloud Storage bucket.

To identify issues at parse time, you can execute Airflow commands using gcloud composer. For example, to run airflow list_dags, the gcloud CLI equivalent would be:

$ gcloud composer environments --location=$REGION run $ENV_NAME -- list_dags --report

Note that the second -- is intentional. This is so that the command argument parser can differentiate between arguments to gcloud and arguments to be passed to the Airflow subcommand (in this case list_dags).

Upvotes: 1

Related Questions