CODEWITHSUNDEEP
google-cloud-platformgoogle-cloud-composergoogle-cloud-logging
Tim
Tim

Reputation: 7464

Is it possible to display Airflow logs in GCP Logging?

GCP documentation distinguishes between two kinds of logs for Composer:

Cloud Composer has the following Airflow logs:

Airflow logs: These logs are associated with single DAG tasks. You can view the task logs in the Cloud Storage logs folder associated with the Cloud Composer environment. You can also view the logs in the Airflow web interface.

Streaming logs: These logs are a superset of the logs in Airflow. To access streaming logs, you can go to the logs tab of Environment details page in Google Cloud Console, use the Cloud Logging, or use Cloud Monitoring.

As documented, I can see the Airflow logs in the buckets, but they do not display in GCP Logging, should they? In fact, when I query Logging for resource.type="cloud_composer_environment" I can see very few logs including no errors while I know that in this period I noticed some failed jobs that resulted in errors. Does Logging need any special configuration for the Airflow logs to be displayed?

Upvotes: 2

Views: 2384

Answers (1)

Wojtek_B
Wojtek_B

Reputation: 4443

Airflow logs are stored in the corresponding buckets and are not visible in the Logs Explorer - you have to go to the Cloud storage and find a bucket corresponding to the environment you want to view:

The logs folder includes folders for each workflow that has run in the environment. Each workflow folder includes a folder for its DAGs and sub-DAGs. Each folder contains log files for each task. The task filename indicates when the task started.

The following example shows the logs directory structure for an environment.

Logs are stored in a folder structure described also in the documentation;

us-central1-my-environment-60839224-bucket
   └───dags
   |   │
   |   |   dag_1
   |   |   dag_2
   |   |   ...
   |
   └───logs
       │
       └───dag_1
       |   │
       |   └───task_1
       |   |   │   datefile_1
       |   |   │   datefile_2
       |   |   │   ...
       |   |
       |   └───task_2
       |       │   datefile_1
       |       │   datefile_2
       |       │   ...
       |
       └───dag_2
           │   ...

Streaming logs however are visible in Logs Explorer and you can view them using resource.type="cloud_composer_environment" query which will display all the logs from all the environments.

If you want to specify which environment you want to view just add another parameter to your query:

resource.type="cloud_composer_environment"
resource.labels.environment_name="composer_env_name_here".

You can also view the same logs in the Console in Composer environment details under "Logs" tab. If in doubt have a look at the documentation describing the process.

Upvotes: 2

Related Questions