CODEWITHSUNDEEP
azureloggingblobairflowlog-shipping
dan-0
dan-0

Reputation: 199

How to ship Airflow logs to Azure Blob Store

I'm having trouble following this guide section 3.6.5.3 "Writing Logs to Azure Blob Storage"

The documentation states you need an active hook to Azure Blob storage. I'm not sure how to create this. Some sources say you need to create the hook in the UI, and some say you can use an environment variable. Either way, none of my logs are getting written to blob store and I'm at my wits end.

Upvotes: 1

Views: 2453

Answers (2)

Ganesh
Ganesh

Reputation: 757

I achieved writing logs to blob using below steps

  1. Create folder named config inside airflow folder
  2. Create empty __init__.py and log_config.py files inside config folder
  3. Search airflow_local_settings.py in your machine

/home/user/env/lib/python2.7/site-packages/airflow/config_templates/airflow_local_settings.py /home/user/env/lib/python2.7/site-packages/airflow/config_templates/airflow_local_settings.pyc

run

cp /home/user/env/lib/python2.7/site-packages/airflow/config_templates/airflow_local_settings.py config/log_config.py
  1. Edit airflow.cfg [core] section

remote_logging = True

remote_log_conn_id = log_sync

remote_base_log_folder=wasb://[email protected]/logs/

logging_config_class =log_config.DEFAULT_LOGGING_CONFIG

  1. Add log_sync connection object as below enter image description here

  1. install airflow azure dependency

    pip install apache-airflow[azure]

  2. Restart webserver and scheduler

Upvotes: 0

Kunal Deo
Kunal Deo

Reputation: 2298

Azure Blob Store hook(or any hook for that matter) tells overflow how to write to into Azure Blob Store. This is already included in recent versions of airflow, wasb_hook.

You will need to make sure that the hook is able to write to Azure Blob Store. Just mention the REMOTE_BASE_LOG_FOLDER bucket should be named like wasb-xxx. Once you take care of these two things instructions works without a hitch,

Upvotes: 2

Related Questions