CODEWITHSUNDEEP
password-hashairflow
Qwerty
Qwerty

Reputation: 227

How to hide the password from log and rendered template when pass another airflow connection to airflow SSH Operator

Summary of my DAG:

I am using SSH Operator to SSH to an EC2 instance and run a JAR file which will connect to multiple DBs. I've declared the Airflow Connection in my DAG file and able to pass the variables into the EC2 instance. As you can see from below, I'm passing properties into JAVA command.

Airflow version - airflow-1-10.7
Package installed - apache-airflow[crypto]

from airflow import DAG
from datetime import datetime, timedelta
from airflow.contrib.hooks.ssh_hook import SSHHook
from airflow.contrib.operators.ssh_operator import SSHOperator
from airflow.hooks.base_hook import BaseHook
from airflow.models.connection import Connection

ssh_hook = SSHHook(ssh_conn_id='ssh_to_ec2')
ssh_hook.no_host_key_check = True
redshift_connection = BaseHook.get_connection("my_redshift")

rs_user = redshift_connection.login
rs_password = redshift_connection.password

mongo_connection = BaseHook.get_connection("my_mongo")
mongo_user = mongo_connection.login
mongo_password = mongo_connection.password


default_args = {
    'owner': 'AIRFLOW',
    'start_date': datetime(2020, 4, 1, 0, 0),
    'email': [],
    'retries': 1,
}

dag = DAG('connect_to_redshift', default_args=default_args)

t00_00 = SSHOperator(
    task_id='ssh_and_connect_db',
    ssh_hook=ssh_hook,
    command="java "
            "-Drs_user={rs_user} -Drs_pass={rs_pass} "
            "-Dmongo_user={mongo_user} -Dmongo_pass={mongo_pass} "
            "-jar /home/airflow/root.jar".format(rs_user=rs_user,rs_pass=rs_pass,mongo_user=mongo_user,mongo_pass=mongo_pass),
    dag=dag)

t00_00

Problem

The value for rs_pass,mongo_pass will be exposed in Rendered_Template/Airflow log which is not good and I would like to have a solution that can hide all these sensitive information from log and rendered template with SSH Operator.

So far I've tried to minimum the log verbose to ERROR in airflow.cfg, but it still shows in Rendered_Template.

Please enlighten me.

Thanks

Upvotes: 7

Views: 1104

Answers (1)

willshen
willshen

Reputation: 63

You can use jinja template for username and password. The password will be masked automatically then. You can find documentation how to template connection here

Your code will look like:

rs_user = "{{ conn['my_redshift'].login }}"
rs_password = "{{ conn['my_redshift'].password }}"

mongo_user = "{{ conn['my_mongo'].login }}"
mongo_password = "{{ conn['my_mongo'].password }}"

Upvotes: 0

Related Questions