CODEWITHSUNDEEP
google-cloud-platformsmtpairflowgoogle-secret-manager
Jinyu
Jinyu

Reputation: 169

"Authentication Credentials Invalid" when setup GCP Composer(Airflow) SMTP server

I am using GCP Composer (Airflow) to run some scheduled tasks. And using AWS SES SMTP server to send notification email.

According to GCP Composer's document Configuring your environment with Secret Manager and Configure an SMTP password for a third-party SMTP service, I set the Airflow configuration overrides as below

secrets
    backend airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend

smtp
    smtp_port 587
    smtp_mail_from ...
    smtp_host email-smtp.us-west-1.amazonaws.com
    smtp_starttls True
    smtp_password_secret smtp-password
    smtp_ssl False
    smtp_user ...
email
    email_backend airflow.utils.email.send_email_smtp

And add a secret entry airflow-variables-smtp-password in Secret Manager.

Airflow has an error when send the email:

...
Authentication Credentials Invalid

Upvotes: 1

Views: 888

Answers (1)

Jinyu
Jinyu

Reputation: 169

TL;DR It is a mistake in GCP document Configuring your environment with Secret Manager. The prefix are different in these two backends. You need to use prefix airflow-config for backend airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend, and airflow-variables for backend airflow.contrib.secrets.gcp_secrets_manager.CloudSecretsManagerBackend to access the secret in SecretManager. So my secret name in SecretManager should be airflow-config-smtp-password instead of airflow-variables-smtp-password.

According to document Configuring your environment with Secret Manager, it suggest to use secret backend airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend.

And the document especially has a warning says

Caution: Do not use airflow.contrib.secrets.gcp_secrets_manager.CloudSecretsManagerBackend because this value does not allow you to view logs on the Airflow web server UI.

And the document Configure an SMTP password for a third-party SMTP service gives an example about how to store an SMTP password in Secret Manager:

echo -n "SMTP_PASSWORD" | gcloud beta secrets create \
  airflow-variables-smtp-password \
  --data-file=- \
  --replication-policy=automatic

It use airflow-variables as prefix of the secret variable.

But the problem is, the backend airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend use prefix airflow-config for secret configs

in source code /opt/python3.6/lib/python3.6/site-packages/airflow/providers/google/cloud/secrets/secret_manager.py

    def __init__(
        ...
        variables_prefix: str = "airflow-variables",
        config_prefix: str = "airflow-config",
        ...
    ) -> None:
        ...

    def get_variable(self, key: str) -> Optional[str]:
        ...
        return self._get_secret(self.variables_prefix, key)

    def get_config(self, key: str) -> Optional[str]:
        ...
        return self._get_secret(self.config_prefix, key)

And the backend airflow.contrib.secrets.gcp_secrets_manager.CloudSecretsManagerBackend use prefix airflow-variables for both variables and secret configs

in source code /usr/local/lib/airflow/airflow/contrib/secrets/gcp_secrets_manager.py

    def __init__(
        ...
        variables_prefix="airflow-variables",  # type: str
        ...
    ):
        ...

    def get_variable(self, key):
        ...
        return self._get_secret(self.variables_prefix, key)

So if you are following the suggested backend, then you should use airflow-config as secret variable prefix. In my case, it should be airflow-config-smtp-password in secret manager.

Upvotes: 4

Related Questions