CODEWITHSUNDEEP
google-cloud-platformairflowgoogle-cloud-composer
Amit Choudhary
Amit Choudhary

Reputation: 41

How to use Airflow Stable Rest API [Airflow version 2.0.0] deployed on GCP Cloud Composer

Env : We using GCP cloud composer to run airflow dags. Aim : To use apache airflow stable apis to trigger dags externally using REST.

We tried to use airflow experimental apis to trigger dags externally by setting airflow override configuration in cloud composer: airflow.api.auth.backend.default and making IAP request. It worked fine. We followed steps described in https://cloud.google.com/composer/docs/how-to/using/triggering-with-gcf.

Now starting from Airflow 2.0.0 + , airflow experimental apis are deprecated and new stable apis are introduced. The new stable api doesn't supports airflow.api.auth.backend.default. Hence it is throwing :403 FORBIDDEN.

Also we tried with auth_backend : airflow.api.auth.backend.basic_auth , but this also not working as when we pass user/password in Authorization header, we are not able to pass IAP Bearer token.

  1. Please share the step to access new airflow apis similar to steps shown to access experimental apis in https://cloud.google.com/composer/docs/how-to/using/triggering-with-gcf
  2. Do we need to do double Auth i.e first Authorization with IAP and then with Airflow ?

Upvotes: 4

Views: 3424

Answers (2)

Anton Kumpan
Anton Kumpan

Reputation: 344

TL'DR version:

In order to make Airflow Stable API work at GCP Composer:

  1. Set "api-auth_backend" to "airflow.composer.api.backend.composer_auth"
  2. Make sure your service account email length is <64 symbols
  3. Make sure your service account has required permissions (Composer User role should be sufficient)

Longread:

We are using Airflow for a while now, and started with version 1.x.x with "experimental" (now deprecated) API's.

To Authorize, we are using "Bearer" token obtained with service account:

# Obtain an OpenID Connect (OIDC) token from metadata server or using service account.
google_open_id_connect_token = id_token.fetch_id_token(Request(), client_id)

# Fetch the Identity-Aware Proxy-protected URL, including an
# Authorization header containing "Bearer " followed by a
# Google-issued OpenID Connect token for the service account.
resp = requests.request(
    method, url,
    headers={'Authorization': 'Bearer {}'.format(
        google_open_id_connect_token)}, **kwargs)

Now we are migrating to Airflow 2.x.x and faced with exact same issue: 403 FORBIDDEN.

Our environment details are:

composer-1.17.3-airflow-2.1.2 (Google Cloud Platform)

"api-auth_backend" is set to "airflow.api.auth.backend.default".

Documentation claims that:

After you set the api-auth_backend configuration option to airflow.api.auth.backend.default, the Airflow web server accepts all API requests without authentication.

However, this does not seem to be true.

In experimental way, we found that if "api-auth_backend" is set to "airflow.composer.api.backend.composer_auth", Stable REST API (Airflow 2.X.X) starting to work.

But there is other caveat to this: for us, some of our service accounts did work, and some did not. The ones that did not work were throwing "401 Unauthorized" error. We figured out that accounts having email length > 64 symbols were throwing error. Same was observed at this answer.

So after setting "api-auth_backend" to "airflow.composer.api.backend.composer_auth" and making sure that our service account email length is <64 symbols - our old code for Airflow 1.x.x started to work for Authentication. Then we needed to make changes (API URLs and response handling) and stable Airflow (2.x.x) API started to work for us in the same way as it was for Airflow 1.x.x.

UPD: this is a defect in Airflow and will be fixed here: https://github.com/apache/airflow/pull/19932

Upvotes: 3

user2262504
user2262504

Reputation: 7297

Cloud Composer's support for Airflow2 stable API starts from composer-1.17.0-preview.12 onwards.

The how-to documentation is here:

Upvotes: 1

Related Questions