CODEWITHSUNDEEP
pythongoogle-cloud-platformairflowgoogle-cloud-composer
Kyle
Kyle

Reputation: 19

Cloud Composer + Airflow: Setting up DAWs to trigger on HTTP (or should I use Cloud Functions?)

Ultimately, what I want to do is have a Python script that runs whenever a HTTP request is created, dynamically. It'd be like: App 1 runs and sends out a webhook, Python script catches the webhook immediately and does whatever it does.

I saw that you could do this in GCP with Composer and Airflow.

But I'm having several issues following these instrutions https://cloud.google.com/composer/docs/how-to/using/triggering-with-gcf:

  1. Running this in Cloud Shell to grant blob signing permissions:

    gcloud iam service-accounts add-iam-policy-binding
    [email protected]
    --member=serviceAccount:[email protected]
    --role=roles/iam.serviceAccountTokenCreator

When I put in my project ID, I get a "Gaia id not found for [email protected]"

  1. When I run the airflow_uri = environment_data['config']['airflowUri'] bit, I get a key error on 'config'.

Is there a better way to do what I'm trying to do (i.e. run Python scripts dynamically)?

Upvotes: 1

Views: 1149

Answers (1)

aga
aga

Reputation: 3903

The reason for getting: Gaia id not found for email <project-id>@appspot.gserviceaccount.com error is not enabling all needed APIs in your project. Please follow the steps:

  1. Create or select Google Cloud Platform Project you wish to work with.
  2. Enable the Cloud Composer, Google Cloud Functions and Cloud Identity and Google Identity and Access Management (IAM) APIs. You can find it in Menu -> Products -> Marketplace and typing the name of corresponding API.
  3. Grant blob signing permissions to the Cloud Functions Service Account. In order for GCF to authenticate to Cloud IAP, the proxy that protects the Airflow webserver, you need to grant the Appspot Service Account GCF the Service Account Token Creator role. Do so by running the following command in your Cloud Shell, substituting the name of your project for <your-project-id>:
gcloud iam service-accounts add-iam-policy-binding \
<your-project-id>@appspot.gserviceaccount.com \
--member=serviceAccount:<your-project-id>@appspot.gserviceaccount.com \
--role=roles/iam.serviceAccountTokenCreator

I tested the scenario, firstly without enabling APIs and I've retrieved the same error as you. After enabling the APIs, error disappear and IAM policy has been updated correctly.

There is already well described Codelabs tutorial, which shows the workflow of triggering the DAG with Google Cloud Functions.

Upvotes: 1

Related Questions