CODEWITHSUNDEEP
airflowgoogle-cloud-pubsubdirected-acyclic-graphsgoogle-cloud-composer
harry77
harry77

Reputation: 117

Trigger Cloud Composer DAG with a Pub/Sub message

I am trying to create a Cloud Composer DAG to be triggered via a Pub/Sub message. There is the following example from Google which triggers a DAG every time a change occurs in a Cloud Storage bucket: https://cloud.google.com/composer/docs/how-to/using/triggering-with-gcf

However, on the beginning they say you can trigger DAGs in response to events, such as a change in a Cloud Storage bucket or a message pushed to Cloud Pub/Sub. I have spent a lot of time try to figure out how that can be done, but no result.

Can you please help or giving me some directions? Thanks!

Upvotes: 4

Views: 7647

Answers (2)

Ryan Yuan
Ryan Yuan

Reputation: 2566

There are 2 ways to trigger a DAG by a Pub/Sub events.

  1. You can place a PubSubPullSensor at the beginning of your DAG. Your DAG will be triggered every time a Pub/Sub message can be pulled by the PubSubPullSensor. And it will execute the rest of the tasks in your DAG.
  2. You can also create a Cloud Function that acts as Pub/Sub trigger. And put the Composer DAG triggering logic inside the Cloud Function trigger. When a message is published to the Pub/Sub topic, the Cloud Function should be able to trigger the Composer DAG.

Upvotes: 7

hexacyanide
hexacyanide

Reputation: 91649

To extend the public documentation page you already posted, you can configure a Cloud Function to run each time a message is published to a Cloud Pub/Sub topic. There is more information about that in another public documentation page.

To attach a function to a topic, set the --trigger-topic flag when deploying the function:

gcloud functions deploy $FUNCTION_NAME --runtime $RUNTIME --trigger-topic $TOPIC_NAME

Upvotes: 0

Related Questions