CODEWITHSUNDEEP
google-cloud-platformairflowgoogle-cloud-pubsubgoogle-cloud-composer
manish dev
manish dev

Reputation: 97

How to trigger a google composer DAG on a pub/sub publish message?

Google cloud function cannot be used to trigger the composer DAG on Pub/Sub message

I have tried the PubSubPullSensor 

pull_messages = PubSubPullSensor(
    task_id="pull_messages",
    ack_messages=True,
    project='xxxx',
    subscription='xxxx',
)

but this doesn't trigger ,as DAG as expected

Any help to trigger the DAG on Pub/Sub message will be greatly appreciated?

Upvotes: 3

Views: 2187

Answers (1)

Tlaquetzal
Tlaquetzal

Reputation: 2850

The PubSubPullSensor is part of a DAG. The dag needs to be running in order to the sensor to be executed, that's why it didn't work.

I think that the most striaghtforward approach would be to use cloud functions, but if that's not possible, the second option would be to use another server (it could be another computing option within GCP: Cloud AppEngine, Cloud Run, etc.) to receive the Pub/Sub message and trigger the dag. Basically, the same idea as Pub/Sub + Cloud Functions, without functions.

If you don't want to follow this approach, and want to trigger the Dag within the same Composer environment, you could use PubSubPullSensor on a running dag and use the TriggerDagRunOperator when needed.

This idea can be used in many forms; however, the tricky thing is that PubSubPullSensor needs to be executed in a running dag. A possible solution for this, is to schedule the dag to run often, for example every 5 minutes

Upvotes: 4

Related Questions