CODEWITHSUNDEEP
google-cloud-platformgoogle-cloud-dataflowapache-beamgoogle-cloud-pubsub
Rafaël
Rafaël

Reputation: 1027

How to trigger an event when Dataflow Pipeline is finished?

I'm running a Dataflow Template of my own from a Cloud function and I would like to do something when Pipeline is over.

Is there an official way of doing it ?

I can't use the waitUntilFinish() since my Cloud function won't last that long. So I need some kind of event, to trigger some other processes.

Pubsub sounds great but how to send a single message ?

Any idea would be great !

Upvotes: 2

Views: 1764

Answers (1)

guillaume blaquiere
guillaume blaquiere

Reputation: 75735

The idea is the following.

  • Go to logs, select advanced filter (arrow on the right of the filter bar) and paste this custom filter
resource.type="dataflow_step" textPayload="Worker pool stopped."

If nothing is returned, try this

resource.type="gce_instance_group"
resource.labels.instance_group_name:"dataflow-"
jsonPayload.event_subtype="compute.instanceGroups.removeInstances"
jsonPayload.event_type="GCE_OPERATION_DONE"

You should see only your end of dataflow

  • Go to export
  • Select advance filter and paste again the filter
  • Clic on create export
  • set a sink name
  • Set the destination to PubSub
  • Select your topic
  • Now, plug a function on this topic, it will be trigger only at the end of dataflow. For the second filter, it's not really at the end, but when the VM are destroy, thus all the processing is finished.

Upvotes: 4

Related Questions