CODEWITHSUNDEEP
python-3.xgoogle-cloud-platformairflowgoogle-cloud-dataflowgoogle-cloud-composer
Shriyut Jha
Shriyut Jha

Reputation: 79

Trigger Dataflow job on file arrival in GCS using Cloud Composer

I'm new to GCP and trying to gain a better understanding of ETL services offered by Google Cloud. I'm trying to do a POC where a dataflow job runs whenever a file arrives at a specified location in GCS, I want to utilize Cloud Composer for the same.

How can I utilize different airflow operators (one to watch for file in GCS and other to trigger dataflow job) in a DAG?

Upvotes: 0

Views: 1337

Answers (1)

MBHA Phoenix
MBHA Phoenix

Reputation: 2217

  1. Create a Cloud Function triggered when a file is created / arrives at your bucket. You can find how to deploy such function in the following documentation https://cloud.google.com/functions/docs/calling/storage#deploy
  2. Code your function to trigger your DAG, it's well explained in the following documentation: https://cloud.google.com/composer/docs/how-to/using/triggering-with-gcf
  3. Use one of the following airflow operators to run a dataflow job as a task within your DAG
  • DataflowCreateJavaJobOperator if you are using Java SDK
  • DataflowCreatePythonJobOperator if you are using Python SDK

More details here : https://airflow.apache.org/docs/apache-airflow-providers-google/2.1.0/operators/cloud/dataflow.html

That was the answer to your question about composer, but you should consider replacing Composer by Cloud Workflows, and rather than using an airflow dataflow operator, you can use Cloud Workflows dataflow connector, and you can code your cloud function to trigger a Cloud Workflows execution. There are many clients library in different languages to do that. Here the link to choose your library: https://cloud.google.com/workflows/docs/reference/libraries

Finally Cloud Workflows pros compared to Composer.

  • it's serverless: you don't have to bother about machine types, network, ...

  • you use yaml to describe your workflow in easy and fast way, so you don't have to worry about learning airflow

  • it's way cheaper than composer

  • ...

Upvotes: 0

Related Questions