CODEWITHSUNDEEP
google-cloud-platformgoogle-cloud-data-fusion
sacoder
sacoder

Reputation: 189

Cloud Data Fusion pricing - development vs execution

we are looking to get some clarity on Cloud Data Fusion pricing. It looks like if we create a Cloud Data Fusion instance, as long as the instance is alive, we incur hourly rates. This can be quite high: $1100 per month for development and $3000 per enterprise instance. https://cloud.google.com/data-fusion/pricing There seems to be no way to stop an instance - this was confirmed by support, only delete.

However, the pricing talks of development vs execution. Wondering if we can avoid the instance charges once we are done deploying a pipeline. Not clear if this is possible or even a deployed pipeline requires an instance.

Thanks.

Upvotes: 3

Views: 2548

Answers (1)

guillaume blaquiere
guillaume blaquiere

Reputation: 75715

You can deploy your pipeline in 2 modes:

  • Either Cloud Data fusion create an ephemeral cluster, deploy your pipeline and tear down the cluster at the end -> Here you need to keep Data Fusion to tear down the cluster. But you can delete it before
  • Or run the pipeline on an existing cluster. This time, after the pipeline deployment and start, you can shut down the instance.

I agree, it's not clear but you can deduce this when you know how work an Hadoop cluster.

Note: don't forget to export your pipeline before deleting the instance

Note2: the instance also offer trigger scheduling to run the pipeline. Of course, if you delete the instance this feature is useless for you!

Upvotes: 1

Related Questions