CODEWITHSUNDEEP
google-cloud-platformgoogle-cloud-dataflow
Lak Fu
Lak Fu

Reputation: 195

GCP Dataflow vCPU usage and pricing question

I submited a GCP dataflow pipeline to receive my data from GCP Pub/Sub, parse and store to GCP Datastore. It seems work perfect.

Through 21 days, I found the cost is $144.54 and worked time is 2,094.72 hour. It means after I submitted it, it will be charged every sec, even there is not receive (process) any data from Pub/Sub.

Is this behavior normal? Or I set a wrong parameters? I thought CPU use time only be counted when data is received.

Is there have any way to reduce the cost in same working model (receive from Pub/Sub and store to Datastore)?

Upvotes: 1

Views: 1520

Answers (1)

F10
F10

Reputation: 2893

The Cloud Dataflow service usage is billed in per second increments, on a per job basis. I guess your job used 4 n1-standard-1 workers, which used 4 vCPUs giving an estimated of 2,000 vCPU hr resource usage. Therefore, this behavior is normal. To reduce the cost, you can use either autoscaling, to specify the maximum number of workers, or the pipeline options, to override the resource settings that are allocated to each worker. Depending on your necessities, you could consider using Cloud Functions which cost less, but considering its limits.

Hope it helps.

Upvotes: 6

Related Questions