CODEWITHSUNDEEP
google-cloud-platformdataflow
Quack
Quack

Reputation: 740

Where is FlexRS type in dataflow part of batch or stream processing?

I have a question about the FlexRS type while I was looking at the dataflow of Google Cloud Platform. :)

As far as I know, dataflow supports batch and stream, but I was curious to know that there is a FlexRS type.

Simply understanding FlexRS was difficult to identify in the document except that it was cheaper than a typical workflow.

Can I ask you to explain FlexRS?

Thank you.

Upvotes: 0

Views: 480

Answers (1)

Alexandre Moraes
Alexandre Moraes

Reputation: 4051

Dataflow is a fully managed batch and streaming analytics service that minimises latency, processing time through autoscaling.

Regarding FlexRS for Dataflow, ou can use it in batch processing pipelines which are not time-critical, such as daily or weekly jobs that can be completed within a certain time-window. Normally, Dataflow uses both and preemptible and regular workers to execute your job. It takes into account the availability of preemptible VM`s and you are charged according to the documentation.

On the other hand, FlexRS offers a discounted rate for CPU and memory pricing for batch processing. It can delay your Dataflow batch job within a 6-hour window to identify the best point in time to start the job, based on the availability of resources. When enabled, FlexRS selects preemptible VMs for 90% of workers in the worker pool by default.

Therefore, FlexRS is used only for non time-critical batch workloads.

Upvotes: 1

Related Questions