user10332687
Reputation:
Optimizing apache beam / cloud dataflow startup
I have done a few tests with apache-beam using both auto-scale workers and 1 worker, and each time I see a startup time of around 2 minutes. Is it possible to reduce that time, and if so, what are the suggested best practices for reducing the startup time?
Upvotes: 1
Views: 1489
Answers (1)
John Hanley
Reputation: 81464
IMHO: Two minutes is very fast for a product like Cloud Dataflow. Remember, Google is launching a powerful Big Data service for you that autoscales.
Compare that time to the other cloud vendors. I have seen some clusters (Hadoop) take 15 minutes to come live. In any event, you do not control the initialization process for Dataflow so there is nothing for you to improve.
Upvotes: 3
Related Questions
- Apache beam Initializer
- How to deal with (Apache Beam) high IO bottlenecks?
- It may take a long time to start a data flow job
- Optimising GCP costs for a memory-intensive Dataflow Pipeline
- How to make dataflow apache beam job faster
- Apache Beam / Google Dataflow Final step to run only once
- Datastore poor performance with Apache Beam & Dataflow
- Performance issues on Dataflow batch loads using Apache Beam
- Apache Beam job stalled on Google Cloud - CPU is high
- Apache beam on google cloud dataflow