mmziyad
Reputation: 324
How are WorkerHarnessThreads managed in Cloud Dataflow?
Is the option numberOfWorkerHarnessThreads used by cloud-dataflow runner now?
Earlier the PipelineOptions property numberOfWorkerHarnessThreads was specified in the doc and was displayed in Dataflow Job Monitoring UI under Pipeline options. Both are missing now.
If this is not used, how are the worker threads managed now?
Upvotes: 3
Views: 2920
Answers (1)
ihji
Reputation: 1496
The option is still there. You can find it in DataflowPipelineDebugOptions.
Upvotes: 1
Related Questions
- How does the dataflow authenticate the worker service account?
- Understanding Threading in Google Cloud DataFlow Workers
- Google Cloud Dataflow Worker Threading
- what is the entry point of the workers in apache beam? (what methods are called?)
- What could restrict dataflow pipeline to only using a single worker?
- How does Dataflow perform Parallel Processing?
- Apache Beam: Share resources among all threads within instance
- Google dataflow streaming pipeline is not distributing workload over several workers after windowing
- How does Dataflow autoscale and distribute workload?
- Cloud Dataflow - how does Dataflow do parallelism?