CODEWITHSUNDEEP
javagoogle-cloud-platformgoogle-cloud-dataflowapache-beamautoscaling
Shushu
Shushu

Reputation: 792

Dataflow job on Java SDK 2.11.0 does not scale

I have a streaming job that reads from pubsub and writes to pubsub, doing some basic processing in the middle.
When working on SDK 2.5.0, and executing it "directly", it works as expected, and scale as needed from 1 worker to as many workers as it should. Trying to upgrade to latest version, 2.11.0, and trying to use templates - it works ok, but does not scale.
Attached are both jobs' pipeline options.
Any suggestion ? Anything that might have been changed between those 2 versions that I am missing ?
Thanks,
Shushu SDK 2.5.0 example SDK 2.11.0 example

Clarification:
Even though I set 20 workers - I only get 1. enter image description here

Upvotes: 3

Views: 148

Answers (1)

Brent Worden
Brent Worden

Reputation: 10974

numWorkers is set to 20. As such, when the job starts, 20 workers will be allocated based on this value.

maxNumWorkers is also set to 20. So, no more than 20 workers will ever be allocated.

If need the job to scale beyond the initial 20 workers, modify the maxNumWorkers value.

Upvotes: 1

Related Questions