sgccarey
sgccarey

Reputation: 492

Streamz + Dask worker occupancy is lower than I'd expect

I've got a Dask cluster with 32 workers running on a local machine, and have tried to run the following Streamz workflow against it:

enter image description here

I'm only seeing a couple of the workers occupied at any given time:

enter image description here

I see increased occupancy when running locally using:

client = Client(n_workers=32, processes=True, threads_per_worker=1, memory_limit='32GB')

but still nowhere near 32 workers are occupied at any given time (max about 8).

Why is this, and why does the task stream appear to show more tasks running in parallel than the occupancy would suggest?

Upvotes: 1

Views: 73

Answers (0)

Related Questions