CODEWITHSUNDEEP
elixir
Ivan Yurov
Ivan Yurov

Reputation: 1618

Elixir GenStage scaling depending on load

I'm trying to build a following pipeline: [A] -> [B] -> [C]

A takes entries from database which have timestamp older than one day.

B performs some heavy task using external node.

C stores result of computation back to DB.

Each B-node can perform more than one task concurrently, and we can set the level of concurrency on node spawn.

What I want is to spawn as many B-nodes as needed depending on how many tasks are waiting in DB.

Is GenStage a good fit for that task or better to build something with queue and bunch of GenServers that talk to each other? How to approach it? Custom dispatcher, that not only distributes tasks, but also spawns B-nodes?

Upvotes: 0

Views: 265

Answers (1)

Aleksei Matiushkin
Aleksei Matiushkin

Reputation: 121000

I want to spawn as many B-nodes as needed depending on how many tasks are waiting in DB.

In this particular case, neither GenStage nor any Queue is required. GenStage is solving an exactly opposite task: it helps when you want to spawn at most N B-nodes, despite how many tasks are waiting and keep this value more or less constant.

I would implement what you need explicitly: just load all the waiting tasks from the database and Stream.map/2 them to GenServer’s instances.

These GenServers should be responsible for running long task and storing the result into the database afterward. Then they silently die.

Upvotes: 3

Related Questions