CODEWITHSUNDEEP
erlangelixir
Paulo Janeiro
Paulo Janeiro

Reputation: 3221

Why does Elixir need `Flow`? Isn't that handled by BEAM by default?

After seeing José Valim's last keynote speech about Flow & GenStage I became confused because:
1) If Flow & GenStage is being added to Elixir is because it really makes sense to improve concurrency;
2) on the other hand I had in my mind that one of the key Erlang/BEAM strengths was that it was concurrent/used all cores by default and a developer should not worry about that.
So, if Flow & GenStage is not (yet) to help with distributed systems running on different machines, could someone explain the differences between Flowand the native Erlang/BEAM concurrency?

Upvotes: 3

Views: 603

Answers (1)

Fred the Magic Wonder Dog
Fred the Magic Wonder Dog

Reputation: 9109

Flow and GenStage address a different problem than just concurrency.

What always happens in distributed systems is that once you have easy concurrency, you eventually run into the "thundering herd" problem where a single process or a small group of processes becomes a bottleneck for all the other processes in the system. This video is demonstrates the problem clearly.

https://youtu.be/8NPzLBSBzPI

The key concept of Flow and GenStage is back-pressure in the system to prevent the overflow of incoming requests to the workers. This back-pressure concept allows you to either add more workers or slow down the assembly line.

These modules are attempts to generalize the problem of self-scheduling that every sufficiently large BEAM application runs into. There have already been many one-off implementations of the ideas in them to allow BEAM applications to fail gracefully in the face of large loads.

Upvotes: 7

Related Questions